Data buffer switching method, chip and system

Abstract

The application relates to the technical field of integrated circuits, and discloses a data buffer switching method, a chip and a system. In a model loading stage, a plurality of virtual addresses and corresponding physical addresses are allocated to tensors that need to switch data buffers, and address mapping relationships from all virtual addresses to physical addresses are established in advance in an SMMU page table, the SMMU page table is determined when the model is loaded and remains unchanged; in a task dispatching stage, address slots of the model are stored in an internal storage unit of a computing core; in a task execution stage, when the data buffers need to be switched, the virtual addresses of the address slots in the system memory are replaced by virtual addresses corresponding to the switched data buffers; the computing core reacquires the address slots from the system memory and stores them in the internal storage unit, so that the corresponding data buffers are accessed based on the switched virtual addresses. The application avoids performance jitter caused by TLB invalidation, improves address translation efficiency, and meets the demand of high frame rate real-time scenes.

Description

Technical Field

[0001] This application relates to the field of integrated circuit technology, and in particular to a data buffer switching method, chip, and system. Background Technology

[0002] With the rapid development of artificial intelligence technology, hardware accelerators (such as Neural Processing Units (NPUs), Tensor Processing Units (TPUs), and General Purpose Graphics Processing Units (GPGPUs)) are widely used in various computing systems as hardware units specifically designed to accelerate specific computing tasks. In scenarios such as autonomous driving, intelligent monitoring, and real-time image processing, hardware accelerators need to process data from different data sources or at different times. To achieve pipelined operation and improve processing efficiency, systems typically prepare multiple data buffers for each tensor that requires dynamic data buffer switching, and perform round-robin switching of data buffers between different data sources or at different times. However, existing technologies mainly suffer from the following problems:

[0003] First, frequent modifications to the SMMU page table are required when switching data buffers, resulting in significant performance overhead.

[0004] In existing technologies, even if the virtual address remains fixed, the address mapping entries in the SMMU page table still need to be updated when the data buffer rotates. Specifically, each rotation requires performing unmap and map operations on the SMMU, accompanied by a Translation Lookaside Buffer (TLB) invalidation operation. The overhead of these operations is typically on the microsecond scale, causing performance jitter in the system and failing to meet the requirements of high frame rate real-time inference scenarios.

[0005] II. Low address translation efficiency in multi-frame processing scenarios affects real-time performance.

[0006] In high frame rate video streaming scenarios (such as 30 or 60 frames per second), the processing time for each frame is limited. Frequent TLB failures cause subsequent memory access requests to miss the TLB, requiring a re-traversal of the SMMU page table for address translation, significantly increasing address translation latency. This sudden and unstable latency makes system performance unpredictable, making it difficult to guarantee that the inference task will be completed within the specified time for each frame.

[0007] Therefore, how to achieve low-latency, jitter-free data buffer switching in high frame rate data stream processing scenarios has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] To address the aforementioned technical problems, the technical solution adopted in this application is as follows:

[0009] In a first aspect, one embodiment of this application provides a data buffer switching method, applied to a system including a system memory management unit (SMMU), a runtime module, and a hardware accelerator, the method comprising:

[0010] During the model loading phase, the runtime module allocates multiple virtual addresses for tensors that need to switch data buffers during the same model execution, allocates a physical address corresponding to a data buffer for each virtual address, establishes address mapping entries from the multiple virtual addresses to the corresponding physical addresses in the SMMU page table corresponding to the model, and writes the virtual address currently used by each tensor into the corresponding address slot in the system memory; the SMMU page table is determined during model loading and remains unchanged after model loading;

[0011] During the task dispatch phase, the runtime module stores the address slot of the model into the internal storage unit of the computing core of the hardware accelerator.

[0012] During the task execution phase, when it is necessary to switch the data buffer of the target tensor, the runtime module replaces the virtual address in the address slot corresponding to the target tensor in the system memory with the virtual address corresponding to the switched data buffer.

[0013] After the virtual address in the address slot of the target tensor is replaced with the virtual address corresponding to the switched data buffer, the computing core retrieves the address slot of the model from the system memory and stores it in the internal storage unit to access the corresponding data buffer based on the switched virtual address.

[0014] Secondly, one embodiment of this application provides a chip, comprising:

[0015] The System Memory Management Unit (SMMU) is configured with an SMMU page table. The SMMU page table contains address mapping entries for each tensor of the model. The address mapping entries correspond to the mapping relationship between the virtual address and the physical address of the data buffer of each tensor. Among them, tensors that need to switch data buffers have multiple data buffers, and each data buffer corresponds to an address mapping entry. The SMMU page table is determined when the model is loaded and remains unchanged after the model is loaded.

[0016] Multiple computing cores, each of which is configured with an internal storage unit, the internal storage unit being used to cache the address slots corresponding to the tensors, the address slots being used to store the virtual addresses of the currently used data buffers;

[0017] The computing core is used to replace the virtual address in the address slot corresponding to the tensor in the internal storage unit with the virtual address corresponding to the switched data buffer when the data buffer of the tensor needs to be switched during task execution, so as to access the corresponding data buffer based on the switched virtual address.

[0018] Thirdly, one embodiment of this application provides a data buffer switching system, including:

[0019] The chip as described in the second aspect;

[0020] The system memory, connected to the chip, includes multiple data buffers and stores the address slots;

[0021] The runtime module is used to allocate multiple virtual addresses for tensors that need to switch data buffers during the operation of the same model, allocate a physical address corresponding to a data buffer for each virtual address, establish address mapping entries from the multiple virtual addresses to the corresponding physical addresses in the SMMU page table corresponding to the model, and write the virtual address currently used by each tensor into the corresponding address slot in the system memory.

[0022] The runtime module is also used to store the address slot of the model into the internal storage unit of the computing core of the chip during the task dispatch phase.

[0023] The runtime module is also used during the task execution phase to replace the virtual address in the address slot corresponding to the target tensor in the system memory with the virtual address corresponding to the switched data buffer when it is necessary to switch the data buffer of the target tensor.

[0024] The data buffer switching method, chip, and system provided in this application have at least the following beneficial effects:

[0025] First, switching the data buffer does not modify the SMMU page table, thus avoiding performance jitter caused by TLB failure.

[0026] During the model loading phase, multiple virtual addresses and corresponding physical addresses are allocated for tensors that require data buffer switching, and the address mapping relationship from virtual address to physical address is pre-established in the SMMU page table. When a data buffer needs to be switched, only the virtual address in the address slot needs to be modified to the virtual address corresponding to the switched data buffer. Since the address mapping relationship of all data buffers has been pre-established in the SMMU page table during the model loading phase, the SMMU page table does not need to be modified during the switch. Therefore, there is no need to perform TLB invalidation operations, the cached entries in the TLB remain valid, and subsequent memory access requests can reliably hit the TLB. This avoids the increased latency and performance jitter caused by traversing the SMMU page table after a TLB miss due to a TLB invalidation, thus improving address translation efficiency.

[0027] Second, the data buffer switching latency is low, meeting the requirements of high frame rate real-time scenarios.

[0028] Since the switching operation only involves modifying the virtual address in the address slot (i.e., a single memory write operation), there is no need to perform SMMU demapping and remapping operations, nor TLB invalidation operations. This significantly reduces switching latency from microseconds to nanoseconds. In high-frame-rate real-time inference scenarios, this solution ensures stable and controllable processing time for each frame, meeting real-time requirements.

[0029] Third, there is no need to configure a separate address register for each tensor, which simplifies the hardware structure.

[0030] In existing technologies, the NPU requires a separate address register for each tensor to store its address information. In this application, the virtual address of the tensor is stored in an address slot in system memory, rather than in a register within the NPU. When switching data buffers, only the virtual address in the address slot needs to be modified, eliminating the need to occupy register resources within the NPU, thus simplifying the hardware structure.

[0031] Fourth, the data buffer allocation is flexible and supports switching any number of data buffers.

[0032] During the model loading phase, any number of data buffers can be allocated to tensors that require data buffer switching, based on actual needs, and the address mapping relationship of all data buffers is pre-established in the SMMU page table. During model execution, these data buffers can be freely switched by modifying the virtual address in the address slot. The switching operation is not limited by the number of data buffers, providing high flexibility.

[0033] Fifth, the address slot configuration enables the separation of address management between the offline compilation stage and the model loading stage.

[0034] During the offline compilation phase, the compiler only reserves address slot locations or allocates entry indexes based on the model's logical structure, without needing to bind to runtime data buffer allocation strategies or physical address layouts. During the model loading phase, the runtime module independently allocates and writes virtual addresses based on the reserved locations or entry indexes. This mechanism makes the generation of the model binary file and runtime data buffer management independent of each other, allowing the same compilation output to flexibly adapt to different runtime memory environments. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the hardware architecture of a computing system provided in an embodiment of this application;

[0037] Figure 2 A flowchart illustrating a data buffer switching method provided in an embodiment of this application;

[0038] Figure 3 A schematic flowchart of a data buffer switching method provided in an embodiment of this application during the model loading stage;

[0039] Figure 4 A flowchart illustrating the data buffer switching method provided in an embodiment of this application during the task dispatch phase;

[0040] Figure 5 Another flowchart illustrating the data buffer switching method provided in an embodiment of this application during the task execution phase;

[0041] Figure 6 A flowchart illustrating the data buffer switching method provided in an embodiment of this application during the data buffer switching stage;

[0042] Figure 7 This is a schematic diagram of the internal structure of an address generation unit provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0044] The data buffer switching method provided in this application can be applied to systems that include a system memory management unit (SMMU) and hardware accelerators, including but not limited to neural network processors (NPUs), tensor processing units (TPUs), general-purpose graphics processing units (GPGPUs), and video codecs / decoders. The following detailed description uses an NPU as an example to illustrate the technical solution of this application; however, those skilled in the art should understand that this description does not constitute a limitation on the scope of protection of this application.

[0045] Tensor: A tensor is the basic organizational form of data in a neural network model, which can be understood as a multidimensional array. During model inference, all data involved in the computation exists in the form of tensors. Specific types of tensors in a neural network model include, but are not limited to: input tensors (such as image data input during inference), output tensors (such as classification results, bounding box coordinates, etc. output during inference), weight tensors (weight parameters of each layer of the model), bias tensors (bias parameters of each layer of the model), and intermediate activation tensors (intermediate results generated by calculations in each layer of the model).

[0046] To improve translation speed during address translation by the SMMU, a Translation Lookaside Buffer (TLB) is configured internally. The TLB caches recently used address mapping entries from virtual addresses to physical addresses. When the SMMU receives a translation request for a virtual address, it first checks the TLB to see if a corresponding address mapping entry exists. If it does, the corresponding physical address can be obtained directly without accessing the SMMU page table; if it does not exist, the SMMU page table needs to be traversed to search for the entry, and the found address mapping entry is cached in the TLB.

[0047] When an address mapping entry in the SMMU page table is modified, the old mapping entry cached in the TLB becomes inconsistent with the modified page table content. In this case, a TLB invalidation operation needs to be performed to clear the old mapping entry from the TLB. After the TLB invalidation operation, subsequent memory access requests will not be able to hit the TLB, requiring a re-traversal of the SMMU page table for address translation until the new address mapping entry is re-cached in the TLB. During this period, the address translation latency increases, and this latency is sudden and unstable, resulting in performance jitter.

[0048] The data buffer switching method provided in this application is applied to a computing system including a hardware accelerator and a system memory management unit (SMMU) to dynamically switch the tensor data buffer during model execution. This method can be applied to scenarios requiring continuous processing of multiple frames of data (e.g., video stream inference, autonomous driving perception, real-time image processing, etc.), where input data changes between frames, necessitating frequent data buffer switching to ensure pipeline efficiency.

[0049] like Figure 1 As shown, the computing system provided in this application embodiment includes: a system memory management unit (SMMU) 10, a neural network processor (NPU) 20, system memory 30, and a runtime module 40.

[0050] The runtime module 40 runs on the host central processing unit (CPU) and is a software / firmware layer component in the computing system. It is used to manage the models and hardware resources in the computing system, issue tasks to the NPU20, and configure page tables to the SMMU10, among other operations that require software execution.

[0051] System memory 30 (e.g., DDR memory) is used to store data required for the operation of the computing system. This data includes, but is not limited to, model binary files for each model and tensor data (including input tensors, output tensors, weight tensors, bias tensors, etc.) that need to be read and written during the operation of each model. The data buffer is a contiguous storage area in system memory 30 used to store tensor data, and each data buffer has a defined physical address. Figure 1 As shown, the system memory 30 includes multiple data buffers (buffer0, buffer1, buffer2, etc.). The same quantity can correspond to multiple data buffers. During the model operation, the pipelined processing of data is achieved by switching between different data buffers.

[0052] SMMU10 is the system memory management unit, configured with an SMMU page table. The SMMU page table records address mapping entries from the virtual addresses of each tensor in the model to their corresponding physical addresses. SMMU10 receives memory access requests initiated by NPU20 and performs address translation based on the virtual addresses in the memory access requests.

[0053] When the computing system loads a model, the runtime module 40 reads the model binary file from non-volatile storage into the system memory 30, establishes an SMMU page table corresponding to the model in SMMU10, and establishes address mapping entries from virtual addresses to physical addresses for each tensor of the model in the SMMU page table.

[0054] When the computing core 22 in the NPU20 sends a memory access request carrying a virtual address to the SMMU10, the SMMU10 looks up the corresponding physical address in the SMMU page table based on the virtual address carried in the request, and returns the physical address to the computing core 22.

[0055] It should be noted that in scenarios where multiple models share the same hardware accelerator, each model has its own corresponding SMMU page table, and the virtual address space of each model is managed separately by its respective SMMU page table. The SMMU looks up the corresponding physical address in the SMMU page table of the corresponding model based on the virtual address carried in the memory access request. The SMMU page table of each model is established when the model is loaded and remains unchanged after the model is loaded.

[0056] The NPU20 includes a Task Scheduler (TSM) 21 and multiple compute cores 22 (e.g., ...). Figure 1 (Core0~Core3 in the system). The task scheduler 21 is used to receive tasks issued by the runtime module 40 and allocate the tasks to the corresponding computing cores 22. The computing cores 22 are used to execute the tasks allocated by the task scheduler 21, and when they need to obtain the data required to execute the task, they send a memory access request carrying a virtual address to the SMMU 10, receive the physical address returned by the SMMU 10, and obtain the required data from the system memory 30 or store the data into the system memory 30 based on the physical address.

[0057] Each computing core 22 is equipped with a DMA (Direct Memory Access) control unit 222, an internal storage unit 223, and an address generation unit 224.

[0058] The DMA control unit 222 is used to read data required by the computing core to perform tasks from the system memory 30, or to write the processing results of the computing core into the system memory 30. The data includes, but is not limited to, address slot contents, model instructions, weight data, and input data.

[0059] Internal storage unit 223 is used to cache data read from system memory 30 by DMA control unit 222.

[0060] Address generation unit 224 is used to read the contents of the address slot from the internal storage unit 223 and generate the virtual address required for the memory access request based on the virtual address in the address slot.

[0061] This application provides a data buffer switching method, applied to a system including a System Memory Management Unit (SMMU), a runtime module, and a hardware accelerator. (See also...) Figure 2 The data buffer switching method includes at least the following steps:

[0062] S100. During the model loading phase, the runtime module allocates multiple virtual addresses for tensors that need to switch data buffers during the same model operation, allocates a physical address corresponding to a data buffer for each virtual address, establishes multiple address mapping entries from virtual addresses to corresponding physical addresses in the SMMU page table corresponding to the model, and writes the virtual address currently used by each tensor into the corresponding address slot in the system memory.

[0063] The SMMU page table is determined when the model is loaded and remains unchanged after the model is loaded.

[0064] S200. During the task dispatch phase, the runtime module stores the model's address slots into the internal storage unit of the hardware accelerator's computing core.

[0065] S300. During the task execution phase, when it is necessary to switch the data buffer of the target tensor, the runtime module replaces the virtual address in the address slot corresponding to the target tensor in the system memory with the virtual address corresponding to the switched data buffer.

[0066] S400: After the virtual address in the address slot of the target tensor is replaced with the virtual address corresponding to the switched data buffer, the computing core retrieves the address slot of the model from the system memory and stores it in the internal storage unit to access the corresponding data buffer based on the switched virtual address.

[0067] By following the steps above, only the virtual address in the address slot needs to be modified when switching data buffers, and the SMMU page table itself does not need to be modified, thereby avoiding TLB failure operations and reducing switching latency.

[0068] The specific implementation methods of each stage described above are described in detail below with reference to the accompanying drawings.

[0069] Before the aforementioned computing system loads the model, the model is first compiled offline using a compiler to generate the corresponding model binary file (model.bin), which is then stored in a non-volatile storage medium (such as a hard drive or flash memory). The model binary file contains a command bundle, an NPU-executable instruction stream, weight data, and address slots for each tensor in the model.

[0070] The command package is used by the task scheduler 21 to perform task scheduling. It contains a sequence of commands for all computation partitions of the model. Each command includes the task type, task identifier, dependencies, and task priority.

[0071] The instruction stream is used by the NPU computing core to execute computational tasks. It contains a sequence of instructions that the NPU can execute (such as convolution instructions, pooling instructions, activation instructions, DMA instructions, etc.) and weight data.

[0072] Address slots are storage spaces reserved by the compiler for each tensor in the model binary file, used to store the virtual address of that tensor during model runtime. There are two ways address slots are stored:

[0073] In one implementation, address slots are distributed throughout the model binary file, with each tensor corresponding to one address slot. During the offline compilation phase, the compiler reserves address slots for each tensor and records the offset of each address slot within the model binary file. The runtime module locates the address slot for each tensor based on its corresponding address slot offset and writes the virtual address into the corresponding address slot.

[0074] In another implementation, address slots are aggregated into an input parameter table. The input parameter table resides at a fixed virtual address, and each entry in the table corresponds to an address slot. During the offline compilation phase, the compiler assigns entry indices to each tensor, and the runtime module writes the virtual address into the corresponding entry in the input parameter table based on the entry index of each tensor. Because the input parameter table resides at a fixed virtual address, the hardware can directly read the contents of the input parameter table at this fixed virtual address without requiring software configuration of the base address register.

[0075] During the operation of the aforementioned computing system, when a model needs to be loaded, the runtime module 40 performs a model loading operation, referencing... Figure 3 Specifically, it includes the following steps:

[0076] S101, The runtime module 40 obtains the model binary file of the model and stores it in the system memory 30.

[0077] Specifically, the runtime module 40 reads the model binary file from non-volatile storage medium into system memory 30. The model binary file contains command packages, NPU-executable instruction streams, weight data, and address slots reserved by the compiler for each tensor.

[0078] In this application, a tensor is a data unit that participates in the calculation in the model, and each tensor corresponds to one or more data buffers in the system memory 30. The data buffer is a contiguous storage area in the system memory used to store tensor data, and has a defined physical address.

[0079] Each tensor is assigned a virtual address, which is mapped to a physical address in the data buffer via the SMMU page table. For static tensors (tensors that do not require switching data buffers, such as weight tensors and bias tensors), one tensor corresponds to one virtual address and one physical address; for dynamic tensors (tensors that require switching data buffers, such as input tensors and output tensors), one tensor corresponds to multiple virtual addresses and multiple physical addresses, with each virtual address mapped to a different physical address.

[0080] The following table shows an example of the correspondence between tensors, data buffers, virtual addresses, and physical addresses:

[0081]

[0082] During continuous inference of the same model, the input data changes between frames, requiring the data of the new frame to be stored in different data buffers. The system allocates multiple data buffers for tensors (such as input tensors) that require data buffer switching and rotates them between frames. Since the mapping relationship between the virtual address and physical address of all data buffers is pre-established in the SMMU page table during the model loading phase, switching only requires modifying the virtual address in the address slot to point to the physical address corresponding to the new data buffer; the SMMU page table itself does not need to be modified.

[0083] For example, two virtual addresses, VA1 and VA2, are allocated to the input tensor input_0 in the table above, which are mapped to the physical addresses PA1 and PA2 of two data buffers, respectively. The current frame uses VA1 (pointing to Buffer1). When switching to the next frame, the runtime module changes the virtual address in the address slot to VA2 (pointing to Buffer2). The NPU will then automatically read the data from Buffer2 when accessing this tensor subsequently. The entire switching process involves only one address slot write operation, and the SMMU page table remains unchanged.

[0084] S102, Runtime module 40 establishes the SMMU page table corresponding to the model in SMMU10.

[0085] The SMMU page table is a data structure used by SMMU10 for address translation, storing the address mapping relationship from virtual address to physical address.

[0086] S103, runtime module 40 allocates virtual addresses for each tensor of the model.

[0087] During the offline compilation phase, the compiler allocates virtual addresses for each tensor and writes these virtual addresses directly into the corresponding address slots in the model binary file as the initial values ​​of the address slots. When loading the model, the runtime module 40 can either directly use the existing virtual addresses in the address slots, or it can reallocate new virtual addresses for each tensor and write the reallocated virtual addresses into the corresponding address slots in the model binary file.

[0088] In one alternative embodiment, the compiler records the number of data buffers (i.e., buffer pool depth) corresponding to each tensor during the offline compilation phase, and this information is stored in the model binary file. During the model loading phase, the runtime module 40 allocates a corresponding number of virtual addresses to each tensor based on the number of data buffers. For static tensors, the runtime module 40 allocates one virtual address and one data buffer; for dynamic tensors, the runtime module 40 allocates a corresponding number of virtual addresses and data buffers based on the number of data buffers.

[0089] S104, runtime module 40 allocates data buffers for each tensor in system memory 30.

[0090] For tensors that do not require switching data buffers during model execution (such as the model's weight tensor and bias tensor), i.e., static tensors, the runtime module 40 allocates a data buffer for each static tensor, and each data buffer corresponds to a physical address.

[0091] For tensors that require switching data buffers during model execution (such as the model's input and output tensors), i.e., dynamic tensors, the runtime module 40 allocates multiple virtual addresses for these dynamic tensors and allocates multiple data buffers in the system memory 30 when loading the model. Each data buffer corresponds to a physical address, and an address mapping relationship is established between each virtual address and its corresponding physical address. These virtual addresses are then added as address mapping entries to the model's SMMU page table.

[0092] For example, for an input tensor, runtime module 40 allocates three virtual addresses VA1, VA2, and VA3, and three data buffers Buffer1, Buffer2, and Buffer3, with physical addresses PA1, PA2, and PA3, respectively. Three address mapping entries are established in the SMMU page table: VA1→PA1, VA2→PA2, and VA3→PA3. This input tensor has a corresponding address slot in the model binary file, which stores the currently used virtual address. Runtime module 40 writes the currently used virtual address (e.g., VA1) into this address slot. The other virtual addresses of the tensor are recorded in the metadata area of ​​the model binary file in system memory. During model execution, when a data buffer needs to be switched, runtime module 40 retrieves the switched virtual address from the metadata area and rewrites the virtual address in the address slot (e.g., changing it from VA1 to VA2), causing the data accessed by the NPU to switch from Buffer 1 to Buffer 2.

[0093] S105, Runtime module 40 adds the address mapping relationship from virtual address to physical address of each tensor to the SMMU page table of the model.

[0094] The runtime module 40 adds the virtual-to-physical address mappings of each tensor to the SMMU page table of the model. For a tensor that has been allocated a data buffer, one address mapping is added to the SMMU page table as one address mapping entry. For a tensor that has been allocated multiple data buffers, multiple address mappings are added to the SMMU page table as multiple address mapping entries, with one address mapping entry corresponding to each data buffer.

[0095] After the above operations, the model's binary file has been stored in system memory 30, the model's SMMU page table has been established and contains the address mapping relationship of all tensors in the model, and the virtual address of each tensor has been written into the corresponding address slot. At this point, the model has been loaded, but has not yet run in computing core 22.

[0096] In this application, the SMMU page table is fully established during the model loading phase and remains unchanged during model operation. Therefore, there is no need to perform TLB invalidation operations at runtime, and the cached entries in the TLB always remain valid, avoiding performance jitter caused by TLB invalidation and improving address translation efficiency.

[0097] After the model loading phase is completed, the runtime module configures the physical address of the command package of the model to the task scheduler 21. In the subsequent task dispatch phase, the task scheduler 21 directly retrieves the command package from the system memory 30 based on the physical address and parses each command in sequence, without the runtime module needing to intervene line by line.

[0098] When the task scheduler 21 obtains a command packet, it directly accesses the system memory 30 through the physical address without going through SMMU translation, thereby improving the instruction retrieval speed.

[0099] refer to Figure 4 After the model is loaded, when the loaded model needs to perform a task, perform the following steps:

[0100] S201, runtime module 40 sends tasks and task descriptors to NPU20.

[0101] When the runtime module 40 issues tasks to the computing core 22, it also configures a task descriptor. This task descriptor contains at least the following key information: address slot base address and instruction stream base address. The address slot base address is the starting physical address of the target model's address slot in system memory 30. The instruction stream base address is the starting address of the target model's NPU instruction stream in system memory 30.

[0102] The NPU20 contains multiple computing cores 22, which can execute tasks of different models in parallel.

[0103] S202, Task scheduler 21 assigns the task to computing core 22.

[0104] S203, the computing core 22 reads the address slot and instruction stream corresponding to the target model from the system memory 30 through the DMA control unit 222.

[0105] Specifically, the DMA control unit 222 reads the address slot and instruction stream corresponding to the target model from the system memory 30 based on the address slot base address and instruction stream base address.

[0106] S204, the computing core 22 stores the read address slots and instruction streams into the internal storage unit 223.

[0107] The processes S201 to S204 described above are called dispatch (task dispatch). Unlike the traditional scheme where the CPU needs to pass tensor address information to the computing core through multiple register write operations, in this scheme, when the runtime module dispatches, the computing core autonomously reads the address slot content from the system memory through direct memory access after the task is issued by the task scheduler 21, thus completing the tensor address acquisition. This reduces the communication overhead between the CPU and the NPU.

[0108] After completing the above operations, the computing core 22 has obtained the address slot information of the currently executing target model and enters the task execution phase, such as... Figure 5 As shown, the specific steps include:

[0109] S301, the computing core 22 retrieves the currently executing instruction from the internal storage unit 223.

[0110] The instruction carries access identifier information for the target tensor, which indicates which tensor the instruction needs to access.

[0111] S302, the computing core 22 determines the target tensor that needs to be accessed based on the acquired instructions.

[0112] S303, the computing core 22 reads the virtual address corresponding to the tensor from the address slot of the internal storage unit 223.

[0113] The internal storage unit caches all the address slot contents of the current model. For dynamic tensors, the virtual addresses stored in the address slots are allocated by the runtime module when the model is loaded and can be updated when the data buffer switches. For static tensors (such as weights and biases), the virtual addresses in the address slots can be pre-allocated and written to the address slots by the compiler during offline compilation, or they can be reallocated by the runtime module when the model is loaded and are not modified after loading.

[0114] S304, computing core 22 initiates a memory access request carrying a virtual address to SMMU10.

[0115] S305 and SMMU10 look up the corresponding physical address in the SMMU page table based on the virtual address carried in the memory access request.

[0116] Specifically, SMMU10 first checks its internal TLB to see if an address mapping entry corresponding to the virtual address exists. The TLB is a high-speed cache within the SMMU used to cache address mapping relationships. In this application, each entry in the TLB may contain an identifier for the corresponding model, used to distinguish address mapping relationships for different models. If a TLB hit occurs, the corresponding physical address is directly obtained. If a TLB miss occurs, the corresponding physical address is searched in the SMMU page table corresponding to the model based on the virtual address. The found address mapping relationship is then associated with the model's identifier and cached in the TLB to accelerate the translation process for subsequent addresses with the same address. In this application, the SMMU page table is fully established during the model loading phase and remains unchanged during model operation; therefore, TLB invalidation is not required, and the cached entries in the TLB remain valid.

[0117] S306 and SMMU10 return the found physical address to computing core 22.

[0118] S307, the computing core 22 accesses the corresponding data in system memory 30 based on the physical address returned by SMMU10.

[0119] After the above steps, the computing core 22 completes the data reading operation for this task and obtains the data required by the instruction. After the data reading is completed, the computing core 22 performs calculation operations on the obtained data to obtain the inference result of the task.

[0120] During the execution of the above tasks, the data required by the computing core 22 to perform the tasks are translated into addresses through the collaborative mechanism of address slots and SMMU page tables. Throughout the process, the SMMU page tables do not need to be modified.

[0121] In this application, the virtual addresses of each tensor are pre-stored in address slots located in system memory 30. When a task is issued, the runtime module 40 does not need to transmit the address information of each tensor to the computing core 22 through multiple register write operations. When executing a task, the computing core 22 reads the contents of the address slots from system memory 30 to internal storage unit 223 via the DMA control unit 222, autonomously obtaining the virtual addresses of each tensor. That is, the transmission of tensor address information is autonomously completed by the NPU hardware, rather than relying on the CPU to transmit it one by one through registers. Therefore, the number of communications between the CPU and NPU is significantly reduced, the task issuance and startup process is simplified, and communication overhead is reduced.

[0122] In existing technologies, each tensor requires a separate address register to store its address information within the NPU. As the number of tensors in the model increases, the number of registers also increases, leading to higher hardware complexity. In this application, however, the virtual addresses of tensors are stored in address slots within system memory 30, rather than in registers within the NPU. The computational core 22 reads the address slot content from system memory 30 into internal storage unit 223 via DMA control unit 222, autonomously obtaining the virtual addresses of each tensor without requiring a separate address register for each tensor. Compared to existing technologies, this application reduces the number of internal address registers within the NPU, resulting in a simpler hardware structure and a smaller chip area.

[0123] During the model loading phase, the SMMU page table already contains the address mappings of all tensors in the model. Since the SMMU page table remains unchanged during runtime, there is no need to perform TLB invalidation operations. The cached entries in the TLB are not cleared, and subsequent memory access requests can reliably hit the TLB. This avoids the increased latency and performance jitter caused by traversing the SMMU page table after a TLB miss due to a TLB invalidation, thus improving address translation efficiency.

[0124] refer to Figure 6 During model execution, when it is necessary to switch the data buffer of a certain quantity, the following steps are performed:

[0125] S401, Runtime module 40 obtains the virtual address corresponding to the switched data buffer.

[0126] S402, the runtime module 40 writes the switched virtual address into the address slot corresponding to the tensor in the system memory 30.

[0127] Specifically, the runtime module 40 runs on the host CPU. When it writes data to the system memory 30, the data is first written to the CPU cache. At this time, the corresponding address slot content in the system memory 30 has not yet been updated. In order for the computing core 22 to read the updated address slot content from the system memory 30, the runtime module 40 performs a cache flush operation (flush_dcache) after completing the address slot write operation, forcibly writing the modified address slot data in the CPU cache back to the system memory 30. After the above operation, the address slot content in the system memory 30 is updated to the virtual address corresponding to the switched data buffer.

[0128] S403, the computing core 22 reloads the address slot of the target model from the system memory 30 to the internal storage unit 223 via the DMA control unit 222.

[0129] The address slots of the reloaded target model will overwrite the original address slots in the internal storage unit 223. That is, after the address slots are reloaded, the virtual address corresponding to the switched data buffer is cached in the internal storage unit 223.

[0130] S404, the computing core 22 obtains the virtual address of the tensor from the internal storage unit 223.

[0131] The virtual address obtained through step S404 is the virtual address of the switched data buffer.

[0132] S405 and computing core 22 initiate a memory access request carrying the switched virtual address to SMMU10.

[0133] S406 and SMMU10 look up the corresponding physical address in the SMMU page table corresponding to the model based on the switched virtual address carried in the memory access request.

[0134] S407 and SMMU10 return the found physical address to computing core 22.

[0135] S408 and computing core 22 access the corresponding data in system memory 30 based on the physical address returned by SMMU10.

[0136] For example, suppose a tensor is currently using data buffer buffer1 with a virtual address of VA1, which is stored in an address slot. When switching to data buffer buffer2 is required, the runtime module 40 obtains the virtual address VA2 corresponding to the switched data buffer buffer2 and writes VA2 into the address slot corresponding to the tensor in system memory 30. The computation core 22 reloads the address slot of the target model from system memory 30 to internal storage unit 223, obtains the switched virtual address VA2, and initiates a memory access request carrying VA2. SMMU 10 selects the SMMU page table corresponding to the model, translates VA2 into the corresponding physical address PA2, and the computation core 22 accesses the data in buffer2 in system memory 30 based on PA2. Since the address mapping relationship corresponding to VA2 has been pre-established in the SMMU page table during the model loading stage, the SMMU page table remains unchanged during the switching process.

[0137] The data buffer switching method provided in this application has at least the following beneficial effects:

[0138] First, switching the data buffer does not modify the SMMU page table, thus avoiding performance jitter caused by TLB failure.

[0139] During the model loading phase, multiple virtual addresses and corresponding physical addresses are allocated for tensors that require data buffer switching, and the address mapping relationship from virtual address to physical address is pre-established in the SMMU page table. When a data buffer needs to be switched, only the virtual address in the address slot needs to be modified to the virtual address corresponding to the switched data buffer. Since the address mapping relationship of all data buffers has been pre-established in the SMMU page table during the model loading phase, the SMMU page table does not need to be modified during the switch. Therefore, there is no need to perform TLB invalidation operations, the cached entries in the TLB remain valid, and subsequent memory access requests can reliably hit the TLB. This avoids the increased latency and performance jitter caused by traversing the SMMU page table after a TLB miss due to a TLB invalidation, thus improving address translation efficiency.

[0140] Second, the data buffer switching latency is low, meeting the requirements of high frame rate real-time scenarios.

[0141] Since the switching operation only involves modifying the virtual address in the address slot (i.e., a single memory write operation), there is no need to perform SMMU demapping and remapping operations, nor TLB invalidation operations. This significantly reduces switching latency from microseconds to nanoseconds. In high-frame-rate real-time inference scenarios, this solution ensures stable and controllable processing time for each frame, meeting real-time requirements.

[0142] Third, there is no need to configure a separate address register for each tensor, which simplifies the hardware structure.

[0143] In existing technologies, the NPU requires a separate address register for each tensor to store its address information. In this application, the virtual address of the tensor is stored in an address slot in system memory, rather than in a register within the NPU. When switching data buffers, only the virtual address in the address slot needs to be modified, eliminating the need to occupy register resources within the NPU, thus simplifying the hardware structure.

[0144] Fourth, the data buffer allocation is flexible and supports switching any number of data buffers.

[0145] During the model loading phase, any number of data buffers can be allocated to tensors that require data buffer switching, based on actual needs, and the address mapping relationship of all data buffers is pre-established in the SMMU page table. During model execution, these data buffers can be freely switched by modifying the virtual address in the address slot. The switching operation is not limited by the number of data buffers, providing high flexibility.

[0146] The address management method and data buffer switching method provided in this application can be applied to scenarios requiring continuous processing of multiple frames of data (such as video stream inference and autonomous driving perception). In these scenarios, the system needs to complete the inference task within each frame, and the input data (such as image frames) changes between frames, requiring frequent switching of the data buffer. For example, in a video stream processing scenario of 30 frames per second, the processing time for each frame is approximately 33 milliseconds, and the latency of switching the data buffer must be controlled within a very short time to meet real-time requirements. With the data buffer switching method provided in this application, the runtime module 40 only needs to rewrite the virtual address in the address slot to complete the data buffer switching, while the SMMU page table remains unchanged. There is no need to perform TLB invalidation operations, avoiding the increase in memory access latency caused by TLB invalidation, thereby ensuring the stability and low latency of address translation in high frame rate scenarios.

[0147] In an optional embodiment, the compiler further divides multiple tensors in the same model into at least one tensor group during the offline compilation phase. Tensors within the same tensor group share a base address. The compiler assigns each tensor an address offset within its respective tensor group. Based on the base address, each tensor determines its virtual address using its respective address offset within its tensor group. The compiler reserves address slots for each tensor group in the model binary file to store the base address of that tensor group.

[0148] During the offline compilation phase, the compiler assigns a group index to each tensor group, which uniquely identifies the tensor group. When generating NPU instructions, the compiler encodes the group index into the instructions that need to access tensors within that tensor group. This allows the NPU to determine the tensor group to be accessed based on the group index and read the base address of that tensor group from the corresponding address slot when executing the instructions.

[0149] In practice, the base addresses of each tensor group form the input parameter table (arg_tensor). Each entry in arg_tensor corresponds to an address slot of a tensor group, which is used to store the base address of that tensor group.

[0150] In one example, suppose a model contains two weight tensors, weight_0 and weight_1, one parameter tensor, param_0, six input tensors (input_0 to input_5), five output tensors (output_0 to output_4), and one activation tensor, act_0. The compiler divides these tensors into six tensor groups (Group0 to Group5), and the base addresses of these six tensor groups form an input parameter table (arg_tensor).

[0151] The arg_tensor contains 6 entries, each corresponding to an address slot in one of the six tensor groups (Groups 0 to 5), and is used to store the base address (base_addr) of each tensor group. For example, the entry[0] corresponds to the address slot of Group 0, the entry[1] corresponds to the address slot of Group 1, and so on.

[0152] Group0 contains three tensors: weight_0, weight_1, and param_0. The compiler allocates a base address of 0x0100_0000 to this tensor group, which is stored in entry[0] of arg_tensor. The compiler allocates the following intra-group address offsets for each tensor:

[0153] weight_0 is 0x0000_0000 (virtual address 0x0100_0000);

[0154] weight_1 is 0x0008_0000 (virtual address 0x0108_0000);

[0155] param_0 is 0x0010_0000 (virtual address 0x0110_0000).

[0156] Group1 contains five input tensors, input_0 to input_4. The compiler allocates a base address of 0x0400_0000 to this tensor group, which is stored in the entry[1] of arg_tensor. The offsets of the addresses within the group allocated by the compiler for each tensor are as follows:

[0157] input_0 is 0x0000_0000 (virtual address 0x0400_0000);

[0158] input_1 is 0x0001_0000 (virtual address 0x0401_0000);

[0159] input_2 is 0x0002_0000 (virtual address 0x0402_0000);

[0160] input_3 is 0x0003_0000 (virtual address 0x0403_0000);

[0161] input_4 is 0x0004_0000 (virtual address 0x0404_0000).

[0162] Group2 contains one input tensor, input_5, which is treated as a separate tensor group. The compiler assigns a base address of 0x0500_0000 to this tensor group, which is stored in the entry[2] of arg_tensor. During model execution, input_5 needs to switch data buffers. When loading the model, the runtime module 40 allocates multiple data buffers for this tensor and establishes a corresponding address mapping entry for each data buffer in the SMMU page table. The base address of input_5 is rewritten by the runtime module 40 during each frame switch to point to the currently used data buffer.

[0163] Group3 contains four output tensors: output_0 to output_2 and output_4. The compiler allocates a base address of 0x0600_0000 to this tensor group, which is stored in the entry[3] of arg_tensor. The offsets of the addresses within the group allocated by the compiler for each tensor are as follows:

[0164] output_0 is 0x0000_0000 (virtual address 0x0600_0000);

[0165] output_1 is 0x0001_0000 (virtual address 0x0601_0000);

[0166] output_2 is 0x0002_0000 (virtual address 0x0602_0000);

[0167] output_4 is 0x0003_0000 (virtual address 0x0603_0000).

[0168] Group4 contains one output tensor, output_3, which is a separate tensor group. The compiler allocates a base address of 0x0700_0000 for this tensor group, which is stored in the entry[4] of arg_tensor. During the model run, output_3 needs to switch data buffers, and its processing method is similar to that of input_5. When loading the model, the runtime module 40 allocates multiple data buffers for this tensor and establishes a corresponding address mapping entry for each data buffer in the SMMU page table. The base address of output_3 is rewritten by the runtime module 40 during each frame switch to point to the currently used data buffer.

[0169] Group5 contains an activation tensor, act_0. The compiler assigns a base address of 0x0800_0000 to this tensor group, which is stored in the entry[5] of arg_tensor. act_0 does not need to switch the data buffer during model execution, and its base address is not modified after being written to arg_tensor when the model is loaded.

[0170] Furthermore, the compiler assigns an update policy to each tensor group, which is recorded in the model binary file. The update policy indicates how the base address of the tensor group is updated during model execution.

[0171] For tensor groups that do not require switching data buffers (such as static weight tensor groups and activation tensor groups), the compiler sets their update strategy to the first strategy (model_load), which means that their base address is not modified after being written to the address slot when the model is loaded.

[0172] For tensor groups that require switching data buffers and use the Buffer pool (such as independent input tensors and independent output tensors), the compiler sets its update strategy to the second strategy (per_frame_rotate), which means that its base address is rotated from the pre-mapped Buffer pool. The runtime module 40 rewrites the base address in the address slot during each frame switch to make it point to the currently used data buffer.

[0173] During the offline compilation phase, the compiler determines the tensor groups that require data buffer switching and configures the buffer pool depth (i.e., the number of data buffers) for each tensor group. The number of data buffers indicates the number of data buffers maintained simultaneously by that tensor group during runtime. The compiler records the number of data buffers in the model binary file. During the model loading phase, the runtime module allocates a corresponding number of virtual addresses and data buffers to the tensors based on the number of data buffers corresponding to the tensors.

[0174] The criteria for dividing tensor groups can be determined based on actual application requirements. For example, they can be divided based on factors such as tensor type, data source, lifecycle, and whether the data buffer needs to be switched during operation.

[0175] For example, fixed parameters such as weight tensors and bias tensors can be grouped into the same tensor group, multiple input tensors from the same data source can be grouped into the same tensor group (such as grouping multiple image input tensors from the same camera into one group), or tensors that need to switch data buffers during operation can be grouped into a separate tensor group.

[0176] Taking autonomous driving scenarios as an example, the model might contain image input tensors from cameras, point cloud data input tensors from radar, and positioning data input tensors from GPS. The compiler can group the image input tensors from cameras into an image input group, the point cloud data input tensors from radar into a radar input group, and the positioning data input tensors from GPS into a GPS input group. Input tensors from different data sources have different update frequencies and data characteristics; grouping them allows for differentiated address management strategies.

[0177] During the model loading phase, the runtime module 40 allocates a contiguous physical address space for each tensor group. This contiguous physical address space is aligned to a preset size, consistent with the large page mapping granularity supported by the SMMU (e.g., 2MB, 4MB, etc.). The size of the physical address space allocated to each tensor group is no less than the total size of the virtual address range of all tensors within that tensor group. The physical address space sizes of different tensor groups can be the same or different, determined by the runtime module 40 during loading based on the virtual address range size of each tensor group. The SMMU page table uses large page mapping, mapping the contiguous virtual address range starting from the initial virtual address of each tensor group to the corresponding contiguous physical address space. With large page mapping, a tensor group only needs one page table entry to cover its entire virtual address range, reducing the number of entries in the SMMU page table and lowering TLB pressure.

[0178] For example, for Group0 with a virtual address range of approximately 1.1MB, runtime module 40 allocates a contiguous physical address space (e.g., 2MB) aligned with the SMMU big page granularity; for Group1 with a virtual address range of approximately 320KB, the same contiguous physical address space with the same alignment granularity is allocated. For tensor groups with a larger virtual address range (e.g., 3MB), runtime module 40 can allocate a 4MB contiguous physical address space. Regardless of the allocated space size, alignment is based on the SMMU big page granularity to meet the requirements of SMMU big page mapping.

[0179] For tensor groups that do not require data buffer switching (such as Group0, Group1, Group3, and Group5), runtime module 40 allocates a physical address-contiguous data buffer for the tensor group, writes the base address of the tensor group into the corresponding address slot in arg_tensor, and creates an address mapping entry in the SMMU page table to map the continuous virtual address range starting from the base address to the continuous physical address space. Therefore, only one address mapping entry needs to be created for a tensor group.

[0180] Taking Group0 (containing three tensors: weight_0, weight_1, and param_0) as an example, this tensor group has only one base address, 0x0100_0000, and its starting virtual address is also 0x0100_0000. The runtime module 40 allocates a contiguous physical address space for this tensor group and creates an address mapping entry in the SMMU page table, mapping the contiguous virtual address space starting at virtual address 0x0100_0000 to this contiguous physical address space. This mapping entry covers the contiguous virtual address range of the three tensors: weight_0, weight_1, and param_0. Without tensor group partitioning, each of the three tensors would require an independent address mapping entry, resulting in three mapping entries. With tensor group partitioning, the three tensors are merged into one mapping entry, reducing the number of mapping entries and lowering the storage overhead of the SMMU page table and the pressure on the TLB.

[0181] Similarly, Group1 (containing five tensors from input_0 to input_4) has only one mapping entry in the SMMU page table, but five entries are needed if grouping is not used; Group3 (containing four tensors from output_0 to output_2 and output_4) has only one mapping entry, but four entries are needed if grouping is not used; Group5 (containing one tensor, act_0) has one mapping entry.

[0182] In the example above, without tensor group partitioning, the 13 tensors Group0, Group1, Group3, and Group5 require 13 independent address mapping entries. After tensor group partitioning, these four tensors are combined into 4 address mapping entries, reducing the number of mapping entries from 13 to 4, which significantly reduces the storage overhead of the SMMU page table and the pressure on the TLB.

[0183] For tensor groups that require switching data buffers (such as Group2 and Group4), the runtime module 40 first reads the attributes of the data buffers to be switched for each tensor group in the model binary file to determine the buffer pool depth for each tensor group. Then, the runtime module 40 allocates multiple data buffers according to the determined buffer pool depth, with each data buffer corresponding to a physical address, and establishes an address mapping relationship between each virtual address and its corresponding physical address in the SMMU page table. All address mapping relationships for data buffers are established during the model loading phase, and no new mappings are added or deleted during runtime.

[0184] There are two ways to determine the depth of the buffer pool: Method 1, the runtime module 40 directly uses the number of data buffers recorded by the compiler; Method 2, the runtime module 40 determines the depth of the buffer pool based on the current system memory resource status.

[0185] Taking Group2 as an example, the compiler records that there are 6 data buffers. During the loading phase, the runtime module 40 allocates 6 data buffers Buffer1 to Buffer6 for input_5, with physical addresses PA1 to PA6 respectively. At the same time, it allocates 6 virtual addresses and establishes 6 address mapping relationships.

[0186] In this way, during the model loading phase, the address mapping relationship from virtual address to physical address for all data buffers in the tensor group that require data buffer switching is pre-established in the SMMU page table. Since the SMMU page table is fully determined during the model loading phase, during model execution, when a data buffer needs to be switched, the runtime module 40 only needs to rewrite the base address in the address slot (e.g., rewrite the virtual address of the currently used data buffer to the virtual address of another data buffer), allowing the NPU to switch the data accessed from one data buffer to another, without modifying the address mapping relationship in the SMMU page table. Therefore, the SMMU page table remains unchanged during model execution, eliminating the need for TLB invalidation operations, avoiding performance jitter caused by TLB invalidation, and improving address translation efficiency.

[0187] It should be noted that when the compiler partitions tensor groups, it can group tensors that require unified data buffer switching into the same tensor group, and group tensors that require independent data buffer switching separately or into the same tensor group with other tensors having the same switching strategy. For tensor groups that uniformly switch data buffers on a tensor group basis, their SMMU page table mapping method is the same as that of tensor groups that do not require data buffer switching; that is, all tensors within the same tensor group share the same address mapping entry.

[0188] For example, suppose a model contains five input tensors, input_0 to input_4, all from the same data source. These tensors are updated simultaneously with each frame, requiring a unified switching of the data buffer. The compiler divides these five input tensors into a tensor group (input_group), which has a base address stored in a corresponding address slot. During the model loading phase, the runtime module allocates a contiguous physical address space for this tensor group and creates an address mapping entry in the SMMU page table, covering the contiguous virtual address range of all input tensors in the group. During model execution, each frame switch modifies the base address of this tensor group, making it point to the new contiguous physical address space. Since this tensor group corresponds to only one address mapping entry, and this mapping covers the entire virtual address range of the group, the SMMU page table remains unchanged during switching.

[0189] Through the above-described implementation of tensor group partitioning, this application achieves at least the following technical effects:

[0190] First, merging multiple tensors into a single tensor group and sharing the same SMMU page table mapping entry significantly reduces the number of mapping entries in the SMMU page table, thereby reducing the storage overhead of the SMMU page table and the pressure on the TLB.

[0191] Second, all tensors within the same tensor group share a base address. The runtime module 40 only needs to write the base address into the address slot to complete the configuration of the virtual addresses of all tensors in the tensor group, simplifying the address slot writing operation.

[0192] Third, by configuring update strategies for tensor groups (fixed base address or replaceable base address), differentiated processing of different tensor groups is achieved, which ensures the stability of static data and meets the flexibility requirements of dynamic data.

[0193] During the task assignment and model startup phase, the computing core 22 reads the address slot corresponding to the target model from the system memory 30 through the DMA control unit 222 and stores it in the internal storage unit 223. The address slot stores the base address of each tensor group.

[0194] For the intra-group address offset, it is allocated and fixed by the compiler for each tensor during the offline compilation phase and encoded in the instruction stream of the model. Therefore, the intra-group address offset is loaded into the internal storage unit 223 along with the instruction stream, and is decoded from the currently executing instruction by the address generation unit 224 during the task execution phase.

[0195] During the task execution phase, the computing core 22 obtains the currently executing instruction, determines the address slot corresponding to the tensor group that needs to be accessed based on the group index carried in the instruction, reads the base address of the tensor group from the address slot in the internal storage unit 223, and adds the read base address to the address offset within the tensor group through the address generation unit 224 to generate a complete virtual address.

[0196] like Figure 7 As shown, the address generation unit 224 includes a selector 2241 and an adder 2242. The selector 2241 has multiple input terminals, each connected to multiple entries of the address slots stored in the internal storage unit 223. Each entry corresponds to a tensor group, used to select the corresponding entry from the multiple entries under the control of the group index carried in the instruction, and output the base address stored in that entry. The adder 2242 adds the base address output by the selector 2241 to the intra-group address offset carried in the instruction to generate a complete virtual address.

[0197] Computing core 22 initiates a memory access request carrying a virtual address to SMMU10.

[0198] SMMU10 looks up the corresponding physical address in the SMMU page table based on the virtual address carried in the memory access request, and returns the found physical address to computing core 22.

[0199] The computing core 22 accesses the corresponding data in the system memory 30 based on the physical address.

[0200] During the data buffer switching process, for a tensor group that switches data buffers uniformly on a tensor group basis, the runtime module 40 replaces the base address in the address slot corresponding to the tensor group to which the tensor belongs in the system memory 30 with the base address corresponding to the data buffer after the switch, thereby realizing the synchronous switching of data buffers of all tensors in the tensor group.

[0201] Since the address mapping relationships for all data buffers of each tensor within a tensor group are pre-established in the SMMU page table during the model loading phase, switching only requires modifying the base address in the address slot; the SMMU page table does not need any modification. Therefore, the SMMU does not need to perform a TLB invalidation operation. The cached entries in the TLB remain valid, and address translation can reliably hit the TLB, avoiding performance jitter caused by TLB invalidation and improving address translation efficiency. Furthermore, since the switching operation only involves modifying a single address slot, there is no need to modify the address information for each tensor within the group separately, further reducing switching overhead and improving scheduling efficiency in scenarios involving unified switching of multiple tensors.

[0202] In this embodiment, apart from the tensor group partitioning steps mentioned above, other operations in the offline compilation stage, model loading stage, task dispatch stage, and task execution stage (including the establishment of SMMU page tables, the addition of address mapping relationships, etc.) are the same as in the previous embodiments, and can be referred to the relevant descriptions above for details; the operations in the data buffer switching stage are also the same as in the previous embodiments, and will not be repeated here.

[0203] This application also provides a chip, such as... Figure 1 As shown, the chip includes an SMMU10 and multiple computing cores 22.

[0204] SMMU10 is configured with an SMMU page table, which contains address mapping entries for each tensor in the model. These address mapping entries correspond to the mapping relationship between the virtual and physical addresses of each tensor's data buffer. Tensors that require switching data buffers may have multiple data buffers, each corresponding to one address mapping entry. The SMMU page table is determined during model loading and remains unchanged after model loading.

[0205] Each computing core 22 is configured with an internal storage unit 223, which is used to cache the address slots corresponding to tensors. The address slots are used to store the virtual address of the currently used data buffer.

[0206] The computing core 22 is used to replace the virtual address in the address slot corresponding to the tensor in the internal storage unit 223 with the virtual address corresponding to the switched data buffer when the data buffer of the tensor needs to be switched during the execution of a task, so as to access the corresponding data buffer based on the switched virtual address.

[0207] The chip provided in this application embodiment has an address mapping relationship for all data buffers pre-established in the SMMU page table during model loading. When switching data buffers, only the virtual address of the address slot in the internal storage unit needs to be modified, and the SMMU page table remains unchanged. The chip does not need to perform TLB invalidation operation during operation, thus improving address translation efficiency and making the chip performance more stable.

[0208] This application also provides a data buffer switching system, such as... Figure 1 As shown, the system includes: the chip described in the foregoing embodiment, system memory 30, and runtime module 40.

[0209] System memory 30 is connected to the chip and includes multiple data buffers, which store the address slots. A data buffer is a contiguous storage area in system memory used to store tensor data, and each data buffer has a defined physical address. The same tensor can correspond to multiple data buffers, and pipelined data processing is achieved by switching between different data buffers during model execution.

[0210] Runtime module 40 runs on the host central processing unit (CPU) and is a software / firmware layer component in the system.

[0211] During the model loading phase, the runtime module 40 allocates multiple virtual addresses for tensors that need to switch data buffers during the same model operation, allocates a physical address corresponding to a data buffer for each virtual address, establishes address mapping entries from the multiple virtual addresses to the corresponding physical addresses in the SMMU page table corresponding to the model, and writes the virtual address currently used by each tensor into the corresponding address slot in the system memory 30.

[0212] During the task dispatch phase, the runtime module 40 stores the address slot of the model into the internal storage unit 223 of the computing core 22 of the chip.

[0213] During the task execution phase, when it is necessary to switch the data buffer of the target tensor, the runtime module 40 replaces the virtual address in the address slot corresponding to the target tensor in the system memory 30 with the virtual address corresponding to the switched data buffer.

[0214] The data buffer switching system provided in this application pre-establishes the address mapping relationship of all data buffers during the model loading stage through the runtime module. During switching, only the virtual address in the address slot needs to be modified, without modifying the SMMU page table. This achieves low-latency, jitter-free data buffer switching, meeting the needs of high frame rate real-time scenarios.

[0215] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the descriptions of the above embodiments are only for the purpose of helping to understand the methods of the embodiments of this application and should not be construed as limiting the embodiments of this application. Any changes or substitutions that can be made by those skilled in the art should be covered within the protection scope of the embodiments of this application.

Claims

1. A data buffer switching method, applied to a system including a System Memory Management Unit (SMMU), a runtime module, and a hardware accelerator, characterized in that, The method includes: During the model loading phase, the runtime module allocates multiple virtual addresses for tensors that need to switch data buffers during the same model execution, allocates a physical address corresponding to a data buffer for each virtual address, establishes address mapping entries from the multiple virtual addresses to the corresponding physical addresses in the SMMU page table corresponding to the model, and writes the virtual address currently used by each tensor into the corresponding address slot in the system memory; the SMMU page table is determined during model loading and remains unchanged after model loading; During the task dispatch phase, the runtime module stores the address slot of the model into the internal storage unit of the computing core of the hardware accelerator. During the task execution phase, when it is necessary to switch the data buffer of the target tensor, the runtime module replaces the virtual address in the address slot corresponding to the target tensor in the system memory with the virtual address corresponding to the switched data buffer. After the virtual address in the address slot of the target tensor is replaced with the virtual address corresponding to the switched data buffer, the computing core retrieves the address slot of the model from the system memory and stores it in the internal storage unit to access the corresponding data buffer based on the switched virtual address.

2. The method according to claim 1, characterized in that, The address slots are storage spaces reserved by the compiler for each tensor in the model binary file during the offline compilation stage. The runtime module locates the address slots of each tensor according to the offset of each tensor in the model binary file and writes the virtual address of each tensor into the corresponding address slot.

3. The method according to claim 1, characterized in that, The address slots of each tensor in the same model are aggregated into an input parameter table, which is located at a fixed virtual address. Each entry in the input parameter table corresponds to an address slot. The runtime module writes the virtual address into the corresponding entry in the input parameter table according to the entry index of each tensor.

4. The method according to claim 1, characterized in that, The number of data buffers corresponding to the tensors is configured during the offline compilation stage and recorded in the model binary file; The method further includes: During the model loading phase, the runtime module allocates a corresponding number of virtual addresses and data buffers to the tensor based on the number of data buffers.

5. The method according to any one of claims 1 to 4, characterized in that, Multiple tensors in the same model are divided into at least one tensor group, each tensor group corresponds to an address slot, and tensors within the same tensor group share the same base address, which is stored in the address slot corresponding to the tensor group; each tensor within the same tensor group determines its own virtual address based on the base address and its own address offset within its tensor group.

6. The method according to claim 5, characterized in that, For tensor groups that do not require switching data buffers, in the SMMU page table, tensors within the same tensor group share the same address mapping entry; different tensor groups correspond to different address mapping entries.

7. The method according to claim 5, characterized in that, For a tensor group that switches data buffers uniformly on a tensor group basis, in the SMMU page table, tensors within the same tensor group share the same address mapping entry; different tensor groups correspond to different address mapping entries.

8. The method according to claim 5, characterized in that, The runtime module replaces the virtual address in the address slot corresponding to the target tensor in the system memory with the virtual address corresponding to the switched data buffer, including: The runtime module replaces the base address in the address slot corresponding to the tensor group to which the target tensor belongs in the system memory with the base address corresponding to the switched data buffer.

9. A chip, characterized in that, include: The System Memory Management Unit (SMMU) is configured with an SMMU page table. The SMMU page table contains address mapping entries for each tensor of the model. The address mapping entries correspond to the mapping relationship between the virtual address and the physical address of the data buffer of each tensor. Among them, tensors that need to switch data buffers have multiple data buffers, and each data buffer corresponds to an address mapping entry. The SMMU page table is determined when the model is loaded and remains unchanged after the model is loaded. Multiple computing cores, each of which is configured with an internal storage unit, the internal storage unit being used to cache the address slots corresponding to the tensors, the address slots being used to store the virtual addresses of the currently used data buffers; The computing core is used to replace the virtual address in the address slot corresponding to the tensor in the internal storage unit with the virtual address corresponding to the switched data buffer when the data buffer of the tensor needs to be switched during task execution, so as to access the corresponding data buffer based on the switched virtual address.

10. A data buffer switching system, characterized in that, include: The chip as described in claim 9; The system memory, connected to the chip, includes multiple data buffers and stores the address slots; The runtime module is used to allocate multiple virtual addresses for tensors that need to switch data buffers during the operation of the same model, allocate a physical address corresponding to a data buffer for each virtual address, establish address mapping entries from the multiple virtual addresses to the corresponding physical addresses in the SMMU page table corresponding to the model, and write the virtual address currently used by each tensor into the corresponding address slot in the system memory. The runtime module is also used to store the address slot of the model into the internal storage unit of the computing core of the chip during the task dispatch phase. The runtime module is also used to replace the virtual address in the address slot corresponding to the target tensor in the system memory with the virtual address corresponding to the switched data buffer when the data buffer of the target tensor needs to be switched during the task execution phase.

Images