Address management method in multi-model scheduling, chip and system

Abstract

The application relates to the technical field of integrated circuits, and discloses a method for address management in multi-model scheduling, a chip and a system. In the model loading stage, a unique SSID and an independent SMMU page table are allocated to each model, the page table remains unchanged after the model is loaded, and the virtual addresses of each tensor are stored in a corresponding address slot. In the task dispatching stage, the SSID of the target model is configured to the computing core of the hardware accelerator, and the address slot is stored in the internal storage unit of the computing core. In the task execution stage, the computing core obtains the SSID of the target model, reads the virtual address from the address slot in the internal storage unit, initiates a memory access request carrying the SSID and the virtual address, and the SMMU selects the corresponding page table according to the SSID to complete address translation. The application simplifies the task dispatching process, reduces the model switching overhead, and realizes multi-model hardware-level address space isolation.

Description

Technical Field

[0001] This application relates to the field of integrated circuit technology, and in particular to an address management method, chip, and system in multi-model scheduling. Background Technology

[0002] With the rapid development of artificial intelligence technology, hardware accelerators (such as Neural Processing Units (NPUs), Tensor Processing Units (TPUs), and General Purpose Graphics Processing Units (GPGPUs)) are widely used in various computing systems as hardware units specifically designed to accelerate specific computing tasks. In scenarios such as autonomous driving, intelligent monitoring, and mobile terminals, systems often need to run multiple neural network models simultaneously, such as executing object detection models, semantic segmentation models, and behavior recognition models concurrently.

[0003] In scenarios involving concurrent execution of multiple models, hardware accelerators need to efficiently schedule and switch between multiple models. However, existing technologies mainly suffer from the following problems:

[0004] First, task submission requires a large number of registers and incurs significant communication overhead.

[0005] To enable multi-model switching, existing hardware accelerators require multiple registers (one 64-bit register for each dynamic tensor) to store address information during task execution. When submitting an inference task, the CPU needs to perform multiple register write operations per frame to pass the address information of each tensor to the hardware accelerator. Since the model contains a large number of tensors, each tensor requires one or more registers, resulting in a long communication path between the CPU and the hardware accelerator and low task submission efficiency.

[0006] Second, context switching overhead is large when switching between multiple models.

[0007] In existing solutions, when a system needs to switch between multiple models, the hardware accelerator needs to save the complete register context of the currently executing model and restore the register context of the model to be executed, typically involving hundreds of bytes of data. Simultaneously, since each model uses its own SMMU page table, the SMMU needs to perform a TLB invalidation operation to clear the address mapping cache of the old model when switching models, which itself also takes time. The combined effect of saving and restoring the register context and the TLB invalidation operation results in model switching latency reaching tens of microseconds, making it difficult to meet the high-speed processing requirements of real-time concurrent multi-model scenarios.

[0008] Third, the compiler and runtime modules are tightly coupled, lacking an efficient address space isolation mechanism.

[0009] In existing solutions, compiler artifacts are typically bound to specific virtual address layouts. The compiler needs to understand the address allocation of the runtime environment, resulting in tight coupling between the compilation and loading phases. Furthermore, in scenarios where multiple models share the same hardware accelerator, the lack of hardware-level virtual address space isolation mechanisms means different models can only be allocated non-overlapping virtual address ranges through software. Software must perform cross-model address conflict checks during model loading. As the number of concurrently loaded models increases, the complexity of address conflict checking increases significantly. During runtime, if a model accesses an address outside its allocated range due to a program error or abnormal behavior, it may incorrectly access data from other models, leading to data corruption or system crashes. Due to the lack of hardware-level access isolation protection, such anomalies cannot be detected and blocked in a timely manner, posing a data security risk. Summary of the Invention

[0010] To address the aforementioned technical problems, the technical solution adopted in this application is as follows:

[0011] In a first aspect, an embodiment of this application provides an address management method in multi-model scheduling, applied to a system including a system memory management unit (SMMU), a runtime module, and a hardware accelerator, characterized in that the method includes:

[0012] During the model loading phase, the runtime module assigns an SSID to the loaded model, establishes an SMMU page table corresponding to the SSID in the SMMU, and stores the address slots of the model. Each loaded model in the system has a different SSID, and each SSID corresponds to an independent SMMU page table. The SMMU page table contains the address mapping relationship from the virtual address to the physical address of each tensor in the corresponding model. The SMMU page table is determined when the model is loaded and remains unchanged after the model is loaded. The address slots are used to store the virtual addresses of each tensor in the corresponding model.

[0013] During the task dispatch phase, the runtime module configures the SSID of the target model to the computing core of the hardware accelerator and stores the address slot of the target model into the internal storage unit of the computing core.

[0014] During the task execution phase, the computing core obtains the SSID of the target model, reads the virtual address from the address slot in the internal storage unit, and initiates a memory access request carrying the SSID of the target model and the virtual address.

[0015] The SMMU selects the corresponding SMMU page table based on the SSID carried in the memory access request, and translates the virtual address in the memory access request into the corresponding physical address based on the selected SMMU page table, and returns it to the computing core.

[0016] Secondly, one embodiment of this application provides a chip, comprising:

[0017] The System Memory Management Unit (SMMU) is configured with multiple SMMU page tables. Each SMMU page table corresponds to an SSID, and each SSID corresponds to a model. The SMMU is used to select the corresponding SMMU page table based on the SSID carried in the received memory access request, translate the virtual address of each tensor in the model into a physical address, and return the physical address to the computing core.

[0018] Multiple computing cores, each of which is configured with an SSID register and an internal storage unit;

[0019] The SSID register is used to store the SSID of the target model currently running on the computing core, and the SSID in the SSID register is used to mark the memory access request sent by the computing core to the SMMU;

[0020] The internal storage unit is used to cache the address slots corresponding to each tensor in the target model. The address slots are used to store the virtual addresses of the corresponding tensors for use when generating memory access requests.

[0021] The SMMU page table of the target model is determined during model loading and remains unchanged after model loading.

[0022] Thirdly, one embodiment of this application provides a multi-model scheduling system, including:

[0023] The chip as described in the second aspect;

[0024] System memory, connected to the chip, is used to store the address slots corresponding to each model;

[0025] The runtime module is used to assign an SSID to the model during the model loading phase, establish an SMMU page table corresponding to the SSID in the SMMU of the chip, and save the address slot of the model; and during the task dispatch phase, configure the SSID of the target model to the computing core of the chip, and store the address slot of the target model into the internal storage unit of the computing core.

[0026] The technical solutions provided in this application have at least the following beneficial effects:

[0027] First, the task assignment and initiation process is simple.

[0028] In this application, the virtual addresses of each tensor are pre-stored in address slots located in system memory. During task dispatch, the runtime module only needs to write the SSID of the target model into the SSID register of the computing core, without needing to pass the address information of each tensor to the computing core through multiple register write operations. When executing a task, the computing core reads the contents of the address slots from system memory into its internal storage unit via the DMA control unit, autonomously obtaining the virtual addresses of each tensor. That is, the transfer of tensor address information is autonomously handled by the hardware accelerator, rather than relying on the CPU to transfer it one by one through registers. Therefore, the number of communications between the CPU and the hardware accelerator is significantly reduced, the task dispatch and startup process is simplified, and communication overhead is lowered.

[0029] Second, the hardware structure is simple to implement.

[0030] In existing technologies, hardware accelerators require a separate address register for each tensor to store its address information. As the number of tensors in the model increases, the number of registers also increases, leading to higher hardware complexity. In this application, however, the virtual addresses of tensors are stored in address slots in system memory, rather than in registers within the hardware accelerator. The computational core reads the address slot contents from system memory into its internal storage unit via a DMA control unit, autonomously obtaining the virtual addresses of each tensor. Therefore, each computational core only needs one SSID register to store the SSID of the currently executing model, eliminating the need for a separate address register for each tensor. Compared to existing technologies, this application reduces the number of address registers within the hardware accelerator, resulting in a simpler hardware structure and a smaller chip area.

[0031] Third, the address translation is highly efficient.

[0032] During the model loading phase, the SMMU page table already contains the address mappings of all tensors in the model. Since the SMMU page table remains unchanged during runtime, there is no need to perform TLB invalidation operations. The cached entries in the TLB are not cleared, and subsequent memory access requests can reliably hit the TLB. This avoids the increased latency and performance jitter caused by traversing the SMMU page table after a TLB miss due to a TLB invalidation, thus improving address translation efficiency.

[0033] Fourth, multi-model address space isolation.

[0034] Each model has a unique SSID and an independent SMMU page table. The virtual addresses of each model are independent and do not affect each other. Different models can use the same or partially the same virtual addresses. The SMMU distinguishes the SMMU page tables corresponding to different models based on the SSID carried in the memory access request, achieving hardware-level isolation of the address space. Because each model has an independent SMMU page table, when allocating virtual addresses for a model, the compiler does not need to consider what virtual addresses other models use, nor does it need to adopt different virtual address allocation rules for each model; it can reuse the same virtual address allocation rules. In this way, the compilation phase and the model loading phase are decoupled, and the compiler focuses on the compilation optimization of the model itself without being aware of the model deployment situation in the runtime environment. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the hardware architecture of a multi-model scheduling system provided in an embodiment of this application;

[0037] Figure 2 A flowchart illustrating an address management method in multi-model scheduling provided in an embodiment of this application;

[0038] Figure 3 A flowchart illustrating the address management method in multi-model scheduling provided in an embodiment of this application during the model loading stage;

[0039] Figure 4 A flowchart illustrating the address management method in multi-model scheduling provided in an embodiment of this application during the task dispatch phase;

[0040] Figure 5 Another flowchart illustrating the address management method in multi-model scheduling provided in an embodiment of this application during the task execution phase;

[0041] Figure 6 A flowchart illustrating the address management method in multi-model scheduling provided in an embodiment of this application during the data buffer switching stage;

[0042] Figure 7 This is a schematic diagram of the internal structure of an address generation unit provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0044] The address management method provided in this application can be applied to systems that include a system memory management unit (SMMU) and hardware accelerators. Hardware accelerators include, but are not limited to, neural network processors (NPUs), tensor processing units (TPUs), general-purpose graphics processing units (GPGPUs), and video codecs / decoders. The following detailed description uses an NPU as an example to illustrate the technical solution of this application; however, those skilled in the art should understand that this description does not constitute a limitation on the scope of protection of this application.

[0045] Tensor: A tensor is the basic organizational form of data in a neural network model, which can be understood as a multidimensional array. During model inference, all data involved in the computation exists in the form of tensors. Specific types of tensors in a neural network model include, but are not limited to: input tensors (such as image data input during inference), output tensors (such as classification results, bounding box coordinates, etc. output during inference), weight tensors (weight parameters of each layer of the model), bias tensors (bias parameters of each layer of the model), and intermediate activation tensors (intermediate results generated by calculations in each layer of the model).

[0046] To improve translation speed during address translation by the SMMU, a Translation Lookaside Buffer (TLB) is configured internally. The TLB caches recently used address mapping entries from virtual addresses to physical addresses. When the SMMU receives a translation request for a virtual address, it first checks the TLB to see if a corresponding address mapping entry exists. If it does, the corresponding physical address can be obtained directly without accessing the SMMU page table; if it does not exist, the SMMU page table needs to be traversed to search for the entry, and the found address mapping entry is cached in the TLB.

[0047] When an address mapping entry in the SMMU page table is modified, the old mapping entry cached in the TLB becomes inconsistent with the modified page table content. In this case, a TLB invalidation operation needs to be performed to clear the old mapping entry from the TLB. After the TLB invalidation operation, subsequent memory access requests will not be able to hit the TLB, requiring a re-traversal of the SMMU page table for address translation until the new address mapping entry is re-cached in the TLB. During this period, the address translation latency increases, and this latency is sudden and unstable, resulting in performance jitter.

[0048] The multi-model scheduling system provided in this application embodiment is used to load, schedule, execute, and switch multiple neural network models in a computing system that includes a neural network processor (NPU) and a system memory management unit (SMMU). This system can be applied to computing scenarios that require the simultaneous operation of multiple neural network models, such as simultaneously running object detection models, semantic segmentation models, and lane recognition models in autonomous driving scenarios, or simultaneously running face recognition models, voice wake-up models, and image classification models in mobile terminal scenarios.

[0049] like Figure 1 As shown, the multi-model scheduling system provided in this application embodiment includes: a system memory management unit (SMMU) 10, a neural network processor (NPU) 20, system memory 30, and a runtime module 40.

[0050] The runtime module 40 runs on the host central processing unit (CPU) and is a software / firmware layer component in the multi-model scheduling system. It is used to manage the models and hardware resources in the multi-model scheduling system, issue tasks to the NPU20, and configure page tables to the SMMU10, as well as perform various operations that require software execution.

[0051] The system memory 30 (e.g., DDR memory) is used to store the data required for the operation of the multi-model scheduling system. The data includes, but is not limited to, the model binary files of each model and the tensor data (including input tensors, output tensors, weight tensors, bias tensors, etc.) that need to be read and written during the operation of each model.

[0052] SMMU10 is the system memory management unit, configured with multiple SMMU page tables. Each SMMU page table uniquely corresponds to a Substream ID (SSID), and each SSID uniquely corresponds to a model. SMMU10 is used to receive memory access requests initiated by NPU20 and perform address translation based on the SSID and virtual address carried in the memory access request. For example, Figure 1The system has loaded four models (Model 1, Model 2, Model 3, and Model 4), corresponding to SSID=1, SSID=2, SSID=3, and SSID=4 respectively. SMMU10 is configured with four SMMU page tables (SMMU page table 1, SMMU page table 2, SMMU page table 3, and SMMU page table 4) corresponding one-to-one with the above four SSIDs. Each SMMU page table stores the address mapping relationship from virtual addresses to physical addresses of the tensors of the corresponding model. When the computing core 22 within NPU20 initiates a memory access request, the request carries the SSID and virtual address. SMMU10 selects the corresponding SMMU page table based on the SSID carried in the request, looks up the corresponding physical address in that SMMU page table based on the virtual address, and returns the physical address to the computing core 22. Thus, the virtual address spaces of different models are isolated through SSIDs.

[0053] When the multi-model scheduling system loads a model, the runtime module 40 reads the model binary file from non-volatile storage into the system memory 30, assigns a unique SSID to the model, establishes an SMMU page table corresponding to the SSID in SMMU10, and establishes address mapping entries from virtual addresses to physical addresses for each tensor of the model in the SMMU page table.

[0054] When the computing core 22 in NPU20 sends a memory access request carrying the SSID and virtual address to SMMU10, SMMU10 looks up the corresponding SMMU page table based on the SSID carried in the request, looks up the corresponding physical address in the SMMU page table based on the virtual address, and returns the physical address to the computing core 22.

[0055] The NPU20 includes a Task Scheduler (TSM) 21 and multiple compute cores 22 (e.g., ...). Figure 1 (Core0~Core3 in the system). The task scheduler 21 is used to receive tasks issued by the runtime module 40 and allocate the tasks to the corresponding computing cores 22. The computing cores 22 are used to execute the tasks allocated by the task scheduler 21, and when they need to obtain the data required to execute the task, they send a memory access request carrying the SSID and virtual address to the SMMU 10, receive the physical address returned by the SMMU 10, and obtain the required data from the system memory 30 or store the data into the system memory 30 based on the physical address.

[0056] Each computing core 22 is equipped with an SSID register 221, a DMA (Direct Memory Access) control unit 222, an internal storage unit 223, and an address generation unit 224.

[0057] SSID register 221 is used to store the SSID of the target model currently running on computing core 22.

[0058] The DMA control unit 222 is used to read data required by the computing core to perform tasks from the system memory 30, or to write the processing results of the computing core into the system memory 30. The data includes, but is not limited to, address slot contents, model instructions, weight data, and input data.

[0059] Internal storage unit 223 is used to cache data read from system memory 30 by DMA control unit 222.

[0060] Address generation unit 224 is used to generate the virtual address required for memory access requests.

[0061] This application provides an address management method applied to a system including a System Memory Management Unit (SMMU), a runtime module, and a hardware accelerator. (See reference...) Figure 2 This address management method includes at least the following steps:

[0062] S100. During the model loading phase, the runtime module assigns an SSID to the loaded model, establishes an SMMU page table corresponding to the SSID in the SMMU, and saves the model's address slot.

[0063] Each loaded model in the system has a different SSID, and each SSID corresponds to an independent SMMU page table. The SMMU page table contains the address mapping relationship from the virtual address to the physical address of each tensor in the corresponding model. The SMMU page table is determined when the model is loaded and remains unchanged after the model is loaded. The address slots are used to store the virtual address of each tensor in the corresponding model.

[0064] S200. During the task dispatch phase, the runtime module configures the SSID of the target model to the computing core of the hardware accelerator and stores the address slot of the target model into the internal storage unit of the computing core.

[0065] S300. During the task execution phase, the computing core obtains the SSID of the target model, reads the virtual address from the address slot in the internal storage unit, and initiates a memory access request carrying the SSID and virtual address of the target model.

[0066] The S400 and SMMU select the corresponding SMMU page table based on the SSID carried in the memory access request, and translate the virtual address in the memory access request into the corresponding physical address based on the selected SMMU page table, and return it to the computing core.

[0067] Through the above steps, only the SSID needs to be configured during task dispatch. The tensor address information is carried by the address slot and obtained autonomously by the computing core. The SMMU page table does not need to be modified, thus avoiding TLB failure operations and realizing ultra-lightweight task dispatch and address translation.

[0068] The following combination Figures 3 to 7 The specific implementation methods for each of the above stages are described in detail.

[0069] Before the multi-model scheduling system loads the model, the neural network model is first compiled offline by the compiler to generate the corresponding model binary file (model.bin), which is then stored in a non-volatile storage medium (such as hard disk or flash memory). The model binary file contains a command bundle, an NPU-executable instruction stream, weight data, and address slots for each tensor in the model.

[0070] The command package is used by the task scheduler 21 to perform task scheduling. It contains a sequence of commands for all computation partitions of the model. Each command contains a task type, task identifier, dependency, task priority, and sub-stream identifier (SSID) placeholder.

[0071] The instruction stream is used by the NPU computing core to execute computational tasks. It contains a sequence of instructions that the NPU can execute (such as convolution instructions, pooling instructions, activation instructions, DMA instructions, etc.) and weight data.

[0072] Address slots are storage spaces reserved by the compiler for each tensor in the model binary file, used to store the virtual address of that tensor during model runtime. There are two ways to store address slots.

[0073] In one implementation, address slots are distributed throughout the model binary file, with each tensor corresponding to one address slot. During the offline compilation phase, the compiler reserves address slots for each tensor and records the offset of each address slot within the model binary file. The runtime module locates the address slot for each tensor based on its corresponding address slot offset and writes the virtual address into the corresponding address slot.

[0074] In another implementation, address slots are aggregated into an input parameter table. The input parameter table resides at a fixed virtual address, and each entry in the table corresponds to an address slot. During the offline compilation phase, the compiler assigns entry indices to each tensor, and the runtime module writes the virtual address into the corresponding entry in the input parameter table based on the entry index of each tensor. Because the input parameter table resides at a fixed virtual address, the hardware can directly read the contents of the input parameter table at this fixed virtual address without requiring software configuration of the base address register.

[0075] During the operation of the multi-model scheduling system, when a model needs to be loaded, the runtime module 40 performs the model loading operation, referencing... Figure 3Specifically, it includes the following steps:

[0076] S101, The runtime module 40 obtains the model binary file of the model and stores it in the system memory 30.

[0077] Specifically, the runtime module 40 reads the model binary file from non-volatile storage medium into system memory 30. The model binary file contains command packages, NPU-executable instruction streams, weight data, and address slots reserved by the compiler for each tensor.

[0078] In this application, a tensor is a data unit that participates in the calculation in the model, and each tensor corresponds to one or more data buffers in the system memory 30. The data buffer is a contiguous storage area in the system memory used to store tensor data, and has a defined physical address.

[0079] Each tensor is assigned a virtual address, which is mapped to a physical address in the data buffer via the SMMU page table. For static tensors (tensors that do not require switching data buffers, such as weight tensors and bias tensors), one tensor corresponds to one virtual address and one physical address; for dynamic tensors (tensors that require switching data buffers, such as input tensors and output tensors), one tensor corresponds to multiple virtual addresses and multiple physical addresses, with each virtual address mapped to a different physical address.

[0080] Table 1 below shows an example of the correspondence between tensors, data buffers, virtual addresses, and physical addresses: Table 1

[0081] During continuous inference of the same model, the input data changes between frames, requiring the data of the new frame to be stored in different data buffers. The system allocates multiple data buffers for tensors (such as input tensors) that require data buffer switching and rotates them between frames. Since the mapping relationship between the virtual address and physical address of all data buffers is pre-established in the SMMU page table during the model loading phase, switching only requires modifying the virtual address in the address slot to point to the physical address corresponding to the new data buffer; the SMMU page table itself does not need to be modified.

[0082] For example, two virtual addresses, VA1 and VA2, are allocated to the input tensor input_0 in Table 1 above, which are mapped to the physical addresses PA1 and PA2 of two data buffers, respectively. The current frame uses VA1 (pointing to Buffer1). When switching to the next frame, the runtime module changes the virtual address in the address slot to VA2 (pointing to Buffer2). The NPU will then automatically read the data from Buffer2 when accessing this tensor subsequently. The entire switching process involves only one address slot write operation, and the SMMU page table remains unchanged.

[0083] S102, the runtime module 40 assigns a unique SSID to the model and creates an SMMU page table corresponding to the SSID in SMMU10.

[0084] The SSID is used to identify the SMMU page table corresponding to the model in SMMU10. The SMMU page table is a data structure that SMMU10 looks up during address translation, used to store the address mapping relationship from virtual address to physical address. In multi-model concurrent scenarios, different models have different SSIDs, and SMMU10 distinguishes the SMMU page table corresponding to different models based on the SSID carried in the memory access request.

[0085] Different models have independent SMMU page tables, and the tensors of each model are stored in different physical address regions of system memory. Each model is assigned a unique SSID upon loading, and an independent SMMU page table corresponding to that SSID is created in the SMMU. Because the SMMU page tables of different models are independent of each other, even if two models use the same virtual address, they will be translated to different physical addresses. Therefore, when switching models, the physical address region that the computation core needs to access changes, and the address must be changed so that the SMMU can translate to the physical address region of the new model.

[0086] S103, runtime module 40 allocates virtual addresses for each tensor of the model.

[0087] During the offline compilation phase, the compiler allocates virtual addresses for each tensor and writes these virtual addresses directly into the corresponding address slots in the model binary file as the initial values ​​of the address slots. When loading the model, the runtime module 40 can either directly use the existing virtual addresses in the address slots, or it can reallocate new virtual addresses for each tensor and write the reallocated virtual addresses into the corresponding address slots in the model binary file.

[0088] S104, runtime module 40 allocates data buffers for each tensor in system memory 30.

[0089] For tensors that do not require switching data buffers during model execution (such as the model's weight tensor and bias tensor), i.e., static tensors, the runtime module 40 allocates a data buffer for each static tensor, and each data buffer corresponds to a physical address.

[0090] For tensors that require switching data buffers during model execution (such as the model's input and output tensors), i.e., dynamic tensors, the runtime module 40 allocates multiple virtual addresses for these dynamic tensors and allocates multiple data buffers in the system memory 30 when loading the model. Each data buffer corresponds to a physical address, and an address mapping relationship is established between each virtual address and its corresponding physical address. These virtual addresses are then added as address mapping entries to the model's SMMU page table.

[0091] For example, for an input tensor, runtime module 40 allocates three virtual addresses VA1, VA2, and VA3, and three data buffers Buffer1, Buffer2, and Buffer3, with physical addresses PA1, PA2, and PA3, respectively. Three address mapping entries are established in the SMMU page table: VA1→PA1, VA2→PA2, and VA3→PA3. This input tensor has a corresponding address slot in the model binary file, which stores the currently used virtual address. Runtime module 40 writes the currently used virtual address (e.g., VA1) into this address slot. The other virtual addresses of the tensor are recorded in the metadata area of ​​the model binary file in system memory. During model execution, when a data buffer needs to be switched, runtime module 40 retrieves the switched virtual address from the metadata area and rewrites the virtual address in the address slot (e.g., changing it from VA1 to VA2), causing the data accessed by the NPU to switch from Buffer 1 to Buffer 2.

[0092] S105, Runtime module 40 adds the address mapping relationship from virtual address to physical address of each tensor to the SMMU page table of the model.

[0093] The runtime module 40 adds the virtual-to-physical address mappings of each tensor to the SMMU page table of the model. For a tensor that has been allocated a data buffer, one address mapping is added to the SMMU page table as one address mapping entry. For a tensor that has been allocated multiple data buffers, multiple address mappings are added to the SMMU page table as multiple address mapping entries, with one address mapping entry corresponding to each data buffer.

[0094] After the above operations, the model's binary file has been stored in system memory 30, the model's SMMU page table has been established and contains the address mapping relationship of all tensors in the model, and the virtual address of each tensor has been written into the corresponding address slot. At this point, the model has been loaded, but has not yet run in computing core 22.

[0095] In this application, the SMMU page table is fully established during the model loading phase and remains unchanged during model operation. Therefore, there is no need to perform TLB invalidation operations at runtime, and the cached entries in the TLB always remain valid, avoiding performance jitter caused by TLB invalidation and improving address translation efficiency.

[0096] After the model loading phase is completed, the runtime module configures the physical address of the command package of the model to the task scheduler 21. In the subsequent task dispatch phase, the task scheduler 21 directly retrieves the command package from the system memory 30 based on the physical address and parses each command in sequence, without the runtime module needing to intervene line by line.

[0097] When the task scheduler 21 retrieves a command packet, it directly accesses system memory 30 via the physical address, without SMMU translation and without carrying the SSID, thus improving instruction fetching speed. Memory access requests initiated by the computing core 22 carry the SSID. The SMMU selects the corresponding SMMU page table based on the SSID to complete address translation before accessing the data. Therefore, the instruction fetch path and data path form a dual address space isolation—the instruction fetch path does not require address translation, improving instruction fetching efficiency; the data path is isolated by SSID through the SMMU, ensuring the security of multi-model data.

[0098] refer to Figure 4 After the model is loaded, when the loaded model needs to perform a task, perform the following steps:

[0099] S201, runtime module 40 sends tasks and task descriptors to NPU20.

[0100] When the runtime module 40 issues tasks to the computing core 22, it also configures a task descriptor. This task descriptor contains at least the following key information: address slot base address and instruction stream base address. The address slot base address is the starting physical address of the target model's address slot in system memory 30. The instruction stream base address is the starting address of the target model's NPU instruction stream in system memory 30.

[0101] The NPU20 contains multiple computing cores 22, which can execute tasks of different models in parallel.

[0102] S202, Task scheduler 21 assigns the task to computing core 22.

[0103] When parsing command packets, the task scheduler 21 arbitrates multiple ready tasks based on the task priorities in the command packets, and prioritizes the allocation of computing resources to high-priority tasks to achieve real-time scheduling of multiple models.

[0104] S203, runtime module 40 writes the SSID of the target model into the SSID register 221 of computing core 22.

[0105] S204, the computing core 22 reads the address slot and instruction stream corresponding to the target model from the system memory 30 through the DMA control unit 222.

[0106] Specifically, the DMA control unit 222 reads the address slot and instruction stream corresponding to the target model from the system memory 30 based on the address slot base address and instruction stream base address.

[0107] S205, the computing core 22 stores the read address slots and instruction streams into the internal storage unit 223.

[0108] The processes S201 to S205 described above are called dispatch (task assignment). Unlike traditional solutions where the CPU needs to pass tensor address information to the computational core through multiple register write operations, in this solution, the runtime module only needs to write the target model's SSID to the computational core's SSID register during dispatch, without needing to pass tensor data addresses. The computational core autonomously reads the address slot content from system memory through direct memory access to obtain the tensor address, reducing the communication overhead between the CPU and NPU.

[0109] After completing the above operations, computing core 22 has obtained the SSID and address slot information of the target model being executed, and enters the task execution phase, such as... Figure 5 As shown, the specific steps include:

[0110] S301, the computing core 22 obtains the SSID of the target model from the SSID register 221.

[0111] SSID register 221 is used to store the SSID of the target model currently running on the compute core. This SSID serves as the context label for all subsequent memory access requests initiated by the compute core, enabling the Memory Management Unit (SMMU) to identify the model to which the request belongs and select the corresponding SMMU page table.

[0112] S302, the computing core 22 retrieves the currently executing instruction from the internal storage unit 223.

[0113] The instruction carries access identifier information for the target tensor, which indicates which tensor the instruction needs to access.

[0114] S303, the computing core 22 determines the target tensor that needs to be accessed based on the acquired instructions.

[0115] S304, the computing core 22 reads the virtual address corresponding to the target tensor from the address slot of the internal storage unit 223.

[0116] The internal storage unit caches all the address slot contents of the current model. For dynamic tensors, the virtual addresses stored in the address slots are allocated by the runtime module when the model is loaded and can be updated when the data buffer switches. For static tensors (such as weights and biases), the virtual addresses in the address slots can be pre-allocated and written to the address slots by the compiler during offline compilation, or they can be reallocated by the runtime module when the model is loaded and are not modified after loading.

[0117] S305 and computing core 22 initiate a memory access request carrying the SSID and virtual address to SMMU10.

[0118] S306 and SMMU10 select the SMMU page table corresponding to the SSID carried in the memory access request, and look up the corresponding physical address in the SMMU page table according to the virtual address.

[0119] Specifically, SMMU10 first checks its internal TLB to see if an address mapping entry exists corresponding to the SSID and the virtual address. The TLB is a high-speed cache within the SMMU used to cache address mapping relationships. Each entry in the TLB contains a corresponding SSID identifier to distinguish address mapping relationships for different models. If a TLB hit occurs, the corresponding physical address is directly obtained. If a TLB miss occurs, the corresponding physical address is searched in the SMMU page table corresponding to the SSID based on the virtual address. The found address mapping relationship is then associated with the SSID and cached in the TLB to accelerate the translation process for subsequent addresses with the same address. In this application, the SMMU page table is fully established during the model loading phase and remains unchanged during model operation; therefore, TLB invalidation is not required, and the cached entries in the TLB remain valid.

[0120] S307 and SMMU10 return the found physical address to computing core 22.

[0121] S308 and computing core 22 access the corresponding data in system memory 30 based on the physical address returned by SMMU10.

[0122] After the above steps, the computing core 22 completes the data reading operation for this task and obtains the data required by the instruction. After the data reading is completed, the computing core 22 performs calculation operations on the obtained data to obtain the inference result of the task.

[0123] During the execution of the above tasks, the data required by the computing core 22 to perform the tasks is translated into addresses through the collaborative mechanism of address slots and SMMU page tables. Throughout the process, the SMMU page tables do not need to be modified.

[0124] The address management method provided in this application has at least the following beneficial effects:

[0125] First, the task assignment and initiation process is simple.

[0126] In this application, the virtual addresses of each tensor are pre-stored in address slots located in system memory 30. When dispatching tasks, the runtime module 40 only needs to write the SSID of the target model into the SSID register 221 of the computing core 22, without needing to transmit the address information of each tensor to the computing core 22 through multiple register write operations. When executing tasks, the computing core 22 reads the contents of the address slots from system memory 30 to internal storage unit 223 via the DMA control unit 222, autonomously obtaining the virtual addresses of each tensor. That is, the transmission of tensor address information is autonomously completed by the NPU hardware, rather than relying on the CPU to transmit it one by one through registers. Therefore, the number of communications between the CPU and NPU is significantly reduced, the task dispatch and startup process is simplified, and communication overhead is reduced.

[0127] Second, the hardware structure is simple to implement.

[0128] In existing technologies, each tensor in an NPU requires a separate address register to store its address information. As the number of tensors in the model increases, the number of registers also increases, leading to higher hardware complexity. In this application, however, the virtual addresses of tensors are stored in address slots within system memory 30, rather than in registers within the NPU. The computing core 22 reads the address slot content from system memory 30 into internal storage unit 223 via DMA control unit 222, autonomously obtaining the virtual addresses of each tensor. Therefore, each computing core 22 only needs one SSID register 221 to store the SSID of the currently executing model, eliminating the need for separate address registers for each tensor. Compared to existing technologies, this application reduces the number of internal address registers in the NPU, resulting in a simpler hardware structure and a smaller chip area.

[0129] Third, the address translation is highly efficient.

[0130] During the model loading phase, the SMMU page table already contains the address mappings of all tensors in the model. Since the SMMU page table remains unchanged during runtime, there is no need to perform TLB invalidation operations. The cached entries in the TLB are not cleared, and subsequent memory access requests can reliably hit the TLB. This avoids the increased latency and performance jitter caused by traversing the SMMU page table after a TLB miss due to a TLB invalidation, thus improving address translation efficiency.

[0131] Fourth, multi-model address space isolation. Each model has a unique SSID and an independent SMMU page table, and the virtual addresses of each model are independent and do not affect each other. Different models can use the same or partially the same virtual addresses. SMMU10 distinguishes the SMMU page tables corresponding to different models based on the SSID carried in the memory access request, achieving hardware-level isolation of the address space. Since each model has an independent SMMU page table, when allocating virtual addresses for a model, the compiler does not need to consider what virtual addresses other models use, nor does it need to adopt different virtual address allocation rules for each model; it can reuse the same virtual address allocation rules. In this way, the compilation phase and the model loading phase are decoupled, and the compiler focuses on the compilation optimization of the model itself without being aware of the model deployment situation in the runtime environment.

[0132] For example, suppose model A and model B are simultaneously loaded into a multi-model scheduling system. The virtual address of the input tensor of model A is 0x1000, and the virtual address of the input tensor of model B is also 0x1000. The correspondence between the SSID, virtual address, and physical address of model A and model B is shown in Table 2 below: Table 2

[0133] The SMMU page table (SSID=1) of Model A is shown in Table 3 below: Table 3

[0134] The SMMU page table (SSID=2) of Model B is shown in Table 4 below: Table 4

[0135] When computation core 22 executes the task of model A, it needs to obtain the input tensor data. It retrieves SSID=1 from SSID register 221 and initiates a memory access request carrying SSID=1 and virtual address 0x1000 to SMMU10. SMMU10 selects the SMMU page table corresponding to model A based on SSID=1 and translates virtual address 0x1000 into physical address 0x8000_0000.

[0136] When computation core 22 executes the task of model B, it needs to obtain the input tensor data. It retrieves SSID=2 from SSID register 221 and initiates a memory access request carrying SSID=2 and virtual address 0x1000 to SMMU10. SMMU10 selects the SMMU page table corresponding to model B based on SSID=2 and translates virtual address 0x1000 into physical address 0x9000_0000.

[0137] In the above process, Model A and Model B use the same virtual address 0x1000. However, since each model has an independent SMMU page table, SMMU10 selects the corresponding page table for address translation based on the SSID carried in the request, translating the same virtual address into different physical addresses, thus achieving hardware-level isolation of the address space of multiple models.

[0138] Therefore, when allocating virtual addresses for a model, the compiler does not need to consider what virtual addresses other models use, nor does it need to adopt different virtual address allocation rules for each model; it can reuse the same virtual address allocation rules. In this way, the compilation phase and the model loading phase are decoupled—the compiler focuses on the compilation optimization of the model itself, without needing to be aware of the model deployment situation in the runtime environment.

[0139] refer to Figure 6 During model execution, when it is necessary to switch the data buffer of a certain quantity, the following steps are performed:

[0140] S401, Runtime module 40 obtains the virtual address corresponding to the switched data buffer.

[0141] S402, the runtime module 40 writes the switched virtual address into the address slot corresponding to the tensor in the system memory 30.

[0142] Specifically, the runtime module 40 runs on the host CPU. When it writes data to the system memory 30, the data is first written to the CPU cache. At this time, the corresponding address slot content in the system memory 30 has not yet been updated. In order for the computing core 22 to read the updated address slot content from the system memory 30, the runtime module 40 performs a cache flush operation (flush_dcache) after completing the address slot write operation, forcibly writing the modified address slot data in the CPU cache back to the system memory 30. After the above operation, the address slot content in the system memory 30 is updated to the virtual address corresponding to the switched data buffer.

[0143] S403, the computing core 22 reloads the address slot of the target model from the system memory 30 to the internal storage unit 223 via the DMA control unit 222.

[0144] The address slots of the reloaded target model will overwrite the original address slots in the internal storage unit 223. That is, after the address slots are reloaded, the virtual address corresponding to the switched data buffer is cached in the internal storage unit 223.

[0145] S404, the computing core 22 obtains the virtual address of the tensor from the internal storage unit 223.

[0146] The virtual address obtained through step S404 is the virtual address of the switched data buffer.

[0147] S405 and computing core 22 initiate a memory access request to SMMU10, carrying the SSID and the switched virtual address.

[0148] S406 and SMMU10 select the SMMU page table corresponding to the SSID carried in the memory access request, and look up the corresponding physical address in the SMMU page table according to the switched virtual address.

[0149] S407 and SMMU10 return the found physical address to computing core 22.

[0150] S408 and computing core 22 access the corresponding data in system memory 30 based on the physical address returned by SMMU10.

[0151] For example, suppose a tensor is currently using data buffer buffer1 with a virtual address of VA1, which is stored in an address slot. When switching to data buffer buffer2 is required, the runtime module 40 obtains the virtual address VA2 corresponding to the switched data buffer buffer2 and writes VA2 into the address slot corresponding to the tensor in system memory 30. The computation core 22 reloads the address slot of the target model from system memory 30 to internal storage unit 223, obtains the switched virtual address VA2, and initiates a memory access request carrying SSID and VA2. SMMU 10 selects the corresponding SMMU page table based on SSID, translates VA2 into the corresponding physical address PA2, and the computation core 22 accesses the data in buffer2 in system memory 30 based on PA2. Since the address mapping relationship corresponding to VA2 has been pre-established in the SMMU page table during the model loading stage, the SMMU page table remains unchanged during the switching process.

[0152] The data buffer switching method provided in this application has at least the following beneficial effects:

[0153] First, switching the data buffer does not modify the SMMU page table, thus avoiding performance jitter caused by TLB failure.

[0154] During the model loading phase, multiple virtual addresses and corresponding physical addresses are allocated for tensors that require data buffer switching, and the address mapping relationship from virtual address to physical address is pre-established in the SMMU page table. When a data buffer needs to be switched, only the virtual address in the address slot needs to be modified to the virtual address corresponding to the switched data buffer. Since the address mapping relationship of all data buffers has been pre-established in the SMMU page table during the model loading phase, the SMMU page table does not need to be modified during the switch. Therefore, there is no need to perform TLB invalidation operations, the cached entries in the TLB remain valid, and subsequent memory access requests can reliably hit the TLB. This avoids the increased latency and performance jitter caused by traversing the SMMU page table after a TLB miss due to a TLB invalidation, thus improving address translation efficiency.

[0155] Second, the data buffer switching latency is low, meeting the requirements of high frame rate real-time scenarios.

[0156] Since the switching operation only involves modifying the virtual address in the address slot (i.e., a single memory write operation), there is no need to perform SMMU demapping and remapping operations, nor TLB invalidation operations. This significantly reduces switching latency from microseconds to nanoseconds. In high-frame-rate real-time inference scenarios, this solution ensures stable and controllable processing time for each frame, meeting real-time requirements.

[0157] Third, there is no need to configure a separate address register for each tensor, which simplifies the hardware structure.

[0158] In existing technologies, the NPU requires a separate address register for each tensor to store its address information. In this application, the virtual address of the tensor is stored in an address slot in system memory, rather than in a register within the NPU. When switching data buffers, only the virtual address in the address slot needs to be modified, eliminating the need to occupy register resources within the NPU, thus simplifying the hardware structure.

[0159] Fourth, the data buffer allocation is flexible and supports switching any number of data buffers.

[0160] During the model loading phase, any number of data buffers can be allocated to tensors that require data buffer switching, based on actual needs, and the address mapping relationship of all data buffers is pre-established in the SMMU page table. During model execution, these data buffers can be freely switched by modifying the virtual address in the address slot. The switching operation is not limited by the number of data buffers, providing high flexibility.

[0161] The address management method and data buffer switching method provided in this application can be applied to scenarios requiring continuous processing of multiple frames of data (such as video stream inference and autonomous driving perception). In these scenarios, the system needs to complete the inference task within each frame, and the input data (such as image frames) changes between frames, requiring frequent switching of the data buffer. For example, in a video stream processing scenario of 30 frames per second, the processing time for each frame is approximately 33 milliseconds, and the latency of switching the data buffer must be controlled within a very short time to meet real-time requirements. With the address management method provided in this application, the runtime module 40 only needs to rewrite the virtual address in the address slot to complete the data buffer switching, while the SMMU page table remains unchanged. There is no need to perform TLB invalidation operations, avoiding the increase in memory access latency caused by TLB invalidation, thereby ensuring the stability and low latency of address translation in high frame rate scenarios.

[0162] During model execution, when it is necessary to switch from the first model to the second model, perform the following steps:

[0163] S501, runtime module 40 writes the SSID of the second model into the SSID register 221 of computing core 22.

[0164] S502, the computing core 22 obtains the address slot corresponding to the second model from the system memory 30 and stores it in the internal storage unit 223.

[0165] Specifically, the computing core 22 obtains the address slot corresponding to the second model from the system memory 30 through the DMA control unit 222 and stores it in the internal storage unit 223.

[0166] Model switching can be completed quickly through steps S501 and S502. During the model switching process, the SMMU page tables of the first model and the second model are not modified, and the SMMU does not need to perform TLB failure operations.

[0167] After the model switch is completed, the SSID register 221 stores the SSID of the second model, and the internal storage unit 223 stores the address slot information of the second model. When executing subsequent tasks, the computing core 22 obtains the SSID of the second model from the SSID register 221, obtains the virtual addresses of each tensor of the second model from the internal storage unit 223, and initiates a memory access request carrying the SSID and virtual address of the second model to the SMMU 10. The SMMU 10 selects the SMMU page table corresponding to the second model based on the SSID of the second model carried in the memory access request, looks up the corresponding physical address in the SMMU page table according to the virtual address, and returns it to the computing core 22. The computing core 22 accesses the corresponding data in the system memory 30 according to the physical address returned by the SMMU 10.

[0168] In existing technologies, the virtual addresses of each tensor are stored in the address registers inside the NPU. When switching from the first model to the second model, in order to ensure that the execution state of the first model can be restored when it runs again, it is necessary to save all the tensor address information of the first model stored in the current register to memory, and then load the tensor address information of the second model into the register. This operation involves reading and writing a large number of registers, resulting in high overhead.

[0169] In this application, the virtual addresses of each tensor are stored in address slots in system memory 30, rather than in registers within the NPU. The address slots of the first model and the second model exist independently in system memory. When switching models, the address information of the first model does not need to be saved (it is already in system memory), and the address information of the second model does not need to be restored to registers (the computing core 22 can directly read it from system memory during execution via the DMA control unit 222). Therefore, no additional register saving or restoring operations are required during switching; only the SSID register needs to be rewritten and the new address slot loaded to complete the switch. The context switching overhead is minimal, achieving ultra-lightweight model switching. Therefore, when switching models, only the value in the SSID register needs to be rewritten to the SSID of the new model, and the address slot of the new model needs to be loaded into the internal storage unit to complete the switch; no tensor address information needs to be saved or restored.

[0170] Compared to existing technologies that require multiple registers to transfer tensor address information, each computing core 22 only needs to be configured with one SSID register 221 to complete model switching, which greatly simplifies the hardware structure and reduces the area. At the same time, the runtime module 40 only needs to write the SSID of the target model into the SSID register 221, without transferring any tensor address information to the computing core, which reduces the communication overhead between the CPU and NPU and shortens the task dispatch and startup process.

[0171] Furthermore, the first and second SMMU page tables are fully established during their respective model loading phases. During model switching, the SMMU page tables remain unchanged, eliminating the need for TLB invalidation. Cache entries in the TLB remain valid, ensuring address translation hits the TLB and preventing performance fluctuations caused by TLB invalidation, thus improving address translation efficiency. Since the SMMU page tables are not modified during model switching, there is no need to save or restore any SMMU page tables, nor is it necessary to copy or rebuild them. Context switching overhead is minimal, achieving ultra-lightweight model switching.

[0172] In an optional embodiment, the compiler further divides multiple tensors in the same model into at least one tensor group during the offline compilation phase. Tensors within the same tensor group share a base address. The compiler assigns each tensor an address offset within its respective tensor group. Based on the base address, each tensor determines its virtual address using its respective address offset within its tensor group. The compiler reserves address slots for each tensor group in the model binary file to store the base address of that tensor group.

[0173] During the offline compilation phase, the compiler assigns a group index to each tensor group, which uniquely identifies the tensor group. When generating NPU instructions, the compiler encodes the group index into the instructions that need to access tensors within that tensor group. This allows the NPU to determine the tensor group to be accessed based on the group index and read the base address of that tensor group from the corresponding address slot when executing the instructions.

[0174] In practice, the base addresses of each tensor group form the input parameter table (arg_tensor). Each entry in arg_tensor corresponds to an address slot of a tensor group, which is used to store the base address of that tensor group.

[0175] In one example, suppose a model contains two weight tensors, weight_0 and weight_1, one parameter tensor, param_0, six input tensors (input_0 to input_5), five output tensors (output_0 to output_4), and one activation tensor, act_0. The compiler divides these tensors into six tensor groups (Group0 to Group5), and the base addresses of these six tensor groups form an input parameter table (arg_tensor).

[0176] The arg_tensor contains 6 entries, each corresponding to an address slot in one of the six tensor groups (Groups 0 to 5), and is used to store the base address (base_addr) of each tensor group. For example, the entry[0] corresponds to the address slot of Group 0, the entry[1] corresponds to the address slot of Group 1, and so on.

[0177] Group0 contains three tensors: weight_0, weight_1, and param_0. The compiler allocates a base address of 0x0100_0000 to this tensor group, which is stored in entry[0] of arg_tensor. The compiler allocates the following intra-group address offsets for each tensor:

[0178] weight_0 is 0x0000_0000 (virtual address 0x0100_0000);

[0179] weight_1 is 0x0008_0000 (virtual address 0x0108_0000);

[0180] param_0 is 0x0010_0000 (virtual address 0x0110_0000).

[0181] Group1 contains five input tensors, input_0 to input_4. The compiler allocates a base address of 0x0400_0000 to this tensor group, which is stored in the entry[1] of arg_tensor. The offsets of the addresses within the group allocated by the compiler for each tensor are as follows:

[0182] input_0 is 0x0000_0000 (virtual address 0x0400_0000);

[0183] input_1 is 0x0001_0000 (virtual address 0x0401_0000);

[0184] input_2 is 0x0002_0000 (virtual address 0x0402_0000);

[0185] input_3 is 0x0003_0000 (virtual address 0x0403_0000);

[0186] input_4 is 0x0004_0000 (virtual address 0x0404_0000).

[0187] Group2 contains one input tensor, input_5, which is treated as a separate tensor group. The compiler assigns a base address of 0x0500_0000 to this tensor group, which is stored in the entry[2] of arg_tensor. During model execution, input_5 needs to switch data buffers. When loading the model, the runtime module 40 allocates multiple data buffers for this tensor and establishes a corresponding address mapping entry for each data buffer in the SMMU page table. The base address of input_5 is rewritten by the runtime module 40 during each frame switch to point to the currently used data buffer.

[0188] Group3 contains four output tensors: output_0 to output_2 and output_4. The compiler allocates a base address of 0x0600_0000 to this tensor group, which is stored in the entry[3] of arg_tensor. The offsets of the addresses within the group allocated by the compiler for each tensor are as follows:

[0189] output_0 is 0x0000_0000 (virtual address 0x0600_0000);

[0190] output_1 is 0x0001_0000 (virtual address 0x0601_0000);

[0191] output_2 is 0x0002_0000 (virtual address 0x0602_0000);

[0192] output_4 is 0x0003_0000 (virtual address 0x0603_0000).

[0193] Group4 contains one output tensor, output_3, which is a separate tensor group. The compiler allocates a base address of 0x0700_0000 for this tensor group, which is stored in the entry[4] of arg_tensor. During the model run, output_3 needs to switch data buffers, and its processing method is similar to that of input_5. When loading the model, the runtime module 40 allocates multiple data buffers for this tensor and establishes a corresponding address mapping entry for each data buffer in the SMMU page table. The base address of output_3 is rewritten by the runtime module 40 during each frame switch to point to the currently used data buffer.

[0194] Group5 contains an activation tensor, act_0. The compiler assigns a base address of 0x0800_0000 to this tensor group, which is stored in the entry[5] of arg_tensor. act_0 does not need to switch the data buffer during model execution, and its base address is not modified after being written to arg_tensor when the model is loaded.

[0195] Furthermore, the compiler assigns an update policy to each tensor group, which is recorded in the model binary file. The update policy indicates how the base address of the tensor group is updated during model execution.

[0196] For tensor groups that do not require switching data buffers (such as static weight tensor groups and activation tensor groups), the compiler sets their update strategy to the first strategy (model_load), which means that their base address is not modified after being written to the address slot when the model is loaded.

[0197] For tensor groups that require switching data buffers and use the Buffer pool (such as independent input tensors and independent output tensors), the compiler sets its update strategy to the second strategy (per_frame_rotate), which means that its base address is rotated from the pre-mapped Buffer pool. The runtime module 40 rewrites the base address in the address slot during each frame switch to make it point to the currently used data buffer.

[0198] During the offline compilation phase, the compiler identifies tensor groups that require data buffer switching and configures the buffer pool depth for each such group. The buffer pool depth represents the number of data buffers that the tensor group maintains simultaneously during runtime. The compiler records the buffer pool depth information in the model binary file.

[0199] The criteria for dividing tensor groups can be determined based on actual application requirements. For example, they can be divided based on factors such as tensor type, data source, lifecycle, and whether the data buffer needs to be switched during operation.

[0200] For example, fixed parameters such as weight tensors and bias tensors can be grouped into the same tensor group, multiple input tensors from the same data source can be grouped into the same tensor group (such as grouping multiple image input tensors from the same camera into one group), or tensors that need to switch data buffers during operation can be grouped into a separate tensor group.

[0201] Taking autonomous driving scenarios as an example, the model might contain image input tensors from cameras, point cloud data input tensors from radar, and positioning data input tensors from GPS. The compiler can group the image input tensors from cameras into an image input group, the point cloud data input tensors from radar into a radar input group, and the positioning data input tensors from GPS into a GPS input group. Input tensors from different data sources have different update frequencies and data characteristics; grouping them allows for differentiated address management strategies.

[0202] During the model loading phase, the runtime module 40 allocates a contiguous physical address space for each tensor group. This contiguous physical address space is aligned to a preset size, consistent with the large page mapping granularity supported by the SMMU (e.g., 2MB, 4MB, etc.). The size of the physical address space allocated to each tensor group is no less than the total size of the virtual address range of all tensors within that tensor group. The physical address space sizes of different tensor groups can be the same or different, determined by the runtime module 40 during loading based on the virtual address range size of each tensor group. The SMMU page table uses large page mapping, mapping the contiguous virtual address range starting from the initial virtual address of each tensor group to the corresponding contiguous physical address space. With large page mapping, a tensor group only needs one page table entry to cover its entire virtual address range, reducing the number of entries in the SMMU page table and lowering TLB pressure.

[0203] For example, for Group0 with a virtual address range of approximately 1.1MB, runtime module 40 allocates a contiguous physical address space (e.g., 2MB) aligned with the SMMU big page granularity; for Group1 with a virtual address range of approximately 320KB, the same contiguous physical address space with the same alignment granularity is allocated. For tensor groups with a larger virtual address range (e.g., 3MB), runtime module 40 can allocate a 4MB contiguous physical address space. Regardless of the allocated space size, alignment is based on the SMMU big page granularity to meet the requirements of SMMU big page mapping.

[0204] For tensor groups that do not require data buffer switching (such as Group0, Group1, Group3, and Group5), runtime module 40 allocates a physical address-contiguous data buffer for the tensor group, writes the base address of the tensor group into the corresponding address slot in arg_tensor, and creates an address mapping entry in the SMMU page table to map the continuous virtual address range starting from the base address to the continuous physical address space. Therefore, only one address mapping entry needs to be created for a tensor group.

[0205] Taking Group0 (containing three tensors: weight_0, weight_1, and param_0) as an example, this tensor group has only one base address, 0x0100_0000, and its starting virtual address is also 0x0100_0000. The runtime module 40 allocates a contiguous physical address space for this tensor group and creates an address mapping entry in the SMMU page table, mapping the contiguous virtual address space starting at virtual address 0x0100_0000 to this contiguous physical address space. This mapping entry covers the contiguous virtual address range of the three tensors: weight_0, weight_1, and param_0. Without tensor group partitioning, each of the three tensors would require an independent address mapping entry, resulting in three mapping entries. With tensor group partitioning, the three tensors are merged into one mapping entry, reducing the number of mapping entries and lowering the storage overhead of the SMMU page table and the pressure on the TLB.

[0206] Similarly, Group1 (containing five tensors from input_0 to input_4) has only one mapping entry in the SMMU page table, but five entries are needed if grouping is not used; Group3 (containing four tensors from output_0 to output_2 and output_4) has only one mapping entry, but four entries are needed if grouping is not used; Group5 (containing one tensor, act_0) has one mapping entry.

[0207] In the example above, without tensor group partitioning, the 13 tensors Group0, Group1, Group3, and Group5 require 13 independent address mapping entries. After tensor group partitioning, these four tensors are combined into 4 address mapping entries, reducing the number of mapping entries from 13 to 4, which significantly reduces the storage overhead of the SMMU page table and the pressure on the TLB.

[0208] For tensor groups that require switching data buffers (such as Group2 and Group4), runtime module 40 first reads the attributes of the data buffers to be switched for each tensor group in the model binary file to determine the buffer pool depth for each tensor group. Then, runtime module 40 allocates multiple data buffers according to the determined buffer pool depth, with each data buffer corresponding to a physical address, and establishes an address mapping relationship between each virtual address and its corresponding physical address in the SMMU page table. All address mapping relationships for data buffers are established during the model loading phase, and no new or deleted mappings are added during runtime. Taking Group2 as an example, the compiler records a buffer pool depth of 6. During the loading phase, runtime module 40 allocates 6 data buffers (Buffer1 to Buffer6) for input_5, with physical addresses PA1 to PA6 respectively, and also allocates 6 virtual addresses and establishes 6 address mapping relationships.

[0209] During the model loading phase, the virtual-to-physical address mappings for all data buffers in the dynamic tensor group that require data buffer switching are pre-established in the SMMU page table. Since the SMMU page table is fully determined during the loading phase, when switching data buffers at runtime, only the base address in the address slot needs to be rewritten to switch the data accessed by the NPU to another data buffer, without modifying the address mappings in the SMMU page table. Therefore, the SMMU page table remains unchanged during runtime, eliminating the need for TLB invalidation operations, avoiding performance jitter caused by TLB invalidation, and improving address translation efficiency.

[0210] It should be noted that when the compiler partitions tensor groups, it can group tensors that require unified data buffer switching into the same tensor group, and group tensors that require independent data buffer switching separately or into the same tensor group with other tensors having the same switching strategy. For tensor groups that uniformly switch data buffers on a tensor group basis, their SMMU page table mapping method is the same as that of tensor groups that do not require data buffer switching; that is, all tensors within the same tensor group share the same address mapping entry.

[0211] For example, suppose a model contains five input tensors, input_0 to input_4, all from the same data source. These tensors are updated simultaneously with each frame, requiring a unified switching of the data buffer. The compiler divides these five input tensors into a tensor group (input_group), which has a base address stored in a corresponding address slot. During the model loading phase, the runtime module allocates a contiguous physical address space for this tensor group and creates an address mapping entry in the SMMU page table, covering the contiguous virtual address range of all input tensors in the group. During model execution, each frame switch modifies the base address of this tensor group, making it point to the new contiguous physical address space. Since this tensor group corresponds to only one address mapping entry, and this mapping covers the entire virtual address range of the group, the SMMU page table remains unchanged during switching.

[0212] Through the above-described implementation of tensor group partitioning, this application achieves at least the following technical effects:

[0213] First, merging multiple tensors into a single tensor group and sharing the same SMMU page table mapping entry significantly reduces the number of mapping entries in the SMMU page table, thereby reducing the storage overhead of the SMMU page table and the pressure on the TLB.

[0214] Second, all tensors within the same tensor group share a base address. The runtime module 40 only needs to write the base address into the address slot to complete the configuration of the virtual addresses of all tensors in the tensor group, simplifying the address slot writing operation.

[0215] Third, by configuring update strategies for tensor groups (fixed base address or replaceable base address), differentiated processing of different tensor groups is achieved, which ensures the stability of static data and meets the flexibility requirements of dynamic data.

[0216] During the task dispatch and model startup phase, the computing core 22 reads the address slot corresponding to the target model from the system memory 30 through the DMA control unit 222 and stores it in the internal storage unit 223. The address slot stores the base address of each tensor group.

[0217] For the intra-group address offset, it is allocated and fixed by the compiler for each tensor during the offline compilation phase and encoded in the instruction stream of the model. Therefore, the intra-group address offset is loaded into the internal storage unit 223 along with the instruction stream, and is decoded from the currently executing instruction by the address generation unit 224 during the task execution phase.

[0218] During the task execution phase, the computing core 22 obtains the currently executing instruction, determines the address slot corresponding to the tensor group that needs to be accessed based on the group index carried in the instruction, reads the base address of the tensor group from the address slot in the internal storage unit 223, and adds the read base address to the address offset within the tensor group through the address generation unit 224 to generate a complete virtual address.

[0219] like Figure 7 As shown, the address generation unit 224 includes a selector 2241 and an adder 2242. The selector 2241 has multiple input terminals, each connected to multiple entries of the address slots stored in the internal storage unit 223. Each entry corresponds to a tensor group, used to select the corresponding entry from the multiple entries under the control of the group index carried in the instruction, and output the base address stored in that entry. The adder 2242 adds the base address output by the selector 2241 to the intra-group address offset carried in the instruction to generate a complete virtual address.

[0220] Computing core 22 initiates a memory access request carrying the SSID and virtual address to SMMU10.

[0221] SMMU10 selects the SMMU page table corresponding to the SSID carried in the memory access request, looks up the corresponding physical address in the SMMU page table according to the virtual address, and returns the found physical address to computing core 22.

[0222] The computing core 22 accesses the corresponding data in the system memory 30 based on the physical address.

[0223] During model execution, when switching from the first model to the second model is required, the runtime module 40 rewrites the value in the SSID register 221 of the computing core 22 from the SSID of the first model to the SSID of the second model. When executing subsequent tasks, the computing core 22 retrieves the SSID of the second model from the SSID register 221, loads the address slot corresponding to the second model from the system memory 30 to the internal storage unit 223 via the DMA control unit 222, and continues executing the task.

[0224] The first SMMU page table of the first model and the second SMMU page table of the second model are fully established during the model loading phase, and remain unchanged during the switching process. Since the SMMU page tables are not modified, there is no need to perform TLB invalidation operations; the cached entries in the TLB remain valid, and address translation can hit the TLB, avoiding performance jitter caused by TLB invalidation and improving address translation efficiency. Simultaneously, because the SMMU page tables are not modified, there is no need to save or restore any SMMU page tables, nor is there a need to copy or rebuild page tables, resulting in minimal context switching overhead and achieving ultra-lightweight model switching.

[0225] During the data buffer switching process, for a tensor group whose data buffer needs to be switched, the runtime module 40 can achieve unified switching of the data buffers of all tensors in the tensor group by rewriting the base address in the address slot corresponding to the tensor group and modifying it to the base address corresponding to the switched data buffer group.

[0226] Since the address mapping relationships for all data buffers of each tensor within a tensor group are pre-established in the SMMU page table during the model loading phase, switching only requires modifying the base address in the address slot; the SMMU page table does not need any modification. Therefore, the SMMU does not need to perform a TLB invalidation operation. The cached entries in the TLB remain valid, and address translation can reliably hit the TLB, avoiding performance jitter caused by TLB invalidation and improving address translation efficiency. Furthermore, since the switching operation only involves modifying a single address slot, there is no need to modify the address information for each tensor within the group separately, further reducing switching overhead and improving scheduling efficiency in scenarios involving unified switching of multiple tensors.

[0227] In this embodiment, apart from the tensor group partitioning steps mentioned above, other operations in the offline compilation stage, model loading stage, and task execution stage (including the establishment of SMMU page tables, the addition of address mapping relationships, etc.) are the same as in the previous embodiments, and can be referred to the relevant descriptions above for details; the operations in the model switching stage and data buffer switching stage are also the same as in the previous embodiments, and will not be repeated here.

[0228] like Figure 1 As shown, the chip provided in this application embodiment includes a system memory management unit (SMMU) 10 and multiple computing cores 22.

[0229] SMMU10 is configured with multiple SMMU page tables, each corresponding to a single SSID, and each SSID corresponding to a single model. SMMU10 receives memory access requests initiated by NPU20 and performs address translation based on the SSID and virtual address carried in the memory access request. When computing core 22 within NPU20 initiates a memory access request to SMMU10 carrying the SSID and virtual address, SMMU10 looks up the corresponding SMMU page table based on the SSID in the request, finds the corresponding physical address in that SMMU page table based on the virtual address, and returns the physical address to computing core 22.

[0230] Each computing core 22 is configured with an SSID register 221 and an internal storage unit 223. The SSID register 221 stores the SSID of the target model currently running on the computing core 22, and the SSID in the SSID register 221 is used to mark the memory access request sent by the computing core 22 to the SMMU 10. The internal storage unit 223 is used to cache the address slots corresponding to each tensor in the target model. The address slots are used to store the virtual address of the corresponding tensor for use when generating memory access requests.

[0231] The SMMU page table of the target model is determined when the model is loaded and remains unchanged after the model is loaded.

[0232] The chip provided in this application embodiment only requires one SSID register 221 per computing core 22 to achieve task isolation and switching between multiple models. Since the virtual addresses of each tensor are stored in address slots of the system memory 30, rather than in registers within the chip, there is no need to configure an independent address register for each tensor internally, significantly simplifying the hardware structure and reducing chip area. Simultaneously, the SMMU page table remains unchanged after model loading, eliminating the need for TLB invalidation operations during chip operation and improving address translation efficiency.

[0233] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the descriptions of the above embodiments are only for the purpose of helping to understand the methods of the embodiments of this application and should not be construed as limiting the embodiments of this application. Any changes or substitutions that can be made by those skilled in the art should be covered within the protection scope of the embodiments of this application.

Claims

1. An address management method in multi-model scheduling, applied to a system including a system memory management unit (SMMU), a runtime module, and a hardware accelerator, characterized in that, The method includes: During the model loading phase, the runtime module assigns an SSID to the loaded model, establishes an SMMU page table corresponding to the SSID in the SMMU, and stores the address slots of the model. Each loaded model in the system has a different SSID, and each SSID corresponds to an independent SMMU page table. The SMMU page table contains the address mapping relationship from the virtual address to the physical address of each tensor in the corresponding model. The SMMU page table is determined when the model is loaded and remains unchanged after the model is loaded. The address slots are used to store the virtual addresses of each tensor in the corresponding model. During the task dispatch phase, the runtime module configures the SSID of the target model to the computing core of the hardware accelerator and stores the address slot of the target model into the internal storage unit of the computing core. During the task execution phase, the computing core obtains the SSID of the target model, reads the virtual address from the address slot in the internal storage unit, and initiates a memory access request carrying the SSID of the target model and the virtual address. The SMMU selects the corresponding SMMU page table based on the SSID carried in the memory access request, and translates the virtual address in the memory access request into the corresponding physical address based on the selected SMMU page table, and returns it to the computing core.

2. The method according to claim 1, characterized in that, The computing core is equipped with an SSID register, which is used to store the SSID of the target model currently running on the computing core; The runtime module configures the SSID of the target model to the computing core of the hardware accelerator, specifically including: The runtime module writes the SSID of the target model into the SSID register.

3. The method according to claim 2, characterized in that, The method further includes: When it is necessary to switch the first model running in the computing core to the second model, the runtime module updates the value in the SSID register of the computing core from the SSID of the first model to the SSID of the second model; the SMMU page table of the first model and the SMMU page table of the second model remain unchanged during the model switching process.

4. The method according to claim 1, characterized in that, At least a portion of the tensors of at least two models loaded in the system use the same virtual address; The SMMU selects the corresponding SMMU page table based on the SSID carried in the memory access request, and translates the same virtual address into different physical addresses.

5. The method according to claim 1, characterized in that, The address slots are reserved by the compiler for each tensor during the offline compilation phase; The method further includes: During the model loading phase, the runtime module writes the virtual addresses allocated to each tensor of the model into the address slots corresponding to each tensor.

6. The method according to any one of claims 1 to 5, characterized in that, Multiple tensors in the same model are divided into at least one tensor group, each tensor group corresponds to an address slot, and tensors within the same tensor group share a base address, which is stored in the address slot corresponding to the tensor group; each tensor within the same tensor group determines its own virtual address based on the base address and its own address offset within its tensor group.

7. The method according to claim 6, characterized in that, The method further includes: When it is necessary to switch the first model running in the computing core to the second model, the computing core obtains the address slots corresponding to each tensor group in the second model from the system memory of the system and stores them in the internal storage unit.

8. The method according to claim 6, characterized in that, In the SMMU page table, tensors within the same tensor group share the same address mapping entry; different tensor groups correspond to different address mapping entries.

9. The method according to claim 6, characterized in that, The computing core includes an address generation unit, and the step of reading a virtual address from an address slot in the internal storage unit includes: The address generation unit reads the base address of the tensor group to which the target tensor belongs from the address slot in the internal storage unit; The address generation unit determines the virtual address of the target tensor based on the intra-group address offset between the base address and the target tensor.

10. A chip for executing the address management method in multi-model scheduling as described in claim 1, characterized in that, The chip includes: The System Memory Management Unit (SMMU) is configured with multiple SMMU page tables. Each SMMU page table corresponds to an SSID, and each SSID corresponds to a model. The SMMU is used to select the corresponding SMMU page table based on the SSID carried in the received memory access request, translate the virtual address of each tensor in the model into a physical address, and return the physical address to the computing core. Multiple computing cores, each of which is configured with an SSID register and an internal storage unit; The SSID register is used to store the SSID of the target model currently running on the computing core, and the SSID in the SSID register is used to mark the memory access request sent by the computing core to the SMMU; The internal storage unit is used to cache the address slots corresponding to each tensor in the target model. The address slots are used to store the virtual addresses of the corresponding tensors for use when generating memory access requests. The SMMU page table of the target model is determined during model loading and remains unchanged after model loading.

11. A multi-model scheduling system, characterized in that, include: The chip as described in claim 10; System memory, connected to the chip, is used to store the address slots corresponding to each model; The runtime module is used to assign an SSID to the model during the model loading phase, establish an SMMU page table corresponding to the SSID in the SMMU of the chip, and save the address slot of the model. In addition, during the task dispatch phase, the SSID of the target model is configured to the computing core of the chip, and the address slot of the target model is stored in the internal storage unit of the computing core.

Images