Tensor-aware on-chip cache system for multi-neural network collaborative scenarios

Abstract

This invention proposes a tensor-aware on-chip cache system for multi-neural network collaborative scenarios. This on-chip cache architecture addresses the on-chip cache resource contention problem faced by multiple AI accelerators collaboratively executing multi-neural network workloads. It constructs a three-level address space—virtual address, tensor address, and physical address—and designs instructions such as tensor registration, tensor deregistration, and tensor query to achieve on-chip cache state awareness of tensor data. Simultaneously, this invention proposes an on-chip cache allocation algorithm based on tensor data awareness. By sensing the on-chip cache state, it dynamically maps virtual addresses to tensor addresses, optimizing the distribution of tensor data in the on-chip cache, effectively reducing cache contention among tensor data, and further improving the performance of multi-model collaborative processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of on-chip cache systems in a heterogeneous SoC (System on Chip) architecture integrating multiple artificial intelligence accelerators, and in particular to a tensor-aware on-chip cache system for multi-class neural network collaborative scenarios and a dynamic cache allocation method thereof. BACKGROUND

[0002] In a heterogeneous intelligent SoC system, multiple artificial intelligence accelerators are integrated, and multiple accelerators access data through a shared on-chip cache system. Through the collaborative processing of multiple artificial intelligence accelerators, neural network models have become an important way to efficiently compute artificial intelligence tasks. However, in the process of collaborative execution of multiple neural network models, the shared on-chip cache system often becomes a performance bottleneck. Since the tensor data of each model needs to be frequently exchanged between the on-chip cache and the off-chip memory, this data exchange often leads to competition for on-chip cache resources and invalid cache replacement behavior, thereby reducing the efficiency of the cache.

[0003] Existing cache allocation schemes mainly rely on physical address indexing, which cannot dynamically adapt to the resource allocation needs between multiple models, resulting in significant performance loss when the data of each neural network model is replaced in the on-chip cache. Therefore, how to optimize the allocation of on-chip cache in the context of multiple model collaborative processing to reduce data conflicts has become a pressing problem. SUMMARY

[0004] The present application proposes a tensor-aware on-chip cache system for multi-neural network model collaborative processing, aiming to solve the performance bottleneck problem caused by cache resource competition in a heterogeneous SoC architecture. In the process of multi-neural network model collaborative processing, multiple models will simultaneously access the on-chip cache, leading to the replacement of tensor data in the cache, thereby affecting the performance of the system. The present application introduces a three-level address structure of virtual address, tensor address, and physical address, designs a tensor-aware cache structure and cache allocation algorithm, effectively optimizing the cache allocation and management of tensor data, thereby improving the efficiency of the cache and reducing cache conflicts.

[0005] The technical scheme of the application includes on-chip cache side tensor instruction, tensor address mapping, tensor-aware cache allocation algorithm and other mechanisms. The on-chip cache system supports tensor registration, tensor deregistration and tensor query instructions. The tensor registration instruction contains key information such as the data size of the tensor, the read-write mode, the tensor identifier, the thread identifier and other information. Among them, the data size of the tensor, the read-write mode, the tensor identifier are generated by the compiler in the compilation phase, and the thread identifier is stored in the CSR register. The tensor registration instruction is generated by the compiler and inserted at the start position of the life cycle of the tensor. When the on-chip cache receives the tensor registration instruction, it triggers an interrupt and calls the operating system runtime to build the mapping relationship between the tensor data virtual address and the tensor address, and stores the mapping relationship in the virtual address-tensor address translation module cache. The tensor deregistration instruction is generated by the compiler and inserted at the end position of the life cycle of the tensor. When the on-chip cache receives the tensor deregistration instruction, the on-chip cache removes the related mapping relationship in the virtual address-tensor address translation module cache, and triggers a system interrupt to update the state statistics of each tensor in the operating system runtime. When the on-chip cache misses, the tensor address is translated into a physical address, and the memory access and data replacement behavior are performed. At the same time, the tensor controller updates the storage state of the related tensor according to the memory request.

[0006] The tensor-aware on-chip cache architecture proposed in the application is a key cache structure for multi-class neural network collaborative scenarios. The tensor-aware on-chip cache architecture is constructed in a multi-path group connection mode, uses tensor address as an index, and is based on the LRU (Least Recently Used) replacement strategy. The tensor-aware on-chip cache architecture includes metadata storage, data storage, instruction decoding, tensor awareness, address translation, controller and other modules. Among them, the metadata storage module stores the label of the tensor data; the data storage module stores the tensor data; the instruction decoding module parses the tensor registration, tensor query and tensor deregistration instructions and generates related control signals; the tensor awareness module counts the storage capacity of each tensor data in the on-chip cache. By listening to the memory request of the on-chip cache, when tensor data reading and tensor data replacement occur, the storage state of the related tensor is updated synchronously. When receiving a tensor query request, the storage state of the related tensor is returned; the address translation module supports address conversion between virtual address-tensor address and tensor address-physical address; the controller module is responsible for controlling the data replacement, consistency and cache miss record functions of the on-chip cache.

[0007] The tensor-aware on-chip cache allocation algorithm provided by the application is a key cache allocation algorithm for multi-class neural network collaborative scenarios. The purpose of the cache allocation algorithm is to generate a mapping mode of virtual addresses to tensor addresses for tensor data, effectively reducing cache competition among tensor data, and improving system cache utilization. An interval mapping mode is used between virtual addresses and tensor addresses. When the on-chip cache executes a tensor registration instruction, the tensor-aware on-chip cache allocation algorithm is called through a system interrupt. The algorithm first calculates the storage capacity of each tensor data in the on-chip cache based on a tensor query instruction. Then, the tensor data competition degree of each row of the on-chip cache is calculated. Finally, a greedy algorithm is used to generate an address mapping interval, so that the sum of the data competition degrees covered by the interval is minimized. Assuming that Z(n) represents the tensor data competition degree of the nth row of the on-chip cache, [m(t), n(t)] is the on-chip cache row interval based on the tensor address index of tensor t, and k(t) is the storage capacity of tensor t in the on-chip cache. Therefore, Z(n) can be calculated by the following formula:

[0008]

[0009] The execution process of the address mapping interval generation method based on the greedy algorithm provided by the application is as follows: first, according to the tensor scale and the on-chip cache structure transmitted by the tensor registration instruction, the number of on-chip cache rows covered by the data is calculated. Then, the sum of the tensor data competition degrees in each mapping interval is calculated in a sliding window manner. The mapping interval with the smallest sum of tensor data competition degrees is selected, and the offset between the tensor space address and the virtual address is calculated according to the interval. Assuming that the number of on-chip cache rows in the mapping interval is y, the minimization of the sum of the tensor data competition degrees can be represented as:

[0010] BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a tensor-aware storage system address space relationship diagram.

[0012] Figure 2 is a tensor-aware on-chip cache architecture diagram.

[0013] Figure 3 is a tensor instruction execution flow diagram. DETAILED DESCRIPTION

[0014] The tensor-aware on-chip cache system for multi-class neural network collaborative scenarios provided by the application will be described in detail below. The embodiments described below are part of the embodiments of the application, but not all embodiments. Based on the following embodiments, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0015] As shown in Figure 1 The tensor-aware on-chip cache system for multi-class neural network collaborative scenarios has three levels of virtual address space, tensor address space, and physical address space. The virtual address space is generated by the compiler for each tensor data in the compilation stage. The artificial intelligence accelerator executes the program based on the virtual address and sends a memory access request to the on-chip cache. The tensor-aware on-chip cache architecture uses the tensor address space as the on-chip cache index. The interval mapping method is used between the virtual address and the tensor address. The tensor address is equal to the sum of the virtual address and the address offset. During the running process, the tensor-aware cache allocation algorithm generates the address offset according to the on-chip cache state. The physical address is the storage address of the tensor data in the memory.

[0016] As shown in Figure 2 The tensor-aware on-chip cache architecture proposed by the present application is constructed in a multi-path group connection mode, uses a tensor address as an index, and is based on an LRU (Least Recently Used) replacement strategy. The tensor-aware on-chip cache architecture includes metadata storage, data storage, instruction decoding, tensor awareness, address translation, and controller modules. The metadata storage module stores the labels of the tensor data. The data storage module stores the tensor data. The instruction decoding module parses the tensor registration, tensor query, and tensor deregistration instructions and generates related control signals. The tensor awareness module counts the storage capacity of each tensor data on the on-chip cache. By listening to the memory access request of the on-chip cache, when tensor data reading and tensor data replacement occur, the storage state of the related tensor is updated synchronously. When a tensor query request is received, the storage state of the related tensor is returned. The address translation module supports address conversion between the virtual address and the tensor address and between the tensor address and the physical address. The controller module is responsible for the control of the on-chip cache data replacement, consistency, and cache miss record functions.

[0017] As shown in Figure 3As shown, the tensor-aware on-chip cache system supports tensor registration, tensor lookup, and tensor unregistration instructions. The tensor registration instruction contains key information such as the tensor's data size, read / write mode, tensor identifier, and thread identifier. The tensor's data size, read / write mode, and tensor identifier are generated by the compiler during the compilation phase, while the thread identifier is stored in the CSR register. The tensor registration instruction is generated by the compiler and inserted at the beginning of the tensor's lifecycle. When the on-chip cache receives a tensor registration instruction, it triggers an interrupt, calls the operating system runtime to construct the mapping relationship between the tensor's virtual address and tensor address, and stores the mapping relationship in the virtual address-tensor address translation module cache. The tensor unregistration instruction is generated by the compiler and inserted at the end of the tensor's lifecycle. When the on-chip cache receives a tensor unregistration instruction, it removes the relevant mapping relationship from the virtual address-tensor address translation module cache, triggers a system interrupt, and updates the operating system runtime's status statistics for each tensor. When a memory access miss occurs in the on-chip cache, the tensor address is translated into a physical address, and memory access and data replacement are performed. At the same time, the tensor controller updates the storage status of the relevant tensors according to the memory access request.

[0018] like Figure 3 As shown, when the tensor-aware on-chip cache system executes the tensor registration instruction, it executes the tensor-aware cache allocation algorithm via a system interrupt. The purpose of the cache allocation algorithm is to generate a mapping from virtual addresses to tensor addresses for tensor data. A range mapping method is used between virtual addresses and tensor addresses. When the on-chip cache executes the tensor registration instruction, it calls the tensor-aware on-chip cache allocation algorithm via a system interrupt. This algorithm first uses tensor lookup instructions to calculate the storage capacity of each tensor data in the on-chip cache. Then, it calculates the tensor data contention level for each row of the on-chip cache. Finally, a greedy algorithm is used to generate an address mapping range that minimizes the sum of the data contention levels covered by that range.

[0019] The tensor data contention is calculated as follows: Assume Z(n) represents the tensor data contention of the nth row of the on-chip cache, [m(t), n(t)] is the on-chip cache row range for tensor t based on its tensor address index, and k(t) is the storage capacity of tensor t in the on-chip cache. Therefore, Z(n) can be calculated using the following formula:

[0020]

[0021] The execution process of the address mapping interval generation method based on the greedy algorithm is as follows: First, based on the tensor size and on-chip cache structure passed in the tensor registration instruction, the number of on-chip cache lines covered by the data is calculated. Then, the sum of tensor data contention under each mapping interval is calculated sequentially using a sliding window approach. The mapping interval with the minimum sum of tensor data contention is selected, and the offset between the tensor space address and the virtual address is calculated based on this interval. Assuming the number of on-chip cache lines in this mapping interval is y, minimizing the sum of tensor data contention can be expressed as:

[0022]

[0023] This invention proposes a tensor-aware on-chip cache system for multi-neural network collaborative scenarios. This on-chip cache architecture addresses the on-chip cache resource contention problem faced by multiple AI accelerators collaboratively executing multi-neural network workloads. It constructs a three-level address space—virtual address, tensor address, and physical address—and designs instructions such as tensor registration, tensor deregistration, and tensor query to achieve on-chip cache state awareness of tensor data. Simultaneously, this invention proposes an on-chip cache allocation algorithm based on tensor data awareness. By sensing the on-chip cache state, it dynamically maps virtual addresses to tensor addresses, optimizing the distribution of tensor data in the on-chip cache, effectively reducing cache contention among tensor data, and further improving the performance of multi-model collaborative processing.

[0024] A tensor-aware on-chip cache system suitable for collaborative processing of multiple types of neural networks is disclosed. This system uses tensor space addresses as indexes and dynamically adjusts the mapping relationship between the virtual addresses of tensor data and tensor addresses through the operating system to optimize the distribution of tensor data for different neural network models, reduce cache resource conflicts, and improve the utilization efficiency of on-chip cache.

[0025] The system supports tensor registration, deregistration, and query commands, and can provide real-time statistics on tensor-level storage capacity usage. The on-chip cache adopts a multi-way set-associative structure, including modules for data caching, tag caching, and address translation. The compiler dynamically inserts corresponding instructions during the lifecycle of the neural network model and constructs a mapping relationship between virtual addresses and tensor addresses using a tensor layout-aware cache allocation algorithm.

[0026] The on-chip cache uses a set-associative multiplexed organization, indexed by relevant bits in the tensor space address based on the number of cache paths and the data block size. Tensor data uses a three-level address space: virtual address space, tensor address space, and physical address space.

[0027] The system dynamically obtains the data capacity of tensors in the cache through query instructions, and allocates contiguous address space for them according to the registration instructions of the tensors. By optimizing the mapping relationship between virtual addresses and tensor addresses, it achieves efficient layout of tensor data in the cache and reduces cache contention.

[0028] Tensor addresses are generated by adding an offset to a virtual address. The offset is dynamically calculated by the system at runtime based on the tensor layout.

[0029] The algorithm first determines the storage capacity of each tensor in the cache by querying instructions, calculates the tensor data contention of each row of the on-chip cache, and selects the mapping interval with the minimum contention by a greedy algorithm, thereby reducing the resource conflict of tensor data.

Claims

1. A tensor-aware on-chip cache system for multi-class neural network collaborative scenarios, characterized in that, The system indexes tensor space addresses, dynamically adjusts the mapping relationship between the virtual addresses of tensor data and the tensor addresses through the operating system, optimizes the distribution of tensor data of different neural network models, reduces cache resource conflicts, and improves the use efficiency of on-chip cache; The system adopts a multi-path group connection organization, indexes related bits of tensor space addresses based on the number of cache paths and data block size, and uses three levels of address space for tensor data: virtual address space, tensor address space, and physical address space. The virtual address space is generated by the compiler for each tensor data during the compilation phase. The artificial intelligence accelerator executes programs based on virtual addresses and issues memory access requests to the on-chip cache. The tensor address is generated by adding an offset to the virtual address, and the offset is dynamically calculated by the system during runtime according to the tensor layout. The system supports tensor registration, deregistration, and query instructions, and can real-time statistics storage capacity usage at the tensor level. The compiler dynamically inserts corresponding instructions during the life cycle of the neural network model, and constructs the mapping relationship between the virtual address and the tensor address through the cache allocation algorithm based on the tensor layout awareness. The algorithm first determines the storage capacity of each tensor in the cache through the query instruction, calculates the tensor data competition degree of each row of on-chip cache, and selects the mapping interval with the smallest competition degree through the greedy algorithm, thereby reducing the resource conflicts of tensor data.

2. The tensor-aware on-chip cache system of claim 1, wherein, During the system runtime, the data capacity of tensors in the cache is dynamically obtained through the query instruction, and a continuous address space is allocated for the tensors according to the registration instruction of the tensors. By optimizing the mapping relationship between the virtual address and the tensor address, efficient layout of tensor data in the cache is achieved, and cache competition is reduced.

Images