Tensor sensing on-chip caching system for multi-class neural network collaborative scene

Abstract

The invention provides a tensor sensing on-chip caching system oriented to a multi-class neural network collaborative scene. The on-chip cache architecture aims at the on-chip cache resource competition problem confronted under the condition that multiple artificial intelligence accelerators cooperatively execute multiple types of neural network loads, a virtual address-tensor address-physical address three-level address space is constructed, and tensor registration, tensor logout, tensor query and other instructions are designed; and on-chip cache state sensing of the tensor data is realized. Meanwhile, the invention provides an on-chip cache allocation algorithm based on tensor data perception, by perceiving an on-chip cache state, dynamically mapping a virtual address to a tensor address, and optimizing the distribution of tensor data in the on-chip cache, the cache competition phenomenon among the tensor data is effectively reduced, and the performance of multi-model cooperative processing is further improved.

Description

Technical Field

[0001] The present invention relates to the field of on-chip cache systems in heterogeneous SoC (system-on-chip) architectures integrating multiple artificial intelligence accelerators, and in particular to a tensor-aware on-chip cache system for multi-type neural network collaboration scenarios and a dynamic cache allocation method thereof. Background Art

[0002] Heterogeneous intelligent SoC systems integrate multiple AI accelerators, and multiple accelerators access data through a shared on-chip cache system. Collaborative processing of neural network models by multiple AI accelerators has become an important way to efficiently compute AI tasks. However, in the process of collaborative execution of multiple neural network models, the shared on-chip cache system often becomes a performance bottleneck. Since the tensor data of each model needs to be frequently exchanged between the on-chip cache and the off-chip memory, this data exchange often leads to competition for on-chip cache resources and invalid cache replacement behavior, thereby reducing the efficiency of cache utilization.

[0003] The existing cache allocation scheme mainly relies on the indexing method of physical addresses. This method cannot dynamically adapt to the resource allocation requirements between multiple models, resulting in a large performance loss when the data of each neural network model is replaced with each other in the on-chip cache. Therefore, how to optimize the allocation of on-chip cache to reduce data conflicts in the scenario of collaborative processing of multiple models has become an urgent problem to be solved. Summary of the invention

[0004] The present invention proposes a tensor-aware on-chip cache system for the collaborative processing of multiple neural network models, aiming to solve the performance bottleneck problem caused by cache resource competition in heterogeneous SoC architecture. During the collaborative processing of multiple neural network models, multiple models will access the on-chip cache at the same time, resulting in the mutual replacement of tensor data in the cache, thereby affecting the performance of the system. The present invention introduces a three-level address structure of virtual address, tensor address and physical address, designs a tensor-aware cache structure and cache allocation algorithm, and effectively optimizes the cache allocation and management of tensor data, thereby improving the efficiency of cache use and reducing cache conflicts.

[0005] The technical solution of the present invention includes mechanisms such as on-chip cache side tensor instructions, tensor address mapping, and tensor-aware cache allocation algorithms. The on-chip cache system supports tensor registration, tensor deregistration, and tensor query instructions. The tensor registration instruction contains key information such as the data scale, read-write mode, tensor identification, and thread identification of the tensor. Among them, the data scale, read-write mode, and tensor identification of the tensor are generated by the compiler during the compilation phase, and the thread identification is stored in the CSR register. The tensor registration instruction is generated by the compiler and inserted at the start of the life cycle of the tensor. When the on-chip cache receives the tensor registration instruction, an interrupt is triggered, and the operating system is called to construct a mapping relationship between the tensor data virtual address and the tensor address during runtime, and the mapping relationship is stored in the virtual address-tensor address translation module cache. The tensor deregistration instruction is generated by the compiler and inserted at the end of the life cycle of the tensor. When the on-chip cache receives the tensor deregistration instruction, the on-chip cache removes the relevant mapping relationship of the virtual address-tensor address translation module cache, and triggers a system interrupt to update the status statistics of each tensor during the operating system runtime. When a memory miss occurs in the on-chip cache, the tensor address is translated into a physical address, and memory access and data replacement are performed. At the same time, the tensor controller updates the storage status of the relevant tensor based on the memory access request.

[0006] The tensor-aware on-chip cache architecture proposed in the present invention is a key cache structure for multi-class neural network collaboration scenarios. The tensor-aware on-chip cache architecture is constructed in a multi-way group-connected mode, using tensor addresses as indexes, and based on an LRU (least recently used) replacement strategy. The tensor-aware on-chip cache architecture includes modules such as metadata storage, data storage, instruction decoding, tensor perception, address translation, and a controller. Among them, the metadata storage module stores the labels of tensor data; the data storage module stores tensor data; the instruction decoding module parses tensor registration, tensor query, and tensor deregistration instructions, and generates relevant control signals; the tensor perception module counts the storage capacity of each tensor data in the on-chip cache. By monitoring the memory access request of the on-chip cache, when tensor data reading and tensor data replacement occur, the storage status of the relevant tensors is synchronously updated. When a tensor query request is received, the storage status of the relevant tensor is returned; the address translation module supports address conversion between virtual address-tensor address and tensor address-physical address; the controller module is responsible for the control of on-chip cache data replacement, consistency, and cache miss recording functions.

[0007] The tensor-aware on-chip cache allocation algorithm proposed in the present invention is a key cache allocation algorithm for multi-class neural network collaboration scenarios. The purpose of the cache allocation algorithm is to generate a mapping method from virtual address to tensor address for tensor data, effectively reduce cache competition between tensor data, and improve system cache utilization. An interval mapping method is used between virtual addresses and tensor addresses. When the on-chip cache executes a tensor registration instruction, the tensor-aware on-chip cache allocation algorithm is called through a system interrupt. The algorithm first counts the storage capacity of each tensor data in the on-chip cache based on the tensor query instruction. Then, the tensor data contention of each row of the on-chip cache is calculated. Finally, a greedy algorithm is used to generate an address mapping interval so that the sum of the data contention covered by the interval is minimized. Assume that Z(n) represents the tensor data contention of the nth row of the on-chip cache, [m(t), n(t)] is the on-chip cache row interval of tensor t based on the tensor address index, and k(t) is the storage capacity of tensor t in the on-chip cache. Therefore, Z(n) can be calculated by the following formula:

[0008]

[0009] The execution process of the address mapping interval generation method based on the greedy algorithm proposed in the present invention is: first, according to the tensor scale passed in by the tensor registration instruction and the on-chip cache structure, the number of on-chip cache lines covered by the data is calculated. Then, the sum of the tensor data contention in each mapping interval is calculated in turn using a sliding window method. The method with the smallest sum of tensor data contention is selected as the mapping interval, and the offset between the tensor space address and the virtual address is calculated based on the interval. Assuming that the number of on-chip cache lines in the mapping interval is y, the sum of the minimized tensor data contention can be expressed as:

[0010] BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a tensor-aware storage system address space relationship diagram.

[0012] Figure 2 This is a diagram of the tensor-aware on-chip cache architecture.

[0013] Figure 3 It is the tensor instruction execution flow chart. Specific implementation methods

[0015] The tensor-aware on-chip cache system for multi-type neural network collaboration scenarios proposed by the present invention will be described in detail and completely in conjunction with the accompanying drawings. The embodiments described below are part of the embodiments of the present invention, not all of them. Based on the following embodiments, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] like Figure 1 As shown, the tensor-aware on-chip cache system for multi-type neural network collaboration scenarios proposed by the present invention has three levels: virtual address space, tensor address space, and physical address space. Among them, the virtual address space is generated by the compiler for each tensor data during the compilation stage, and the artificial intelligence accelerator executes the program based on the virtual address and issues a memory access request to the on-chip cache. The tensor-aware on-chip cache architecture uses the tensor address space as the on-chip cache index, and an interval mapping method is used between the virtual address and the tensor address. The tensor address is equal to the sum of the virtual address and the address offset. During operation, the tensor-aware cache allocation algorithm of the present invention generates an address offset according to the on-chip cache status. The physical address is the storage address of the tensor data in the memory.

[0017] like Figure 2 As shown, the tensor-aware on-chip cache architecture proposed in the present invention. The tensor-aware on-chip cache architecture is constructed in a multi-way group-connected mode, using tensor addresses as indexes, and based on an LRU (least recently used) replacement strategy. The tensor-aware on-chip cache architecture includes modules such as metadata storage, data storage, instruction decoding, tensor perception, address translation, and a controller. Among them, the metadata storage module stores the labels of tensor data; the data storage module stores tensor data; the instruction decoding module parses tensor registration, tensor query, and tensor deregistration instructions, and generates relevant control signals; the tensor perception module counts the storage capacity of each tensor data in the on-chip cache. By monitoring the memory access request of the on-chip cache, when tensor data reading and tensor data replacement occur, the storage status of the relevant tensors is synchronously updated. When a tensor query request is received, the storage status of the relevant tensor is returned; the address translation module supports address conversion between virtual address-tensor address and tensor address-physical address; the controller module is responsible for the control of on-chip cache data replacement, consistency, and cache miss recording functions.

[0018] like Figure 3As shown, the tensor-aware on-chip cache system supports tensor registration, tensor query, and tensor deregistration instructions. The tensor registration instruction contains key information such as the data scale, read-write mode, tensor identifier, thread identifier, etc. of the tensor. Among them, the data scale, read-write mode, and tensor identifier of the tensor are generated by the compiler during the compilation phase, and the thread identifier is stored in the CSR register. The tensor registration instruction is generated by the compiler and inserted at the beginning of the life cycle of the tensor. When the on-chip cache receives the tensor registration instruction, an interrupt is triggered, and the operating system runtime is called to construct a mapping relationship between the tensor data virtual address and the tensor address, and the mapping relationship is stored in the virtual address-tensor address translation module cache. The tensor deregistration instruction is generated by the compiler and inserted at the end of the life cycle of the tensor. When the on-chip cache receives the tensor deregistration instruction, the on-chip cache removes the relevant mapping relationship cached by the virtual address-tensor address translation module, triggers a system interrupt, and updates the status statistics of each tensor during the operating system runtime. When a memory miss occurs in the on-chip cache, the tensor address is translated into a physical address, and memory access and data replacement are performed. At the same time, the tensor controller updates the storage status of the relevant tensor based on the memory access request.

[0019] like Figure 3 As shown, when the tensor-aware on-chip cache system executes a tensor registration instruction, a tensor-aware cache allocation algorithm is executed through a system interrupt. The purpose of the cache allocation algorithm is to generate a mapping method from a virtual address to a tensor address for tensor data. An interval mapping method is used between virtual addresses and tensor addresses. When the on-chip cache executes a tensor registration instruction, the tensor-aware on-chip cache allocation algorithm is called through a system interrupt. The algorithm first counts the storage capacity of each tensor data in the on-chip cache based on the tensor query instruction. Then, the tensor data contention of each row of the on-chip cache is calculated. Finally, a greedy algorithm is used to generate an address mapping interval so that the sum of the data contention covered by the interval is minimized.

[0020] The calculation method of tensor data contention is: Assume that Z(n) represents the tensor data contention of the nth row of the on-chip cache, [m(t),n(t)] is the on-chip cache row interval of tensor t based on the tensor address index, and k(t) is the storage capacity of tensor t in the on-chip cache. Therefore, Z(n) can be calculated by the following formula:

[0021]

[0022] The execution process of the address mapping interval generation method based on the greedy algorithm is as follows: First, according to the tensor size passed in by the tensor registration instruction and the on-chip cache structure, the number of on-chip cache lines covered by the data is calculated. Then, the sum of the tensor data contention under each mapping interval is calculated in a sliding window manner. The method with the smallest sum of tensor data contention is selected as the mapping interval, and the offset between the tensor space address and the virtual address is calculated based on the interval. Assuming that the number of on-chip cache lines in the mapping interval is y, the sum of the minimized tensor data contention can be expressed as:

[0023]

[0024] The present invention proposes a tensor-aware on-chip cache system for multi-class neural network collaboration scenarios. The on-chip cache architecture is aimed at the on-chip cache resource competition problem faced by multiple artificial intelligence accelerators that collaborate to execute multiple types of neural network loads. By constructing a three-level address space of virtual address-tensor address-physical address, and designing instructions such as tensor registration, tensor deregistration, and tensor query, on-chip cache status perception of tensor data is achieved. At the same time, the present invention proposes an on-chip cache allocation algorithm based on tensor data perception. By sensing the on-chip cache status, the virtual address is dynamically mapped to the tensor address, and the distribution of tensor data in the on-chip cache is optimized, effectively reducing the cache competition phenomenon between tensor data, and further improving the performance of multi-model collaborative processing.

[0025] A tensor-aware on-chip cache system suitable for collaborative processing of multiple types of neural networks. The system uses tensor space addresses as indexes and dynamically adjusts the mapping relationship between the virtual addresses of tensor data and the tensor addresses through the operating system to optimize the distribution of tensor data for different neural network models, reduce cache resource conflicts, and improve the efficiency of on-chip cache usage.

[0026] The system supports tensor registration, deregistration and query instructions, and can count the storage capacity usage at the tensor level in real time. The on-chip cache adopts a multi-way group-connected structure, including data cache, label cache, address translation and other modules. The compiler dynamically inserts corresponding instructions during the life cycle of the neural network model, and constructs a mapping relationship between virtual addresses and tensor addresses through a tensor layout-aware cache allocation algorithm.

[0027] The on-chip cache is organized in a multi-way group-associative manner, indexed by the relevant bits of the tensor space address based on the number of cache ways and data block size. Tensor data uses three levels of address space: virtual address space, tensor address space, and physical address space.

[0028] The system dynamically obtains the data capacity of tensors in the cache through query instructions, and allocates continuous address space for them according to the registration instructions of the tensors. By optimizing the mapping relationship between virtual addresses and tensor addresses, the system achieves efficient layout of tensor data in the cache and reduces cache contention.

[0029] The tensor address is generated by adding the virtual address to the offset, which is dynamically calculated by the system at runtime based on the tensor layout.

[0030] The algorithm first determines the storage capacity of each tensor in the cache through query instructions, calculates the tensor data contention of each row of the on-chip cache, and selects the mapping interval with the least contention through a greedy algorithm, thereby reducing resource conflicts for tensor data.

Claims

1. A tensor-aware on-chip cache system for multi-class neural network collaboration scenarios, characterized by: The system uses the tensor space address as an index and dynamically adjusts the mapping relationship between the virtual address of the tensor data and the tensor address through the operating system to optimize the tensor data distribution of different neural network models, reduce cache resource conflicts, and improve the efficiency of on-chip cache usage.

2. The tensor-aware on-chip cache system according to claim 1, characterized in that: The system supports tensor registration, deregistration and query instructions, and can count the storage capacity usage at the tensor level in real time. The on-chip cache adopts a multi-way group-connected structure, including data cache, label cache, address translation and other modules. The compiler dynamically inserts corresponding instructions during the life cycle of the neural network model, and constructs a mapping relationship between virtual addresses and tensor addresses through a tensor layout-aware cache allocation algorithm.

3. The tensor address index according to claim 1, characterized in that: The on-chip cache is organized in a multi-way group-associative manner, indexed by the relevant bits of the tensor space address based on the number of cache ways and data block size. Tensor data uses three levels of address space: virtual address space, tensor address space, and physical address space.

4. When the operating system according to claim 1 is running, it is characterized in that: The system dynamically obtains the data capacity of tensors in the cache through query instructions, and allocates continuous address space for them according to the registration instructions of the tensors. By optimizing the mapping relationship between virtual addresses and tensor addresses, the system achieves efficient layout of tensor data in the cache and reduces cache contention.

5. The method for mapping virtual addresses and tensor addresses according to claim 3, characterized in that: The tensor address is generated by adding the virtual address to the offset, which is dynamically calculated by the system at runtime based on the tensor layout.

6. The method for mapping virtual addresses and tensor addresses according to claim 5, characterized in that: The algorithm first determines the storage capacity of each tensor in the cache through query instructions, calculates the tensor data contention of each row of the on-chip cache, and selects the mapping interval with the least contention through a greedy algorithm, thereby reducing resource conflicts for tensor data.

Images