Multi-level cache integration method for artificial intelligence engine in SoC system

Abstract

The invention discloses a multi-level cache integration method for an artificial intelligence engine in an SoC (System on Chip) system, which comprises the following steps: constructing a multi-level cache architecture which takes an SRAM (Static Random Access Memory) as a first-level cache, takes a PSRAM (Priority SRAM) as a second-level cache and limits continuous address access of the SRAM, and dynamically adjusting cache access priorities by combining real-time monitoring of hardware engine workload and task types; a pre-fetching mechanism based on a pre-fetching engine prediction data access mode is adopted, so that direct access of hardware engines to DDR is effectively reduced, DDR resource competition among multiple hardware engines is relieved, and data access delay is reduced; meanwhile, the caching efficiency of the PSRAM is guaranteed, the data access timeliness is improved, the overall operation stability and response speed of the system are enhanced, the high performance of the artificial intelligence engine is guaranteed, meanwhile, the storage requirements of other hardware engines in the SoC system are balanced, and efficient utilization of resources and optimization of the performance are achieved.

Description

Technical Field

[0001] This invention relates to the field of SoC system architecture technology, and in particular to a method for integrating multi-level caches for an artificial intelligence engine in an SoC system. Background Technology

[0002] With the rapid development of artificial intelligence technology, more and more smart devices need to execute complex AI algorithms locally. These algorithms typically require rapid access to large amounts of data, placing higher demands on the response speed and data processing capabilities of storage systems.

[0003] Traditional system-on-a-chip (SoC) storage architectures often cannot meet the storage speed and data throughput requirements of AI applications. Specifically, in an SoC system, in addition to the artificial intelligence engine (AI engine), there are other hardware engines, such as the graphics processing unit (GPU) and the central processing unit (CPU), which also require dynamic random access memory (DDR) as storage units. This leads to competition for storage resources, especially under high load conditions, where the bandwidth and capacity requirements of different hardware engines for DDR may cause performance bottlenecks. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a multi-level cache integration method for artificial intelligence engines in SoC systems, aiming to improve the data processing efficiency and system performance of SoC systems.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] A method for integrating multi-level caches for an artificial intelligence engine in a SoC system includes the following steps:

[0007] A multi-level cache architecture is built based on the SoC system. The multi-level cache architecture includes on-chip memory and off-chip memory. The on-chip memory includes at least SRAM memory units, and the off-chip memory includes at least PSRAM memory units, DDR memory units, and FLASH memory units. The SRAM memory units serve as the first-level cache of the hardware engine, and the PSRAM memory units serve as the second-level cache of the hardware engine. Furthermore, the address accesses issued by the hardware engine to the PSRAM memory units are restricted to be consecutive.

[0008] The workload and task type of each hardware engine in the SoC system are monitored in real time. The grouped workload and task type are used as decision inputs, and the access priority of different hardware engines to cache resources is dynamically adjusted in combination with preset priorities and performance indicators.

[0009] The historical access data of the hardware engine is obtained and used as training data to input the prefetch engine constructed by the neural network. The data access pattern of the hardware engine is predicted, and data is prefetched from the DDR storage unit to the PSRAM storage unit or from the PSRAM storage unit to the SRAM storage unit in advance.

[0010] In some implementations, the method further includes obtaining the real-time load of different storage units and dynamically adjusting the power supply voltage and operating frequency of different storage units using a preset adjustment strategy.

[0011] In some implementations, the cache in the storage unit is refreshed when multiple hardware engines access data in the storage unit at the same time.

[0012] In some implementations, the hardware engine includes at least an AI engine, a central processing unit, and an image processor.

[0013] In some implementations, the source of the historical access data is:

[0014] The cache control unit reads data requests from the hardware engine and prioritizes them for distribution to the first-level cache. If the first-level cache hits, a read / write operation is performed. If the first-level cache misses, the data request is then distributed to the second-level cache. If the second-level cache still misses, the request is distributed to the DDR storage module. For miss events, the physical address and data volume of the miss event are fed back to the prefetch engine as historical access data. The prefetch engine then provides feedback to the hardware engine based on the context.

[0015] In some implementations, the training process of the prefetch engine includes:

[0016] When a cache miss occurs, the physical address and data amount are fed back to all operators in the prefetch engine. During the joint operation of multiple operators, intelligent models and bias values ​​are imported.

[0017] Furthermore, the intelligent model adjusts the prefetch address and data volume based on the access efficiency of the PSRAM storage unit within a preset time range, while the operator allocates data to the first-level cache or the second-level cache based on the cache usage and remaining size of the DDR storage module.

[0018] The beneficial effects of this invention are as follows: By constructing a multi-level cache architecture with SRAM as the first-level cache and PSRAM as the second-level cache and restricting their continuous address access, combined with real-time monitoring of the hardware engine workload and task type to dynamically adjust cache access priority, and a prefetch mechanism based on the prefetch engine to predict data access patterns, the direct access of the hardware engine to DDR is effectively reduced, alleviating DDR resource competition among multiple hardware engines and reducing data access latency; at the same time, PSRAM cache efficiency is ensured, data access timeliness is improved, the overall system stability and response speed are enhanced, the high performance of the artificial intelligence engine is guaranteed, and the storage needs of other hardware engines in the SoC system are balanced, achieving efficient resource utilization and performance optimization. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the SoC system architecture in an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram illustrating the data flow of historical access data in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the training process of the prefetch engine in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the content of this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to this invention are shown in the accompanying drawings, not all of them.

[0023] This embodiment proposes a method for integrating multi-level caches of an artificial intelligence engine in a SoC system, including the following steps:

[0024] Step 1: Build a multi-level cache architecture based on the SoC system. The multi-level cache architecture includes on-chip memory and off-chip memory. On-chip memory includes at least SRAM memory units, and off-chip memory includes at least PSRAM memory units, DDR memory units, and FLASH memory units. SRAM memory units serve as the first-level cache of the hardware engine, PSRAM memory units serve as the second-level cache of the hardware engine, and the address accesses issued by the hardware engine to PSRAM memory units are restricted to be consecutive.

[0025] The reason for the limitations of the aforementioned hardware engine is that the data type stored in the PSRAM memory unit as a secondary cache needs to be closely coordinated with the hardware engine. If the address access issued by the hardware engine is not continuous, it cannot replace the role of DDR.

[0026] Specifically:

[0027] SRAM storage cells are used for rapid data access in mission-critical applications, providing the fastest response times, such as the smallest unit of image data required by AI engines.

[0028] PSRAM memory cells, as fast-access cache storage, provide access speeds close to SRAM, while also offering high integration and low power consumption.

[0029] DDR memory cells are used for large-scale data storage, providing high storage density and low cost, such as shared memory between operating systems, applications, and hardware engines.

[0030] FLASH storage units are used for non-volatile data storage, ensuring that data is not lost in the event of a power outage.

[0031] Therefore, in this solution, SRAM storage units are used as the first-level cache of the hardware engine to store frequently accessed data. Simultaneously, PSRAM storage units serve as the second-level cache to store critical temporary data, reducing direct access to DDR storage units. DDR storage units serve as large-scale data storage for multiple hardware engines and also provide data sharing between them. Flash memory is generally used to store non-volatile data, such as firmware and AI models.

[0032] For a typical customized AI system, the possible storage capacities are: 100KB SRAM, 10MB PSRAM, 100MB / 1GB DDR, and 10GB / 100GB Flash. Faster access speeds result in higher costs and smaller storage capacities.

[0033] like Figure 1 As shown, the hardware engine includes at least an AI engine, a central processing unit (CPU), and an image processor (GPU). It connects to the cache control unit via an on-chip data bus and further requests data from on-chip or off-chip storage through the cache control unit. Subsequent steps 2 and 3 are controlled by the cache control unit, which manages access to and data flow for different storage resources such as SRAM, PSRAM, and DDR memory.

[0034] Step 2: Monitor the workload and task type of each hardware engine in the SoC system in real time, use the grouped workload and task type as decision input, and dynamically adjust the access priority of different hardware engines to cache resources in combination with preset priorities and performance indicators.

[0035] Specifically, the cache control unit includes a scheduler that monitors the workload and task type of each hardware engine and dynamically allocates storage resources for DDR memory units to ensure that each hardware engine receives the necessary storage resources. This scheduler implements a scheduling algorithm that monitors the storage needs of the AI ​​engine, CPU, and GPU in real time and dynamically adjusts the bandwidth allocation of DDR memory units based on preset priorities and performance metrics. Furthermore, the algorithm allows hardware engines to dynamically increase or decrease their cache usage based on their own cache usage. Specifically, each hardware engine requests the scheduler in real time to adjust its priority. The algorithm evaluates the priority levels and balances the usage needs of all parties, thereby dynamically allocating resources. For example, in high-load AI computing tasks, the cache access priority of the AI ​​engine can be increased. Additionally, a Quality of Service (QoS) mechanism is implemented to ensure that critical tasks (such as real-time tasks) receive sufficient cache resources. Different hardware engines can dynamically adjust their QoS levels to ensure system stability and responsiveness. The aforementioned "dynamic priorities and performance metrics" can be adjusted by engineers according to the technical specifications and requirements of the SoC system. For example, if a hardware engine frequently experiences L1 cache overflows during system-level operation, it indicates that L2 cache access speed is slow, thus requiring adjustment of the hardware engine's access priority. It's important to note that higher priority is not always better; rather, the priority should be balanced and stable with other hardware engines.

[0036] Step 3: Obtain historical access data of the hardware engine, use the historical access data as training data to input the prefetch engine built by the neural network, predict the data access pattern of the hardware engine, and prefetch data from the DDR storage unit to the PSRAM storage unit, or prefetch data from the PSRAM storage unit to the SRAM storage unit.

[0037] Specifically, the cache control unit includes a prefetch engine that runs an intelligent prefetch algorithm, which is specifically designed to predict the data access patterns of the hardware engine and preload data from DDR memory to PSRAM memory, or from PSRAM memory to SRAM memory, reducing access latency.

[0038] Step 3 utilizes the principle of locality, making the faster upper-level memory a buffer for the lower-level memory. If data exists in the upper-level memory, it can be read and written directly. This situation is called a hit, and the statistical probability of a hit is called the hit rate. If a hit occurs, it is necessary to access the lower-level memory. This situation is also called a miss.

[0039] like Figure 2 As shown in step 301, the source of historical access data is:

[0040] The cache control unit reads data requests from the hardware engine and prioritizes them for the L1 cache. If the L1 cache hits, read / write operations are performed. If the L1 cache misses, the data request is then forwarded to the L2 cache. If the L2 cache still misses, the request is forwarded to the DDR storage module. For miss events, the physical address and data amount of the miss event are fed back to the prefetch engine as historical access data. The prefetch engine provides feedback to the hardware engine in a unified manner based on the context, which can improve the hit rate.

[0041] In one example, a read data request initiated by the hardware engine is distributed by the cache control unit. It checks whether there is a cache hit. If there is a hit, the data is obtained directly from the L1 cache SRAM storage module. If there is a miss, the miss result is fed back to the prefetch engine. At the same time, it checks whether there is a hit in the L2 cache PSRAM storage module. If there is a hit, the data is obtained from the PSRAM storage module. If there is a miss, the result is also fed back to the prefetch engine, and the data is obtained directly from the DDR storage module.

[0042] Therefore, after self-learning, the prefetch engine sends a prefetch request to the lower-level memory before a potential memory failure occurs, so that the data block is read into the cache in advance when the data is actually used, avoiding increased access latency due to cache misses. The prefetch engine described in step 3 is a software-level optimization scheme designed to reduce latency when the hardware engine accesses main memory. Its core idea is to predict the data the processor might need in the future. Most AI tasks exhibit temporal and spatial locality, meaning that recently accessed data may be accessed again in the near future, and adjacent data lines are also likely to be accessed together. Therefore, this scheme, by analyzing historical access data patterns, allows the intelligent prefetch algorithm to learn and predict future data access needs.

[0043] like Figure 3 As shown, step 302, the training process of the prefetch engine includes:

[0044] When a cache miss occurs, the physical address and data volume are fed back to all operators in the prefetch engine. During the collaborative computation of multiple operators, an intelligent model and bias values ​​are incorporated. It's important to note that the intelligent model is a basic model built using different AI algorithms. Once the system is running, the model is adjusted based on the computation results; essentially, the intelligent model is an iterative and adaptive model. The bias values ​​are also iteratively calculated based on past bias results. These bias values ​​can be multiple factors, and ultimately, the target value for prefetching is obtained through linear or non-linear computation.

[0045] Furthermore, the intelligent model adjusts the prefetch address and data volume based on the access efficiency of the PSRAM memory cells within a preset time range. Simultaneously, the operator allocates data to the L1 or L2 cache based on the cache usage and remaining size of the DDR memory modules. It is worth noting that the aforementioned PSRAM memory cells require contiguous address access to serve the system more efficiently; therefore, the intelligent model adjusts the prefetch address and data volume based on the access efficiency of the PSRAM memory cells.

[0046] Step 4 also includes obtaining the real-time load status of different storage units and dynamically adjusting the power supply voltage and operating frequency of different storage units using a preset adjustment strategy, such as turning off the power and reducing the frequency.

[0047] In step 4 of this solution, customized power supply schemes can be provided for different memory cells to reduce power consumption in highly competitive environments. For example, dynamic voltage and frequency scaling (DVFS) can be developed according to the application scenario of the SoC system to reduce the power consumption of PSRAM and DDR memory cells under low load. DVFS dynamically adjusts the power supply voltage and operating frequency of the memory based on system load and performance requirements, optimizing power consumption and performance, especially in mobile devices and low-power applications. Thermal management strategies can also be set up to monitor the temperature of each memory cell and control heat generation by adjusting the operating frequency and voltage, ensuring the system operates within a safe temperature range and avoiding overheating.

[0048] Step 5 also includes refreshing the cache in the storage unit when multiple hardware engines access data in the storage unit at the same time to ensure data consistency.

[0049] During the operation of this solution, the same data may coexist in one or more cache layers, including Level 1 cache (SRAM), Level 2 cache (PSRAM), and Level 3 cache (DDR). Data consistency means that regardless of which cache layer the hardware engine accesses the data from, it can obtain the latest valid data. When data in any cache layer is modified, deleted, or replaced, the corresponding data copies in other cache layers must be updated or invalidated synchronously to avoid logical errors such as "reading old data" or "write conflicts."

[0050] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for integrating multi-level caches for an artificial intelligence engine in a SoC system, characterized in that, Includes the following steps: A multi-level cache architecture is built based on the SoC system. The multi-level cache architecture includes on-chip memory and off-chip memory. The on-chip memory includes at least SRAM memory units, and the off-chip memory includes at least PSRAM memory units, DDR memory units, and FLASH memory units. The SRAM memory units serve as the first-level cache of the hardware engine, and the PSRAM memory units serve as the second-level cache of the hardware engine. Furthermore, the address accesses issued by the hardware engine to the PSRAM memory units are restricted to be consecutive. The workload and task type of each hardware engine in the SoC system are monitored in real time. The grouped workload and task type are used as decision inputs, and the access priority of different hardware engines to cache resources is dynamically adjusted in combination with preset priorities and performance indicators. The historical access data of the hardware engine is obtained and used as training data to input the prefetch engine constructed by the neural network. The data access pattern of the hardware engine is predicted, and data is prefetched from the DDR storage unit to the PSRAM storage unit or from the PSRAM storage unit to the SRAM storage unit in advance.

2. The method for integrating multi-level caches of an artificial intelligence engine in a SoC system as described in claim 1, characterized in that, It also includes obtaining the real-time load status of different storage units and dynamically adjusting the power supply voltage and operating frequency of different storage units using preset adjustment strategies.

3. The method for integrating multi-level caches of an artificial intelligence engine in a SoC system as described in claim 1, characterized in that, It also includes refreshing the cache in the storage unit when multiple hardware engines perform data access operations in the storage unit at the same time.

4. The method for integrating multi-level caches of an artificial intelligence engine in a SoC system as described in claim 1, characterized in that, The hardware engine includes at least an AI engine, a central processing unit, and an image processor.

5. The method for integrating multi-level caches of an artificial intelligence engine in a SoC system as described in claim 1, characterized in that, The source of the historical access data is: The cache control unit reads data requests from the hardware engine and prioritizes them for distribution to the first-level cache. If the first-level cache hits, a read / write operation is performed. If the first-level cache misses, the data request is then distributed to the second-level cache. If the second-level cache still misses, the request is distributed to the DDR storage module. For miss events, the physical address and data volume of the miss event are fed back to the prefetch engine as historical access data. The prefetch engine then provides feedback to the hardware engine based on the context.

6. The method for integrating multi-level caches of an artificial intelligence engine in a SoC system as described in claim 1, characterized in that, The training process of the prefetch engine includes: When a cache miss occurs, the physical address and data amount are fed back to all operators in the prefetch engine. During the joint operation of multiple operators, intelligent models and bias values ​​are imported. Furthermore, the intelligent model adjusts the prefetch address and data volume based on the access efficiency of the PSRAM storage unit within a preset time range, while the operator allocates data to the first-level cache or the second-level cache based on the cache usage and remaining size of the DDR storage module.

Images