A data caching method, system and apparatus

Abstract

The embodiment of the present application discloses a data caching method, system and device. After the previous computing task is completed, the first output result is stored in the cache, and the first target priority is set as the priority which makes the data not allowed to be replaced. When the current computing task starts, the computing parameters are moved to the cache, the current computing task is calculated according to the computing parameters and the first output result, the second output result obtained is written into the cache, and the priority of the second output result is attached. After the current computing task is completed, the priority of the second output result is set as the first target priority, and the priority of the first output result is adjusted as the second target priority which makes the data allowed to be replaced. Thus, by dynamically adjusting the priority of the data in the cache, the complexity of the inter-layer data reuse can be reduced, and the versatility and scalability of the system can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data caching method, system, and apparatus. Background Technology

[0002] In neural network inference, the efficiency of inter-layer data reuse plays a crucial role in system performance. Existing technologies mainly employ operator fusion and kernel fusion techniques to combine multiple consecutive layers into a single computational task, allowing intermediate data to be stored in registers, shared memory, or LLC, bypassing DRAM (Dynamic Random Access Memory) reads and writes, thereby improving the efficiency of inter-layer data reuse.

[0003] However, operator fusion and kernel fusion technologies can only combine a limited number of operators, rely on specific pattern matching, and have high kernel programming complexity after fusion, making them difficult to adapt to dynamic or complex network structures. Their lack of versatility and scalability limits their application in diverse inference scenarios. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data caching method, system, and apparatus that can reduce the complexity of inter-layer data reuse and improve the versatility and scalability of the system.

[0005] In a first aspect, embodiments of the present invention provide a data caching method, the method comprising: In response to the start of the current computing task, the computing parameters of the current computing task are moved to the cache; The current computing task is calculated based on the calculation parameters and the first output result to obtain a second output result. The first output result is the output result of the previous computing task. The first output result is stored in the cache and its occupancy priority is set to a first target priority, which is the priority that prevents data from being replaced. Write the second output result into the cache, along with the occupancy priority of the second output result; In response to the completion of the current computing task, the occupancy priority of the second output result is set to the first target priority, and the occupancy priority of the first output result is adjusted to the second target priority, where the second target priority is the priority that allows the data to be replaced.

[0006] In some embodiments, the first output result and / or the second output result is a matrix, and the elements of the matrix are data blocks.

[0007] In some embodiments, performing calculations on the current computation task based on the computation parameters and the first output result to obtain a second output result includes: Multiply the first target row of the first output result by the calculation parameter to obtain the second target row of the second output result.

[0008] In some embodiments, the method further includes: Before the current computing task is completed, the occupancy priority of the data block in the first target row that has been computed is set to the second target priority.

[0009] Secondly, embodiments of the present invention provide a data caching system, the system comprising a scheduling module, a data transport module, a computing module, and a high-speed cache, wherein: The scheduling module is used to set the occupancy priority of the first output result of the previous computing task to a first target priority in response to the completion of the previous computing task. The first target priority is the priority that prevents the data from being replaced. The data transfer module is used to move the calculation parameters of the current calculation task to the cache in response to the start of the current calculation task. The calculation module is used to perform calculations on the current calculation task according to the calculation parameters and the first output result to obtain a second output result, write the second output result into the cache, and attach the occupancy priority of the second output result; The scheduling module is also configured to, in response to the completion of the current computing task, set the occupancy priority of the second output result to a first target priority, and adjust the occupancy priority of the first output result to a second target priority, wherein the second target priority is the priority that allows data to be replaced.

[0010] In some embodiments, the first output result and / or the second output result is a matrix, and the elements of the matrix are data blocks.

[0011] In some embodiments, the calculation module is used to multiply the first target row of the first output result with the calculation parameter to obtain the second target row of the second output result.

[0012] In some embodiments, the calculation module is further configured to send a calculation completion instruction to the data transport module in response to the completion of the calculation of the second target row.

[0013] In some embodiments, the data transport module is further configured to, in response to receiving a computation completion instruction, set the occupancy priority of the data block in the first target row corresponding to the computation completion instruction to the second target priority before the current computation task is completed.

[0014] Thirdly, embodiments of the present invention provide a data caching device, the device comprising: The calculation parameter moving unit is used to move the calculation parameters of the current calculation task to the cache in response to the start of the current calculation task. The data calculation unit is used to perform calculations on the current calculation task according to the calculation parameters and the first output result to obtain a second output result. The first output result is the output result of the previous calculation task. The first output result is stored in the cache and its occupancy priority is set to a first target priority, which is a priority that prevents data from being replaced. A data writing unit is used to write the second output result into the cache, and includes the occupancy priority of the second output result; The priority adjustment unit is used to, in response to the completion of the current computing task, set the occupancy priority of the second output result to a first target priority, and adjust the occupancy priority of the first output result to a second target priority, wherein the second target priority is the priority that allows data to be replaced.

[0015] Fourthly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.

[0016] Fifthly, embodiments of the present invention provide a computer program product comprising a computer program, wherein when the computer program is run on a computer, the computer performs the method described in the first aspect above.

[0017] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method described in the first aspect.

[0018] The technical solution of this invention involves storing the first output result in a cache after the previous computation task is completed, with its occupancy priority set to a first target priority that prevents data replacement. When the current computation task begins, computation parameters are moved to the cache. The current computation task is then performed based on the computation parameters and the first output result. The obtained second output result is written to the cache, along with its occupancy priority. After the current computation task is completed, the occupancy priority of the second output result is set back to the first target priority, and the occupancy priority of the first output result is adjusted to a second target priority that allows data replacement. Therefore, by dynamically adjusting the occupancy priority of data in the cache, the complexity of inter-layer data reuse can be reduced, improving the system's versatility and scalability. Attached Figure Description

[0019] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a schematic diagram of data processing according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a data caching system according to an embodiment of the present invention; Figure 3 This is a flowchart of the data caching method according to the first embodiment of the present invention; Figure 4 This is a schematic diagram of the storage state according to an embodiment of the present invention; Figure 5 This is a flowchart of the data caching method according to the second embodiment of the present invention; Figure 6 This is a schematic diagram of matrix operations according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a data caching device according to an embodiment of the present invention; Figure 8 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0021] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0022] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0023] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0024] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0025] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) and various deep neural networks (such as ResNet-50, VGG, and Vision Transformer) have made significant progress in fields such as natural language processing and computer vision. During the inference process of these models, computationally intensive and memory-intensive tasks coexist. Among these, the transmission and storage of a large number of intermediate feature maps has become a key bottleneck restricting inference performance. In typical AI (Artificial Intelligence) inference chip architectures, the Last Level Cache (LLC), as a crucial hub connecting on-chip computing units and off-chip high-bandwidth memory (such as DRAM / HBM), directly determines the system's energy efficiency and latency performance through its data reuse efficiency.

[0026] However, in existing technical architectures, LLC faces significant challenges in reusing feature maps between layers. Specifically, the computation of AI networks exhibits significant inter-layer dependencies, with the output feature map of the previous layer often serving as the input of the next layer.

[0027] Figure 1 This is a schematic diagram of data processing according to an embodiment of the present invention. Figure 1 In the illustrated embodiment, the data processing involves three kernels (kernel functions or computation kernels), namely kernel1, kernel2, and kernel3.

[0028] In this context, the kernel is the computational program code running on a hardware accelerator. The hardware accelerator in this embodiment can be various modules, such as AI chips, GPGPU (General-Purpose computing on Graphics Processing Units) chips, and cached NPU (Neural Processing Unit) chips. Specifically, taking an AI chip as an example, neural network operators, including convolution, matrix multiplication, attention calculation, and normalization, are ultimately compiled into kernels and executed on the hardware accelerator. The kernel is invoked by the host CPU and executed in parallel on the hardware accelerator, typically by multiple threads running the same instructions simultaneously (i.e., the SIMT single-instruction multithreaded model). Each kernel is responsible for completing a specific mathematical operation task; for example, one kernel corresponds to one convolution operation, one kernel to one matrix multiplication operation, and one kernel to one LayerNorm operation.

[0029] exist Figure 1 In the embodiment shown, kernel1 performs a specific mathematical operation task based on weight W1 and initial data to obtain A1.

[0030] The output A1 of kernel1 is used as the input of kernel2. Kernel2 performs specific mathematical operations based on A1 and weight W2 to obtain A2.

[0031] The output A2 of kernel2 is used as the input of kernel3. Kernel3 performs specific mathematical operations based on A2 and weight W3.

[0032] Furthermore, to improve processing efficiency, when performing calculations at each layer, the outputs (such as A1, A2, etc.) and weights (such as W1, W2, W3, etc.) of the previous layer need to be loaded into the LLC (Last Level Cache). The LLC is a large-capacity high-speed cache located between the on-chip hardware accelerator and the off-chip main memory (DRAM). The core function of the LLC is to keep frequently accessed data inside the chip, significantly reducing the number of accesses to off-chip DRAM, thereby reducing memory access latency, saving memory bandwidth, and reducing system power consumption.

[0033] However, in actual hardware operation, because caches typically employ a random access strategy, after the output feature map of the previous layer is written to the LLC, the next layer's computation unit often cannot fully or promptly reuse this data. Especially in the prefill stage of LLM or in deep convolutional networks, this "write-to-invalidate" or "frequent swap-in / swap-out" phenomenon leads to a large number of redundant DRAM accesses, wasting on-chip cache bandwidth and capacity resources.

[0034] For example, in the above Figure 1 In this context, kernel2 relies on input A1, which is written from kernel1 and stored in LLC by default. During the process of kernel2 calculating A2, it may repeatedly use the content from A1, while also reading W2, and finally writing the result A2 back to LLC.

[0035] If the sum of A1 and W2 exceeds the capacity of LLC, then during the computation of kernel2, reading W2 will also create contention for data in A1, leading to a decrease in the reuse rate of A1. Simultaneously, writing A2 will create contention between A1 and W2, reducing the reuse rate of A1 and the residency percentage of A2.

[0036] If the sum of A1 and W2 is less than the capacity of LLC, then during the kernel2 calculation, writing A2 will compete with A1 and W2, reducing the reuse rate of A1 and the residency percentage of A2. At the same time, the memory address of W2, after being hashed, is mapped to the same set as the data of A1, which may also evict the data of A1.

[0037] If the sum of A1, W2, and A2 is less than the capacity of LLC, then theoretically, during the calculation of kernel2, if the allocation is reasonable, there will be no contention, and A1 can be fully reused. At the same time, A2 resides in LLC for reuse by kernel3.

[0038] To address the issue of low data reuse between layers, several technical solutions have been proposed.

[0039] First, there's the operator fusion / kernel fusion technique. By merging consecutive operators such as convolution, activation, and pooling into a single GPU kernel (Graphics Processing Unit), intermediate data is directly stored in registers, L1 cache, or shared memory, thus bypassing DRAM access. However, this technique has drawbacks: the combination of fused operators is limited, and as model structures become increasingly complex, the programming complexity of the fusion strategy increases exponentially, making it difficult to cover all scenarios.

[0040] Secondly, there's tiled and block-based scheduling. This technique uses a scheduling algorithm to divide the feature map into small blocks. After Layer N completes the computation of a block, Layer N+1 is immediately triggered to compute that block, leveraging the locality of data in LLC. However, in practical applications, traversing the entire complex AI network in a block-based manner is challenging. Most complex networks struggle to adapt to this "depth-first" computation order, and the common approach remains to execute individual kernels sequentially, limiting its universality.

[0041] The third is explicitly controlled on-chip memory (Scratchpad Memory Management). This transforms a portion of the LLC into software-controlled memory, allowing the compiler to directly manage the data lifecycle through instruction set extensions. While this is more deterministic than hardware-managed cache mechanisms, it significantly increases the difficulty of operator development, leading to poorer programmability of DSA (Dedicated Architecture) and hindering ecosystem adoption.

[0042] In addition, there are strategies for smooth weight and activation transitions, as well as persistent caching solutions. Strategies for smooth weight and activation transitions balance weight and activation, but cannot completely solve the feature map reuse problem. While persistent caching allows L2 cache to be divided into streaming and persistent regions, it requires manual memory address management in programming and suffers from capacity limitations and poor dynamism. Especially during persistence, even if some feature maps are no longer used, the areas they occupy are not released, which severely reduces cache utilization for edge AI chips with limited cache size.

[0043] In summary, while existing technologies have attempted to alleviate memory access bottlenecks to varying degrees, they consistently suffer from low reusability of inter-layer feature maps within LLCs and difficulties in dynamic management of cache resources. Therefore, a cache management mechanism is proposed to address these issues.

[0044] Figure 2 This is a schematic diagram of a data caching system according to an embodiment of the present invention. Figure 2 As shown, the data caching system of this embodiment includes a scheduling module 1, a data transport module 2, a computing module 3, and a high-speed cache 4. The data caching system of this embodiment is a hardware accelerator, which can be various modules, such as AI chips, GPGPU chips, and NPU chips with cache.

[0045] Scheduling module 1 is the CMD Scheduler. The command scheduler is a hardware control module on the chip responsible for receiving, parsing, and distributing computation task commands. Located at the front end of the chip, it connects the host interface to on-chip computing resources. The CPU encapsulates kernel startup commands, memory operation commands, synchronization commands, etc., into command packets via buses such as PCIe and writes them into the chip's command queue. The command scheduler continuously monitors the queue status. When a new command arrives, it retrieves the command packet from system memory through the DMA (Direct Memory Access) engine, parses the operation type, resource requirements, and dependencies, and then distributes the command to the corresponding execution unit. Furthermore, scheduling module 1 also handles dependencies between commands, ensuring that computation only begins after data is ready. In chips that support multi-task concurrency, the command scheduler is also responsible for multi-stream scheduling, priority arbitration, and time-slice allocation of hardware resources to improve chip utilization and throughput.

[0046] In this embodiment, the scheduling module 1 is also used to set or adjust the occupancy priority of each piece of data in the cache 4. The occupancy priority of data in the cache refers to the priority determination mechanism used to decide which data is retained and which is evicted when cache space is insufficient. Specifically, each data item in the cache is assigned a priority value or sorting criterion; higher-priority data is more likely to be retained in the cache, while lower-priority data is preferentially evicted to free up space.

[0047] Furthermore, the scheduling module 1 is used to set the occupancy priority of the first output result of the previous computing task to a first target priority in response to the completion of the previous computing task. The first target priority is the priority that prevents the data from being replaced.

[0048] Data transfer module 2 is GBD (Global Buffer Distributor). GBD is a data distribution hub module located at the core of the on-chip storage hierarchy, connecting external storage interfaces (such as HBM / DDR controllers) with on-chip cache and computing modules at various levels.

[0049] In this embodiment, the data moving module 2 is used to move the computation parameters of the current computation task to the cache in response to the start of computation of the current computation task. The computation task is a kernel function. That is, when the current computation task detects that the computation module needs to use its computation parameters, it sends an instruction to the data moving module 2, which then moves the computation parameters from memory to the cache 4. The computation parameters are weight matrices.

[0050] Based on the current computing task Figure 1 Taking kernel2 as an example, after the previous computation task kernel1 completes its calculation, the output A1 of kernel1 is stored in the cache. When the current computation task kernel2 starts executing, the computation parameters W2 required by kernel2 are migrated to the cache. This provides computation module 3 with the parameters needed to execute its computation tasks.

[0051] Computation module 3 is an independent hardware module within the chip that performs actual mathematical operations and is a fundamental building block constituting the chip's computing power. In the GPGPU architecture, a computation module typically includes multiple streaming processors (SPs) or CUDA cores, vector arithmetic logic units (ALUs), tensor cores, shared memory, a register file, and a warp scheduler. In the NPU architecture, the computation module is usually represented as an AI core, which integrates a cube unit, a vector unit, a scalar unit, and a dedicated data transfer engine (DMA / MTE). Each computation module can independently execute a thread block or a subset of tasks. Multiple computation modules work in parallel on the chip to jointly complete the computation of large-scale neural network operators. The number and internal configuration of computation modules directly determine the peak computing power, parallelism, and energy efficiency of the AI ​​chip, and are important indicators of chip architecture design.

[0052] In this embodiment, the calculation module 3 is used to perform calculations on the current calculation task according to the calculation parameters and the first output result to obtain a second output result, write the second output result into the cache, and attach the occupancy priority of the second output result. The scheduling module 1 is further used to, in response to the completion of the current calculation task, set the occupancy priority of the second output result to a first target priority, and adjust the occupancy priority of the first output result to a second target priority, wherein the second target priority is the priority that allows data to be replaced.

[0053] In this embodiment, the first target priority and the second target priority are different levels in a multi-level priority system, which includes at least two levels. This embodiment uses a three-level priority system as an example, where the priority order of the first, second, and third levels decreases sequentially. The first level priority is the highest occupancy level priority, meaning it is less likely to be replaced; the second level priority is the middle occupancy level priority, meaning it is normally replaced; and the third level priority is the lowest occupancy level priority, meaning it is more likely to be replaced. In this case, the first target priority can be the first priority, and the second target priority can be either the second or third priority.

[0054] Therefore, when data in the cache needs to be replaced, the system first checks if there is data with a third priority. If so, the third-priority data is replaced. If there is no third-priority data, the system checks if there is data with a second priority. If so, the second-priority data is replaced. Only if neither the second nor the third priority data exists will the system consider replacing the first-priority data. By reasonably setting the cache capacity, the first-priority data will not be replaced under normal circumstances. Therefore, this embodiment of the invention refers to it as the priority that is not allowed to be replaced.

[0055] In summary, during the computation of the previous task, the first output result of the previous task is stored in a cache. Simultaneously, when outputting the first output result, its occupancy priority is assigned, setting it to the first target priority. After the previous task completes, the scheduling module sets the first target priority to a level that cannot be replaced, ensuring that the first output result is not replaced before it is fully utilized.

[0056] In response to the start of the current computing task, the data transfer module 2 migrates the computing parameters required by the current computing task to the cache. The computing module 3 then obtains the second output result based on the first output result of the previous computing task and the computing parameters of the current computing task.

[0057] During the computation of the current computation task, the second output result of the current computation task is stored in the cache. At the same time, the occupancy priority of the second output result is attached when the second output result is output. After the current computation task is completed, the scheduling module sets the occupancy priority of the second output result to the first target priority and sets it to the priority that cannot be replaced, so as to ensure that the second output result is not replaced before it is fully used.

[0058] It should be noted that, in the embodiments of the present invention, when calculating the current calculation task, the second output result needs to be set to not be replaced during the calculation process.

[0059] In this embodiment, Cache 4 is a small-capacity, high-speed SRAM (Static Random Access Memory) storage structure located between computing module 3 and off-chip main memory, used to alleviate the memory wall conflict between processor computing power and memory bandwidth. Caches are typically structured in a multi-level hierarchy based on their distance from the computing module and their size, including L1 cache, L2 cache, and last-level cache. L1 cache has a small capacity and high speed, usually privately owned by each computing core, and is used to store frequently accessed instructions and data by the current thread. L2 cache has a moderate capacity and is either core-private or shared by a group of cores, serving as a second-level buffer after an L1 cache miss. Last-level cache has a large capacity and is shared by all computing modules on the hardware accelerator, serving as the last buffer before data leaves the chip.

[0060] In some embodiments, a suitable LLC capacity is selected through kernel profiling to cover as much kernel data reuse as possible.

[0061] In some embodiments, the LLC capacity is set to be greater than the sum of the input data, calculation parameters, and output data of any layer to improve data reusability.

[0062] Furthermore, in certain computational tasks, if a portion of the second output depends only on the corresponding portion of the first output and is independent of other portions (e.g., matrix multiplication, element-wise operations, depthwise convolution), this embodiment of the invention can also dynamically adjust the first output during the computation of the current task. That is, if the computation type of the current task conforms to a predetermined dynamic adjustment type, the priority of the first output is dynamically adjusted during the computation of the current task. The predetermined dynamic adjustment type includes matrix multiplication, element-wise operations, depthwise convolution, etc.

[0063] Specifically, taking matrix multiplication as an example, when computation module 3 performs calculations on the previous task, it divides the matrix of the first output result into multiple data blocks according to a predetermined data block size. Then, it combines the divided data blocks into a new matrix, with each element of the new matrix being a data block. The resulting new matrix is ​​then stored in a cache. Simultaneously, based on the matrix row numbers of the data blocks in the first output result, the priority of the first output result is divided into multiple levels and written out. Each level can include one or more matrix row numbers.

[0064] When performing calculations on the current task, the first target row of the first output result is multiplied by the calculation parameters to obtain the second target row of the second output result.

[0065] After the second target row of the second output result is calculated, its corresponding first target row will no longer be used, and the occupancy priority of the data block in the calculated first target row will be set to the second target priority.

[0066] Specifically, in response to completing the calculation of the second target row, the calculation module 3 sends a calculation completion instruction to the data transport module. The data transport module 2, before the current calculation task is completed, in response to receiving the calculation completion instruction, sets the occupancy priority of the data block in the first target row corresponding to the calculation completion instruction to the second target priority.

[0067] This invention, in its embodiments, stores the first output result in a cache after the previous computation task is completed, setting its occupancy priority to a first target priority that prevents data replacement. When the current computation task begins, computation parameters are moved to the cache. The current computation task is then performed based on these parameters and the first output result. The resulting second output result is written to the cache, along with its occupancy priority. After the current computation task completes, the occupancy priority of the second output result is set back to the first target priority, and the occupancy priority of the first output result is adjusted to a second target priority that allows data replacement. Therefore, by dynamically adjusting the occupancy priority of data in the cache, the complexity of inter-layer data reuse can be reduced, improving the system's versatility and scalability.

[0068] This invention, in its embodiments, stores the first output result in a cache after the previous computation task is completed, setting its occupancy priority to a first target priority that prevents data replacement. When the current computation task begins, computation parameters are moved to the cache. The current computation task is then performed based on these parameters and the first output result. The resulting second output result is written to the cache, along with its occupancy priority. After the current computation task completes, the occupancy priority of the second output result is set back to the first target priority, and the occupancy priority of the first output result is adjusted to a second target priority that allows data replacement. Therefore, by dynamically adjusting the occupancy priority of data in the cache, the complexity of inter-layer data reuse can be reduced, improving the system's versatility and scalability.

[0069] Figure 3 This is a flowchart of the data caching method according to the first embodiment of the present invention. Figure 3 As shown, the data caching method of this embodiment includes the following steps: Step S110: In response to the start of the current computing task, the computing parameters of the current computing task are moved to the cache.

[0070] In this embodiment, during the computation of the previous computation task, the first output result of the previous computation task is stored in the LLC cache. Simultaneously, the occupancy priority of the first output result is assigned along with the output result. After the previous computation task is completed, the scheduling module sets the occupancy priority of the first output result to a first target priority. This first target priority is a priority that cannot be replaced, ensuring that the first output result is not replaced before it is fully utilized.

[0071] Figure 4 This is a schematic diagram of the storage state according to an embodiment of the present invention. Figure 4 As shown, each box represents a cache line in the cache. A cache line is the unit of data exchanged between the cache and main memory, and it represents the basic granularity for cache storage and management. When accessing a memory address, the hardware does not simply load the single byte or word corresponding to that address; instead, it loads the data into the cache in units of cache lines, leveraging the principle of spatial locality to improve the hit rate of subsequent accesses.

[0072] exist Figure 4 In the illustrated embodiment, the current computation task is kernel2 and the previous computation task is kernel1. The first output result of the previous computation task is A1; the computation parameter of the current computation task kernel2 is W2, and the second output result is A2.

[0073] Where T1 represents the state of the previous computation task kernel1 during the computation process, such as... Figure 4 As shown, the first output result A1 of the previous computation task kernel1 is stored in the LLC cache and occupies 4 cache lines. Simultaneously, when the first output result is output, it is assigned a priority of P1. At this point, P1 is merely an identifier of priority; the mechanism for determining whether replacement is allowed has not yet been determined.

[0074] T2 represents the state of the previous computation task kernel1 after it has completed its computation. Specifically, after the previous computation task kernel1 has completed its computation, the occupancy priority P1 of the first output result A1 is set to the first target priority, where the first target priority is the priority that cannot be replaced, to ensure that the first output result A1 is not replaced before it is fully used.

[0075] Furthermore, the current computation task can be executed after the previous computation task has been completed. In response to the start of computation of the current computation task, the computation parameters of the current computation task are moved to the LLC cache.

[0076] Specifically, when the current computation task begins, cache lines are freely allocated for the computation parameters corresponding to the current computation task. Further, if there are free cache lines, the computation parameters can be stored in the free cache lines; if there are no free cache lines, data with lower priority must be replaced according to their occupancy priority, while data that cannot be replaced is retained.

[0077] As mentioned above, the current computation task is kernel2, and its corresponding computation parameter is W2. Combined with... Figure 4 In state T3, when the current computation task kernel2 starts computation, cache lines are freely allocated for the computation parameter W2 corresponding to the current computation task. Further, if there are free cache lines, the computation parameter can be stored in the free cache lines; if there are no free cache lines, data with lower priority needs to be replaced according to their occupancy priority, while data that cannot be replaced (such as the first output result A1) is retained.

[0078] Step S120: Calculate the current calculation task according to the calculation parameters and the first output result to obtain the second output result.

[0079] In this embodiment, the computing module executes relevant computing processes according to the instructions of the current computing task to obtain a second output result. The first output result is the output result of the previous computing task, and it is stored in the cache with a priority set to a first target priority, which is a priority that prevents data from being replaced.

[0080] Furthermore, after moving the computation parameters of the current computation task to the LLC cache, the computation module reads the required computation parameters and the first output result from the LLC, and then performs computation on the current computation task according to the computation parameters and the first output result to obtain the second output result.

[0081] Step S130: Write the second output result into the cache, along with the occupancy priority of the second output result.

[0082] In this embodiment, during the calculation of the current computing task, the second output result is written into the cache, along with the occupancy priority of the second output result.

[0083] Combination Figure 4 The state T4 shown represents the current state of the computation task kernel2 during the computation process. Figure 4 As shown, during the current computation task, computational parameters W2 are continuously moved from main memory to the LLC cache. Main memory stores computational parameters and other data. Simultaneously, a second output result A2 is continuously output to the LLC cache. When outputting the second output result A2, a priority of P2 is assigned. At this point, P2 is merely an identifier of priority; the mechanism for determining whether replacement is allowed has not yet been determined. P2 and P1 are different identifiers.

[0084] In some embodiments, the LLC capacity is set to be greater than the sum of the input data, calculation parameters, and output data of any layer; that is, the capacity of A1+A2+W2 is less than the total capacity of the LLC. In this case, the LLC can accommodate complete A1, A2, and W2. When writing W2 and A2 into the LLC, it is not necessary to replace the data in A1, A2, and W2; only other data needs to be replaced.

[0085] Step S140: In response to the completion of the current computing task, set the occupancy priority of the second output result to the first target priority, and adjust the occupancy priority of the first output result to the second target priority.

[0086] In this embodiment, the second target priority is the priority that allows data to be replaced.

[0087] exist Figure 4 In the illustrated embodiment, T5 represents the state after the current computation task kernel2 has completed its computation, and before the priority of the first and second output results is adjusted. For example... Figure 4 As shown, after the current computation task kernel2 has completed its computation, and before the priority of the first output result and the second output result is adjusted, the priority of the second output result A2 is P2. At this time, the mechanism for determining whether the second output result A2 can be replaced has not yet been determined.

[0088] T6 represents the state after the current computation task kernel2 has completed its computation, and after the priority of the first and second output results has been adjusted. For example... Figure 5 As shown, after the current computation task kernel2 completes its computation, the priority of the second output result is set to the first target priority, and the priority of the first output result is adjusted to the second target priority. Therefore, when performing the next computation task, both the first output result A1 and the computation parameter W2 are data that can be replaced, while the second output result A2 is data that cannot be replaced.

[0089] This invention, in its embodiments, stores the first output result in a cache after the previous computation task is completed, setting its occupancy priority to a first target priority that prevents data replacement. When the current computation task begins, computation parameters are moved to the cache. The current computation task is then performed based on these parameters and the first output result. The resulting second output result is written to the cache, along with its occupancy priority. After the current computation task completes, the occupancy priority of the second output result is set back to the first target priority, and the occupancy priority of the first output result is adjusted to a second target priority that allows data replacement. Therefore, by dynamically adjusting the occupancy priority of data in the cache, the complexity of inter-layer data reuse can be reduced, improving the system's versatility and scalability.

[0090] It should be noted that when the LLC capacity is greater than the sum of the input data, calculation parameters, and output data of any layer, through... Figure 3 The data caching process shown can achieve a high reuse rate. When the LLC capacity is less than the sum of the input data, calculation parameters, and output data, this embodiment of the invention can further improve the negative data reuse rate by dynamically adjusting the capacity.

[0091] Specifically, Figure 5 This is a flowchart of the data caching method according to the second embodiment of the present invention. Figure 5 As shown, the data caching method of this embodiment includes the following steps: Step S210: In response to the start of the current computing task, the computing parameters of the current computing task are moved to the cache.

[0092] Step S220: Calculate the current calculation task according to the calculation parameters and the first output result to obtain the second output result. The first output result is the output result of the previous calculation task. The first output result is stored in the cache and its occupancy priority is set to the first target priority. The first target priority is the priority that prevents data from being replaced.

[0093] Step S230: Write the second output result into the cache, along with the occupancy priority of the second output result.

[0094] The specific implementation of steps S210-S230 can be referred to steps S110-S130, and will not be repeated here in this embodiment of the invention.

[0095] Step S240: Set the occupancy priority of the data block in the first target row that has been calculated to the second target priority.

[0096] In this embodiment, the first output result and / or the second output result are matrices.

[0097] When performing calculations on the previous computational task, the matrix yielding the first output result is divided into multiple data blocks according to a predetermined block size. Then, the divided data blocks are combined to form a new matrix, with each element of the new matrix representing a data block. Finally, the resulting new matrix is ​​stored in a cache.

[0098] For example, assuming the first output is a 1024×1024 matrix and the predetermined data block size is 16×16, the first output can be divided into 64×64 data blocks. The resulting new matrix is ​​also 64×64, with each element in the new matrix being a 16×16 data block.

[0099] Taking matrix multiplication as an example, the first and / or second output results are matrices, and the elements of the matrices are data blocks. During the computation of the current task, it is often necessary to calculate row by row of the matrix. Using matrix multiplication as an example, the first target row of the first output result is multiplied by the computation parameters to obtain the second target row of the second output result. That is, the i-th row of the first output result is multiplied by the computation parameters to obtain the i-th row of the second output result, where i = 1, 2, ..., N. N is the number of rows in the matrix.

[0100] After a calculation is completed for a certain row, the data block of that row in the first output result will no longer be used. Consider releasing the corresponding space so that the calculation parameters or the data in the second output result can use this space to improve the reuse rate of the cache.

[0101] Therefore, in this embodiment of the invention, when performing calculations on the previous calculation task, the occupancy priority of the first output result is divided into multiple levels according to the spatial coordinates of the first output result data block.

[0102] Specifically, based on the matrix row numbers of the data blocks in the first output result, the occupancy priority of the first output result is divided into multiple levels and written out. Each level may include one or more matrix row numbers.

[0103] by Figure 6 Let's take an example to illustrate. Figure 6 This is a schematic diagram of matrix operations according to an embodiment of the present invention. For example... Figure 6 As shown, the calculation module multiplies the first output result A1 and the calculation parameter W2 to obtain the second output result A2.

[0104] Specifically, for the first output result A1, the calculation module divides the occupancy priority of the first output result A1 into multiple levels according to the number of matrix rows. For example, if the occupancy priority of the first output result A1 is P1 and the number of matrix rows is 4, and each level includes a matrix row number, the occupancy priority of the first output result A1 can be divided into four levels: P1_1, P1_2, P1_3, and P1_4. That is, P1_1, P1_2, P1_3, and P1_4 belong to the same occupancy priority P1, but are at different levels. After outputting the first output result A1, the scheduling module sets the occupancy priority P1 to the first target priority, which is the priority at which data cannot be replaced.

[0105] During matrix multiplication, each element in the second output result A2 depends only on the elements of the corresponding row in the first output result A1 and the data of the corresponding column in W2. For example, block0 in the second output result A2 depends only on the first row in the first output result A1 and the first column in W2.

[0106] Therefore, in this embodiment of the invention, calculating the current calculation task according to the calculation parameters and the first output result to obtain the second output result includes: multiplying the first target row of the first output result with the calculation parameters to obtain the second target row of the second output result.

[0107] After the second target row of the second output result is calculated, its corresponding first target row will no longer be used, and the occupancy priority of the data block in the calculated first target row will be set to the second target priority.

[0108] For example, suppose the first target line is the first row of the first output result A1, and the second target line is the first row of the second output result A2. After calculating the second target line, the first row of the first output result A1 will no longer be used, and the occupancy priority P1_1 of the data block of the first row of the first output result A1 will be set to the priority of the second target line. The second target priority is the priority that allows data to be replaced. Therefore, in subsequent calculations, the calculation module can safely release the cache space occupied by data in the first output result that is no longer needed, and use it to store other data.

[0109] It should be noted that, for ease of explanation, the above description uses matrix multiplication as an example, but this embodiment of the invention is not limited to this. Other calculations are also applicable to the row-by-row release method of this embodiment, depending on the data dependency pattern of the operator. If a part of the second output result depends only on the corresponding part of the first output result and is independent of other parts, then row-by-row release of this embodiment is applicable. For example, element-wise operations and depthwise convolution are also applicable to row-by-row release of this embodiment.

[0110] Step S250: In response to the completion of the current computing task, the occupancy priority of the second output result is set to the first target priority.

[0111] In this embodiment, in response to the completion of the current computing task, the occupancy priority of each data block of the first output result has already been set to the second target priority and does not need to be adjusted further. Therefore, the occupancy priority of the second output result is set to the first target priority.

[0112] This invention, in its embodiments, stores the first output result in a cache after the previous computation task is completed, setting its occupancy priority to a first target priority that prevents data replacement. When the current computation task begins, computation parameters are moved to the cache. The current computation task is then performed based on these parameters and the first output result. The resulting second output result is written to the cache, along with its occupancy priority. After the current computation task completes, the occupancy priority of the second output result is set back to the first target priority, and the occupancy priority of the first output result is adjusted to a second target priority that allows data replacement. Therefore, by dynamically adjusting the occupancy priority of data in the cache, the complexity of inter-layer data reuse can be reduced, improving the system's versatility and scalability.

[0113] Figure 7 This is a schematic diagram of a data caching device according to an embodiment of the present invention. Figure 7 As shown, the data caching device of this embodiment includes a calculation parameter moving unit 71, a data calculation unit 72, a data writing unit 73, and a priority adjustment unit 74. The calculation parameter moving unit 71 moves the calculation parameters of the current calculation task to the cache in response to the start of the current calculation task. The data calculation unit 72 performs calculations on the current calculation task based on the calculation parameters and a first output result to obtain a second output result. The first output result is the output result of the previous calculation task. The first output result is stored in the cache and its occupancy priority is set to a first target priority, which is a priority that prevents data from being replaced. The data writing unit 73 writes the second output result to the cache, along with the occupancy priority of the second output result. The priority adjustment unit 74, in response to the completion of the current calculation task, sets the occupancy priority of the second output result to the first target priority and adjusts the occupancy priority of the first output result to a second target priority, which is a priority that allows data to be replaced.

[0114] This invention, in its embodiments, stores the first output result in a cache after the previous computation task is completed, setting its occupancy priority to a first target priority that prevents data replacement. When the current computation task begins, computation parameters are moved to the cache. The current computation task is then performed based on these parameters and the first output result. The resulting second output result is written to the cache, along with its occupancy priority. After the current computation task completes, the occupancy priority of the second output result is set back to the first target priority, and the occupancy priority of the first output result is adjusted to a second target priority that allows data replacement. Therefore, by dynamically adjusting the occupancy priority of data in the cache, the complexity of inter-layer data reuse can be reduced, improving the system's versatility and scalability.

[0115] Figure 8 This is a schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 8 includes a server, a terminal, etc. Figure 8 As shown, the electronic device 8 includes at least one processor 81; a memory 82 communicatively connected to at least one processor 81; and a communication component 83 communicatively connected to a scanning device, wherein the communication component 83 receives and transmits data under the control of the processor 81; wherein the memory 82 stores instructions executable by at least one processor 81, the instructions being executed by at least one processor 81 to implement the above-described data caching method.

[0116] Specifically, the electronic device includes: one or more processors 81 and a memory 82. Figure 8 Taking a processor 81 as an example, the processor 81 and the memory 82 can be connected via a bus or other means. Figure 8 Taking a bus connection as an example, memory 82, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 81 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 82, thus implementing the aforementioned data caching method.

[0117] The memory 82 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, the memory 82 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 82 may optionally include memory remotely located relative to the processor 81, and these remote memories can be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0118] One or more modules are stored in memory 82, and when executed by one or more processors 81, they perform the data caching method in any of the above method embodiments.

[0119] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

[0120] This invention, in its embodiments, stores the first output result in a cache after the previous computation task is completed, setting its occupancy priority to a first target priority that prevents data replacement. When the current computation task begins, computation parameters are moved to the cache. The current computation task is then performed based on these parameters and the first output result. The resulting second output result is written to the cache, along with its occupancy priority. After the current computation task completes, the occupancy priority of the second output result is set back to the first target priority, and the occupancy priority of the first output result is adjusted to a second target priority that allows data replacement. Therefore, by dynamically adjusting the occupancy priority of data in the cache, the complexity of inter-layer data reuse can be reduced, improving the system's versatility and scalability.

[0121] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0122] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data caching method, characterized in that, The method includes: In response to the start of the current computing task, the computing parameters of the current computing task are moved to the cache; The current computing task is calculated based on the calculation parameters and the first output result to obtain a second output result. The first output result is the output result of the previous computing task. The first output result is stored in the cache and its occupancy priority is set to a first target priority, which is the priority that prevents data from being replaced. Write the second output result into the cache, along with the occupancy priority of the second output result; In response to the completion of the current computing task, the occupancy priority of the second output result is set to the first target priority, and the occupancy priority of the first output result is adjusted to the second target priority, where the second target priority is the priority that allows the data to be replaced.

2. The method according to claim 1, characterized in that, The first output result and / or the second output result are matrices, and the elements of the matrices are data blocks.

3. The method according to claim 2, characterized in that, The step of calculating the current calculation task based on the calculation parameters and the first output result to obtain the second output result includes: Multiply the first target row of the first output result by the calculation parameter to obtain the second target row of the second output result.

4. The method according to claim 3, characterized in that, The method further includes: Before the current computing task is completed, the occupancy priority of the data block in the first target row that has been computed is set to the second target priority.

5. A data caching system, characterized in that, The system includes a scheduling module, a data transport module, a computing module, and a cache, wherein: The scheduling module is used to set the occupancy priority of the first output result of the previous computing task to a first target priority in response to the completion of the previous computing task. The first target priority is the priority that prevents the data from being replaced. The data transfer module is used to move the calculation parameters of the current computing task to the cache in response to the start of the current computing task. The calculation module is used to perform calculations on the current calculation task according to the calculation parameters and the first output result to obtain a second output result, write the second output result into the cache, and attach the occupancy priority of the second output result; The scheduling module is also configured to, in response to the completion of the current computing task, set the occupancy priority of the second output result to a first target priority, and adjust the occupancy priority of the first output result to a second target priority, wherein the second target priority is the priority that allows data to be replaced.

6. The system according to claim 5, characterized in that, The first output result and / or the second output result are matrices, and the elements of the matrices are data blocks.

7. The system according to claim 6, characterized in that, The calculation module is used to multiply the first target row of the first output result with the calculation parameter to obtain the second target row of the second output result.

8. The system according to claim 7, characterized in that, The calculation module is also configured to send a calculation completion instruction to the data transport module in response to the completion of the calculation of the second target row.

9. The system according to claim 8, characterized in that, The data transport module is further configured to, in response to receiving a computation completion instruction, set the occupancy priority of the data block in the first target row corresponding to the computation completion instruction to the second target priority before the current computation task is completed.

10. A data caching device, characterized in that, The device includes: The calculation parameter moving unit is used to move the calculation parameters of the current calculation task to the cache in response to the start of the current calculation task. The data calculation unit is used to perform calculations on the current calculation task according to the calculation parameters and the first output result to obtain a second output result. The first output result is the output result of the previous calculation task. The first output result is stored in the cache and its occupancy priority is set to a first target priority, which is a priority that prevents data from being replaced. A data writing unit is used to write the second output result into the cache, and includes the occupancy priority of the second output result; The priority adjustment unit is used to, in response to the completion of the current computing task, set the occupancy priority of the second output result to a first target priority, and adjust the occupancy priority of the first output result to a second target priority, wherein the second target priority is the priority that allows data to be replaced.

11. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-4.

12. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is run on a computer, the computer performs the method according to any one of claims 1-4.

13. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method as described in any one of claims 1-4.

Images