A data cache architecture

Abstract

This invention relates to the field of chip technology, and more particularly to a data caching architecture, including a cache control module, N cache queues, and a shared cache; the shared cache includes M storage units; the cache control module is used to sequentially acquire data to be cached, A x For the x-th cached data, A x ={A1 x A2 x A3 x}, A1 x For A x The corresponding cache queue identifier, A2 x For A x The corresponding first and last valid data information, A3 x For A x The corresponding cached data; the cache control module is used to retrieve A x At that time, retrieve a valid memory cell B from the shared cache whose status is invalid. x , A3 x Stored in B x In the middle, B x The valid identifier state is set to valid, and an A is generated. x The corresponding queue cache information {D x E x ,F x}, store in A1 x The corresponding cache queue. This invention improves the chip's energy efficiency and performance.

Description

Technical Field

[0001] This invention relates to the field of chip technology, and in particular to a data caching architecture. Background Technology

[0002] Caching technology is an indispensable part of chip design, playing a crucial role in improving system performance. Through proper design and optimization, chip operating efficiency can be significantly improved. Caching technology is the core of chip design, directly impacting chip performance, power consumption, and cost. From simple direct mapping to complex multi-level coherence protocols, cache optimization is an important direction in computer architecture research. In traditional data caching architectures, a fixed, independent cache space is typically allocated to each queue. However, because the access load of each queue often exhibits dynamic fluctuations, it is difficult for each queue's cache space to remain completely full at all times. This leads to a mismatch between the static allocation of cache resources and actual dynamic demands, resulting in significant resource utilization problems in this data caching architecture. As the system scales up and the number of queues increases, the shortcomings of this data caching architecture become further amplified: on the one hand, a large amount of underutilized cache space results in significant hardware area waste; on the other hand, idle resources cannot be dynamically allocated and used by other high-load queues, causing overall cache efficiency to decline with the increase in the number of queues. This problem of inefficient resource utilization is particularly prominent in high-performance computing scenarios that require supporting a large number of parallel processing queues, becoming a major bottleneck restricting chip performance improvement.

[0003] While some existing data caching architectures employ multi-queue shared cache solutions, their designs have certain limitations. Specifically, these architectures are typically optimized only for single-instruction operations. When encountering continuous data access such as burst data, data from different queues can interleave unexpectedly in the cache. This data interleaving not only violates the original intention of cache architecture design to maintain the continuity and locality of data access but also fails to meet the performance requirements of high-throughput data processing. Furthermore, these defects can lead to cache prefetching mechanism failure, increased memory access latency, and potentially additional overhead for cache consistency maintenance, ultimately impacting the overall chip's energy efficiency and performance. These limitations are particularly pronounced in applications requiring efficient processing of burst data streams, such as modern high-performance computing and AI inference. Therefore, providing an efficient and flexible data caching architecture to improve chip energy efficiency and performance is a pressing technical challenge. Summary of the Invention

[0004] The purpose of this invention is to provide a data caching architecture that improves the energy efficiency and performance of the chip.

[0005] This invention provides a data caching architecture, including a cache control module, N cache queues, and a shared cache shared by the N cache queues; The shared cache includes M storage units. The valid identifier status corresponding to each storage unit is set to invalid by default. If cached data is stored, the valid identifier status corresponding to the storage unit is set to valid. The cache control module is used to sequentially obtain the data to be cached {A1, A2, ..., A...} x ,...,A X}, A x Let A be the x-th cached data, where x ranges from 1 to X, and X is the total number of data to be cached. x ={A1 x A2 x A3 x}, A1 x For A x The corresponding cache queue identifier, A2 x For A x The corresponding first and last valid information of the data, the first and last valid information of the data are empty or include the first valid information and / or the last valid information, A3 x For A x The corresponding cached data, A3 x Data whose length is less than or equal to the length of the storage unit, belonging to the same continuous data A x In A1, A2, ..., A x ,...,A X Arranged in chronological order, cache queues A, with the same identifier but not belonging to the same consecutive data, are... x In A1, A2, ..., A x ,...,A X No intersections; The cache control module is used to obtain A x At that time, retrieve a valid memory cell B from the shared cache whose status is invalid. x , A3 x Stored in B x In the middle, B x The valid identifier state is set to a valid state, and based on B x and A2 x Generate A x The corresponding queue cache information {D x E x ,F x}, will {D x E x ,F x} Store in A1 xIn the corresponding cache queue, D x For B x The corresponding pointer, E x For A x The corresponding first identifier, F x For A x The corresponding last identifier, E x F x The default setting is the second identifier, if A2 x Including the first valid piece of information, then E x Set as the first identifier, if A2 x If it is empty, then E x F x All are set as the second identifier, if A2 x Including the last valid information, then F x Set as the first identifier.

[0006] Compared with existing technologies, this invention has significant advantages and beneficial effects. Through the above technical solution, the data caching architecture provided by this invention achieves considerable technological advancement and practicality, and has broad industrial application value, possessing at least the following beneficial effects: The cache architecture described in this invention can flexibly set the number of cache queues, and all cache queues share a common cache. In each cache queue, only the pointer, first bit identifier, and last bit identifier of the corresponding storage unit in the shared cache need to be stored in order. In the shared cache, there is no need to store data in order, which can efficiently cache data and maintain the continuity and locality of data access, thereby improving the energy efficiency ratio and performance of the chip. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 A schematic diagram of a data caching architecture provided in an embodiment of the present invention; Figure 2 A schematic diagram of a data caching architecture provided for another embodiment of the present invention. Detailed Implementation

[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0010] This invention provides a data caching architecture, such as... Figure 1 As shown, the system includes a cache control module, N cache queues, and a shared cache shared by the N cache queues. It should be noted that the number N can be flexibly set according to specific application requirements, and the number of cache queues can also be increased or decreased, achieving a flexible and scalable cache architecture. Specifically, N cache queues can be set according to different cache data types, with one cache queue corresponding to each cache data type. Alternatively, N cache queues can be set according to different clients sending cached data, with one cache queue corresponding to each client. N cache queues can also be set according to the priority of the cached data, with one cache queue corresponding to each priority level. Furthermore, N cache queues can be set according to the source identifier (srcid) corresponding to the cached data, with one cache queue corresponding to each source identifier.

[0011] The shared cache comprises M storage units, each of equal size. The validity flag for each storage unit is set to invalid by default, indicating that no cached data has been stored. If cached data is stored, the validity flag for the storage unit is set to valid. As an example, a valid indicator can be used to identify whether a storage unit currently stores cached data. When a storage unit stores cached data, the valid flag is set, indicating a valid state. When a storage unit stores no cached data, the valid flag is not set, indicating an invalid state.

[0012] The cache control module is used to sequentially obtain the data to be cached {A1, A2, ..., A...} x ,...,A X}, A x Let A be the x-th cached data, where x ranges from 1 to X, and X is the total number of data to be cached. x ={A1 x A2 x A3 x}, A1 x For A x The corresponding cache queue identifier, A2 x For A xThe corresponding first and last valid data information. The first and last valid data information may be empty or may include the first valid data information and / or the last valid data information. As an example, if A... x If a data point is only the starting point of a continuous data sequence and is not the ending point of that sequence, then the length of that continuous data sequence is greater than or equal to 2, meaning it includes two or more A's. x In this case, A2 x Only the first valid piece of information is included. If A x If a data point is the end of a consecutive data set but not the beginning of that set, then the length of that consecutive data set is greater than or equal to 2, meaning it contains two or more A's. x In this case, A2 x Only the last valid digit is included. If A x If a data point is both the start and end point of a continuous data sequence, then the length of that sequence is 1, meaning it contains only one A. x Then A2 x The information includes both the first and last valid pieces of information. If A x If A2 is neither the end nor the beginning of a continuous data set, then A2... x Empty. A3 x For A x The corresponding cached data, A3 x The data length is less than or equal to the length of the storage unit, preferably A3. x The data length is equal to the length of the storage unit, and each A x A storage unit needs to be determined to store the corresponding cached data.

[0013] It should be noted that X pieces of data to be cached can correspond to data in N cache queues, and it is not required that every consecutive piece of data be in A1, A2, ..., A x ,...,A X For a sequence to be considered continuous, it only needs to satisfy the condition that A belongs to the same continuous data set. x In A1, A2, ..., A x ,...,A X Arranged in chronological order, cache queues A, with the same identifier but not belonging to the same consecutive data, are... x In A1, A2, ..., A x ,...,A X There is no overlap between cached data, meaning that cached data in the same cache queue must be sent continuously before the next one can be sent. There is no order requirement between cached data in different cache queues, and they can be transmitted in an interleaved manner.

[0014] The cache control module is used to obtain A xAt that time, retrieve a valid memory cell B from the shared cache whose status is invalid. x B x That is, A x The corresponding storage unit. A3 x Stored in B x In the middle, B x The valid identifier state is set to a valid state, and based on B x and A2 x Generate A x The corresponding queue cache information {D x E x ,F x}, will {D x E x ,F x} Store in A1 x In the corresponding cache queue, D x For B x The corresponding pointer, based on B x The corresponding pointer can determine B x Location. E x For A x The corresponding first identifier, F x For A x The corresponding last digit identifier. E x F x The default setting is the second identifier, if A2 x Including the first valid piece of information, then E x Set as the first identifier, if A2 x If it is empty, then E x F x All are set as the second identifier, if A2 x Including the last valid information, then F x Set as the first identifier. E x Setting it to the first identifier indicates that the corresponding cached data is the first data point of a consecutive data set, F x Setting it to the first identifier indicates that the corresponding cached data is the last element of a consecutive data set. If E x Setting it to the second flag indicates that the corresponding cached data is not the first data of a consecutive data set. If F x Setting it to the second identifier indicates that the corresponding cached data is not the last element of a consecutive data set. Based on E... x F x The combination of identifiers can determine which part of the cached data corresponds to which data. If E x F x If both are the first identifier, it means that the corresponding cached data is both the first and last data, and this consecutive data corresponds to only one A. x The data length is 1. If Ex As the first identifier, F x If E is the second identifier, then the corresponding cached data is the first data point of a contiguous data set, and the length of this contiguous data set is greater than or equal to 2. x For the second identifier, F x If E is the first identifier, then the corresponding cached data is the last element of a consecutive data entry, and the length of this consecutive data entry is greater than or equal to 2. x For the second identifier, F x If the second identifier is used, the corresponding cached data is the intermediate data of a continuous data set, and the length of this continuous data set is greater than or equal to 3. This can be achieved by setting E... x F x It realizes intelligent identification and management of cached data, and sets E x F x The dual-flag mechanism not only accurately describes the boundary characteristics of the same continuous data, but also intelligently identifies special data situations through the combination of flag states, providing complete information support for subsequent cached data reading and processing. As an example, the first flag can be set to 1, and the second flag to 0. By combining the dual-flag mechanism with pointers, efficient management and fast access to cached data are achieved. When the length of a continuous data item is only 1, that is, when the data is only a single data unit, E... x F x The tag is also the first identifier, which further simplifies the cache data management logic and improves the overall efficiency of the chip.

[0015] The queue cache information {D} described in the embodiments of the present invention x E x ,F x The structured storage method enables precise location and efficient management of cached data, ensuring both the accuracy of cached data access and providing sufficient status information to downstream processing modules, thereby optimizing the efficiency of the entire cached data processing pipeline. Pointer D x The synchronous storage design realizes the mapping between the physical storage of the shared cache and the logical management of the cache queue, providing flexible and reliable cache data access capabilities.

[0016] As one embodiment, the continuous data is burst data, which has the characteristic of continuity.

[0017] The cache architecture of this invention breaks through the limitations of traditional cache design by introducing a novel queue management mechanism, enabling flexible expansion from a single queue to multiple queues (1 queue to N queues). This cache design not only effectively processes cached data but also possesses high flexibility and scalability. By supporting dynamic expansion from a single queue to multiple queues, this cache architecture can adapt to cached data processing needs of varying scales and complexities. Furthermore, by sharing cache entities, this cache architecture significantly improves cache space utilization and reduces resource waste caused by increasing the number of queues, thereby achieving higher performance and efficiency within a limited chip area. It is better suited for high-performance computing, data-intensive applications, and complex system design scenarios.

[0018] As one embodiment, the cache control module is used to obtain A x At this time, each memory unit is traversed sequentially, specifically according to its index in ascending order. If the valid identifier of the currently traversed memory unit is invalid, then the currently traversed memory unit is designated as A. x Corresponding storage unit B x The iteration will not continue.

[0019] If all storage units have been traversed and all valid identifiers are in a valid state, it indicates that the shared cache is full. The cache control module then sends a backpressure signal to the upstream module sending the data to be cached when all valid identifiers for the storage units in the shared cache are in a valid state. This forcibly pauses the transmission of all data to be cached, thus preventing data overflow or loss. This mechanism effectively achieves dynamic management of flow control and storage resources, ensuring the reliability of cached data access while also enabling coordinated throttling between upstream and downstream modules through the backpressure mechanism.

[0020] As one embodiment, the cache control module is also used to send a backpressure signal based on the nth cache queue to the upstream module that sends the data to be cached when not all the valid identifier states corresponding to the storage units in the shared cache are valid and the nth cache queue is full, so as to achieve local backpressure without affecting other cache queues.

[0021] As one embodiment, the data caching architecture further includes an arbitration module and a cached data reading module, such as... Figure 2As shown, the arbitration module's scheduling strategy has clear timing constraints for taking effect: when a cache queue has been granted permission and is reading cached data from the shared cache, the scheduling algorithm will pause its evaluation of that cache queue, at which point the shared cache resources are exclusively used by that cache queue. Only when no cache queue is in a data reading state will the arbitration module's scheduling strategy take effect in the first queue cache unit of the pending queue. x Only when it is the first identifier is it eligible to participate in scheduling. Specifically, the arbitration module is used to select the first queue cache unit of a cache queue, E, when no cache queue is currently in a data reading state. x The cache queue with the first identifier is used as the current cache queue to be read, and the D in the queue cache unit in the cache queue to be read is read in sequence. x And send it to the data reading module until F in the cache queue is empty. x D as the first identifier x The data is sent to the data reading module to pause reading from the currently pending cache queue. This means that after the pause, the cache queue is no longer a pending cache queue. It's important to note that once a cache queue successfully obtains authorization from the arbitration module, its data reading process remains continuous—it must be processed sequentially from the first data to the last data, and cannot be preempted by other cache queues during this period. The resource occupation of this cache queue is only released after it has completed reading all currently cached data. Afterward, it must wait for the next time it regains authorization through the arbitration module before it can restart the cache data reading operation. This mechanism design fully considers both the integrity of cached data processing and the fairness of scheduling: it avoids transmission fragmentation caused by frequent switching of cache queues, and ensures balanced resource allocation among the cache queues in long-term operation through strict admission conditions (triggered by the first data) and release conditions (completed by the last data). The cache data reading module is used to sequentially read data based on each received D... x Read the cached data from the corresponding storage unit in the shared cache and output it, and set the valid identifier status of the storage unit after reading the cached data to invalid status.

[0022] To ensure timing requirements are met, as one embodiment, if the data caching architecture does not meet the preset timing, a preset number of delay units are inserted between the cached data read module and the shared cache. Each delay unit is used to delay the output of transmitted information by one cycle, and the preset number of delay units are configured in a pipelined manner. It should be noted that by inserting a preset number of delay units between the cached data read module and the shared cache, the effect of delaying the read cached data can be achieved by only delaying the data transmitted by the cached data read module. This satisfies the timing requirements, simplifies the timing adjustment logic, and further improves chip performance.

[0023] As one embodiment, the arbitration module is used to select the first queue cache unit of a cache queue, E, based on a round-robin scheduling strategy, when no cache queue is currently in a data reading state. x The cache queue with the first identifier is designated as the current cache queue to be read. Using a round-robin scheduling method ensures that each cache queue receives an equal opportunity to be read.

[0024] As one embodiment, the arbitration module is used to obtain the bandwidth quota (per-queuebandwidth) of each cache queue. The bandwidth quota is proportional to the length of the data already read, and when no cache queue is currently in a data reading state, it selects the first queue cache unit E. x The cache queue with the first identifier and the smallest bandwidth quota is selected as the current cache queue to be read, thereby meeting the differentiated bandwidth requirements of different cache queues.

[0025] As one embodiment, the arbitration module is used to obtain the priority of each cache queue, and when no cache queue is currently in a data reading state, it selects the first cache unit of the highest priority queue, E. x The cache queue with the first identifier is designated as the current cache queue to be read.

[0026] The arbitration module employs a configurable hybrid scheduling algorithm that intelligently balances per-request fairness and bandwidth fairness based on actual application needs. It can guarantee equal scheduling opportunities for all requests through algorithms, while also meeting the bandwidth guarantee requirements of different service flows through weighted allocation and other methods, providing the optimal scheduling solution for complex application scenarios.

[0027] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A data caching architecture, characterized in that, It includes a cache control module, N cache queues, and a shared cache shared by the N cache queues; The shared cache includes M storage units. The valid identifier status corresponding to each storage unit is set to invalid by default. If cached data is stored, the valid identifier status corresponding to the storage unit is set to valid. The cache control module is used to sequentially obtain the data to be cached {A1, A2, ..., A...} x ,...,A X }, A x Let A be the x-th cached data, where x ranges from 1 to X, and X is the total number of data to be cached. x ={A1 x A2 x A3 x }, A1 x For A x The corresponding cache queue identifier, A2 x For A x The corresponding first and last valid information of the data, the first and last valid information of the data are empty or include the first valid information and / or the last valid information, A3 x For A x The corresponding cached data, A3 x Data whose length is less than or equal to the length of the storage unit, belonging to the same continuous data A x In A1, A2, ..., A x ,...,A X Arranged in chronological order, cache queues A, with the same identifier but not belonging to the same consecutive data, are... x In A1, A2, ..., A x ,...,A X No intersections; The cache control module is used to obtain A x At that time, retrieve a valid memory cell B from the shared cache whose status is invalid. x , A3 x Stored in B x In the middle, B x The valid identifier state is set to a valid state, and based on B x and A2 x Generate A x The corresponding queue cache information {D x E x ,F x }, will {D x E x ,F x } Store in A1 x In the corresponding cache queue, D x For B x The corresponding pointer, E x For A x The corresponding first identifier, F x For A x The corresponding last identifier, E x F x The default setting is the second identifier, if A2 x Including the first valid piece of information, then E x Set as the first identifier, if A2 x If it is empty, then E x F x All are set as the second identifier, if A2 x Including the last valid information, then F x Set as the first identifier.

2. The data caching architecture according to claim 1, characterized in that, The cache control module is used to obtain A x At that time, each memory cell is traversed sequentially. If the valid identifier of the currently traversed memory cell is invalid, then the currently traversed memory cell is determined as A. x Corresponding storage unit B x .

3. The data caching architecture according to claim 1 or 2, characterized in that, The cache control module is also used to send an overall backpressure signal to the upstream module that sends the data to be cached when all the valid identifiers corresponding to the storage units in the shared cache are in a valid state.

4. The data caching architecture according to claim 1, characterized in that, The cache control module is also used to send a backpressure signal based on the nth cache queue to the upstream module that sends the data to be cached when not all the valid identifiers corresponding to the storage units in the shared cache are valid and the nth cache queue is full.

5. The data caching architecture according to claim 1, characterized in that, The data caching architecture also includes an arbitration module and a cached data reading module. The arbitration module is used to select the first cache unit of a cache queue, E, when no cache queue is currently in a data reading state. x The cache queue with the first identifier is used as the current cache queue to be read, and the D in the queue cache unit in the cache queue to be read is read in sequence. x And send it to the data reading module until F in the cache queue is empty. x D as the first identifier x Send this to the data reading module to pause reading from the current pending cache queue; The cached data reading module is used to sequentially read data based on each received D. x Read the cached data from the corresponding storage unit in the shared cache and output it, and set the valid identifier status of the storage unit after reading the cached data to invalid status.

6. The data caching architecture according to claim 5, characterized in that, If the data caching architecture does not meet the preset timing, a preset number of delay units are inserted between the cached data reading module and the shared cache. Each delay unit is used to delay the output of the transmitted information by one cycle. The preset number of delay units are set in a pipeline manner.

7. The data caching architecture according to claim 5, characterized in that, The arbitration module is used to select the first queue cache unit of a cache queue, E, based on a round-robin scheduling strategy, when no cache queue is currently in a data reading state. x The cache queue with the first identifier is designated as the current cache queue to be read.

8. The data caching architecture according to claim 5, characterized in that, The arbitration module is used to obtain the bandwidth quota for each cache queue, and when no cache queue is currently in a data reading state, it selects the first cache unit E. x The cache queue with the first identifier and the smallest bandwidth quota is selected as the current cache queue to be read.

9. The data caching architecture according to claim 5, characterized in that, The arbitration module is used to obtain the priority of each cache queue. When no cache queue is currently in a data reading state, it selects the first cache unit with the highest priority, E. x The cache queue with the first identifier is designated as the current cache queue to be read.

10. The data caching architecture according to claim 1, characterized in that, The continuous data refers to bursty data.

Images