A system and method for improving the cache utilization rate of a computing engine

Abstract

The present invention provides a system and method for improving the cache utilization of a computing engine, the system comprising an external device, an external device controller, and an inline computing engine. The present invention introduces an over-allocation mechanism, which can achieve a virtual cache that is larger than the physical cache. Through this virtual cache, the inline computing engine can support more concurrent tasks of external devices, thereby improving cache utilization and the performance of the inline computing engine. By dynamically adjusting the virtual cache size through parameters such as the current external device and memory, it is possible to avoid the need to instantiate an overly large physical cache according to the worst case scenario during chip design. It is possible to achieve the same performance while using only a smaller physical cache to adapt to various usage scenarios, effectively reducing chip costs.

Description

Technical Field

[0001] The present invention relates to the field of chip technology, and in particular to a system and method for improving cache utilization of a computing engine. Background Art

[0002] The development of big data and artificial intelligence technologies is placing ever-increasing demands on the computing power of computer systems. Consequently, a growing number of computer systems are employing specialized chips for computational acceleration. The computing engine, at the heart of computational acceleration chips, is also evolving towards faster speeds and higher memory utilization, and the cache within the computing engine is often central to achieving these performance goals.

[0003] Among them, taking a new type of inline computing engine as an example, an inline computing engine is a computing engine located between the data source and the memory in the data flow path. Compared with traditional computing engines, it can reduce the number of accesses to the memory and effectively save the demand for memory bandwidth.

[0004] Figure 1 This is the data exchange process in a traditional computing engine. Data 1 used for calculation is stored in memory, and data 2 comes from an external device. Data 2 is first written from the external device to memory. The computing engine then reads both data 1 and data 2 from memory, obtains the calculation result, and writes it back to memory. Therefore, in the most typical task described above, memory needs to be read and written twice.

[0005] When the inline calculation engine is used, the data flow is as follows Figure 2 As shown in the figure, the external device writes data 2 directly to the inline computing engine. At this point, the inline computing engine appears to the external device as a memory device. Simultaneously with receiving data 2 from the external device, the inline computing engine reads data 1 from the memory, performs the calculations, obtains the result, and then writes the result to the memory. Therefore, in this typical scenario, the memory only needs to be read and written once.

[0006] There are many ways to implement the internal implementation of an inline computing engine. Regardless of the specific implementation method, from an abstract perspective, the scenarios share the following common characteristics:

[0007] 1. Data acquisition from external devices always has a certain delay. There's a time lag between sending a command to an external device and data output. During this time lag, the system, driven by performance requirements, must continue sending commands to the device or other devices, rather than waiting for one task to complete before starting the next. Therefore, the inline computing module must support multiple concurrent tasks.

[0008] 2. For high-performance devices, tasks often complete out of order. This order is particularly difficult to guarantee when using multiple external devices. In other words, the order in which tasks are completed is often different from the order in which they were issued. For some common protocols, such as PCIe, multiple data blocks within a task may also be out of order.

[0009] Therefore, due to out-of-order processing and multitasking, the inline computing engine must only begin loading the task information corresponding to the current data after receiving data written from an external device, and then read the corresponding data from memory. 1 Inline computing cannot predict the information related to the next incoming data, which is one of the core issues facing inline computing engines. Before the task information and corresponding data are retrieved, the data written from the external device must be temporarily stored in the inline computing engine's internal cache.

[0010] Some key data flows of the inline computing engine can be abstracted as Figure 3 As mentioned above, due to multitasking and out-of-order processing, data from external devices needs to be temporarily stored in the peripheral data cache after entering the inline computing engine. While storing data, the read control module of the inline computing engine obtains information about the data written by the external device, queries the storage location of another data participating in the calculation in the system memory, and then reads another operand from the system memory. When the system memory returns the data to the inline computing engine, it will be sent to a smaller memory data cache and then sent to the computing unit together with the data from the peripheral data cache.

[0011] Due to the randomness of data written by external devices, data in system memory cannot be read in advance. Therefore, the peripheral data cache is a key component that cannot be omitted in the inline computing engine. Its usage efficiency is also directly related to the cost and performance of the inline computing engine.

[0012] Traditional cache management methods require requesting cache space before issuing data read commands to external devices. Based on the requested cache size, read commands are then issued to the external device, not exceeding that size to prevent cache overflow. Once the data to be computed in memory is read into the cache and then enters the computation module along with the external data cache, the corresponding external data cache can be reclaimed and allocated to new requests.

[0013] This is the most typical and safest method. This ensures that at any given moment, the total amount of unreturned data from read commands sent to the external device will not exceed the remaining space in the current cache. Therefore, there is no need to deal with issues caused by cache overflow, resulting in higher security and lower hardware complexity.

[0014] However, this method also has its drawbacks. When the latency of the external device increases, the above method requires more cache. Therefore, in the scenario of high-latency peripheral devices, the cost of the above method will increase significantly. The root cause of this drawback is that the cache utilization rate of this method is not high. Summary of the Invention

[0015] In view of this, the purpose of the present invention is to propose a system and method for improving the cache utilization of a computing engine. Taking a new type of inline computing engine as an example, the system and method are applicable to inline computing engines and other types of computing engines, and can improve cache utilization to solve the above-mentioned problems.

[0016] Based on the above objectives, in one aspect, the present invention provides a system for improving cache utilization of a computing engine, the system comprising:

[0017] External devices, used to send data to the inline computing engine and into the physical cache;

[0018] an external device controller, configured to send a cache allocation request to the inline computing engine, and further configured to send a read command to the external device;

[0019] The inline computing engine uses a virtual cache to increase cache capacity. The inline computing engine is used to receive data input from an external device for physical cache, and the cached data is output from the physical cache to the computing unit.

[0020] As a further solution of the present invention, the inline computing engine includes a cache allocation module, a cache capacity management module, and an external parameter monitoring module;

[0021] The cache allocation module is used to receive a cache allocation request sent by the external device controller, and perform cache allocation through the cache capacity management module based on the cache allocation request;

[0022] The cache capacity management module is provided with a virtual cache expansion and recycling module, wherein the virtual cache expansion and recycling module is provided with a virtual cache to increase the overall cache capacity of the inline computing engine;

[0023] The external parameter monitoring module is used to connect with the virtual cache expansion and reclaiming module and detect the cache allocation in the inline computing engine based on external parameters.

[0024] As a further solution of the present invention, the virtual cache and the physical cache constitute the total cache size of the inline computing engine.

[0025] As a further solution of the present invention, in the inline computing engine, the cache allocation module is further configured to query the cache capacity management module for cache status, and determine cache allocation based on the queried total cache size and used cache size.

[0026] As a further solution of the present invention, the virtual cache expansion and reclaiming module is used to record the current size of the virtual cache, evaluate the reasonable size of the virtual cache, and perform expansion and reclaim operations on the virtual cache based on these two data.

[0027] As a further solution of the present invention, the virtual cache expansion and reclaiming module is further configured to estimate a reasonable size of the virtual cache by collecting parameters of the current IO operation.

[0028] As a further solution of the present invention, the parameters of the IO operation include the read delay of the current memory data and the frequency of overflow events.

[0029] As a further solution of the present invention, when the inline computing engine section performs physical cache size calculation, the physical cache size is set to:

[0030] Physical cache size = (fixed delay + average memory delay) * design bandwidth.

[0031] As a further solution of the present invention, the inline computing engine performs delay statistics, including the following steps:

[0032] When the read control module sends a read command to the memory, the relevant parameters of the command will be written into the descriptor queue;

[0033] When the memory returns the data to be read to the memory data cache module, the timestamp of the moment when it enters the operation unit from the memory data cache module is recorded and subtracted from the timestamp of the read command to calculate the delay;

[0034] When the first data of each read command is sent to the operation unit, the descriptor queue sends the timestamp of the read command to the delay estimation module, and the delay estimation module uses the delay to estimate and adjust the cache capacity.

[0035] As a further solution of the present invention, the read control module sends a read command to the memory, and the relevant parameters of the command are the timestamp when the read command is issued and the data size of the read command.

[0036] As a further solution of the present invention, the inline computing engine section performs delay statistics and also includes monitoring the data interface of the memory data cache entering the computing unit, and the monitoring process is based on the read command issued to the memory.

[0037] As a further solution of the present invention, the size of the physical memory is designed taking into account the average memory delay, and the average memory delay is used as the delay module threshold. The method for calculating the delay module threshold parameter is:

[0038] Delay module threshold = (actual cache size - minimum cache size) ÷ designed bandwidth.

[0039] In another aspect of the present invention, a method for improving cache utilization of a computing engine is provided, which is applicable to an inline computing engine. The method comprises the following steps:

[0040] Based on the inline computing engine receiving data from external devices, the virtual cache is used to increase the cache capacity. The cache allocation module queries the cache capacity management module for cache status and determines the cache allocation based on the total cache size and used cache size.

[0041] The virtual cache expansion and recycling module records the current virtual cache size, evaluates the reasonable size of the virtual cache, and expands and recycles the virtual cache based on these two data. The reasonable size of the virtual cache is estimated by collecting the parameters of the current IO operation.

[0042] In another aspect of the present invention, a computer-readable storage medium is provided, storing computer program instructions, which, when executed, implement any of the above methods for improving cache utilization of a computing engine according to the present invention.

[0043] In another aspect of the present invention, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, any one of the above methods for improving cache utilization of a computing engine according to the present invention is executed.

[0044] Compared with the traditional implementation method, the main advantages of the present invention are:

[0045] 1. The method of the present invention introduces an over-allocation mechanism, which can realize a virtual cache that is larger than the physical cache. Through this virtual cache, the inline computing engine can support more concurrent tasks of external devices, thereby improving cache utilization and the performance of the inline computing engine.

[0046] 2. Dynamically adjusting the virtual cache size based on current external device and memory parameters can avoid the need to instantiate an overly large physical cache based on the worst-case scenario during chip design. This allows for the use of a smaller physical cache to accommodate various usage scenarios while maintaining the same performance, effectively reducing chip costs.

[0047] These and other aspects of the present application will be more clearly understood in the following description of the embodiments. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying any creative work.

[0049] In the figure:

[0050] Figure 1 This is a data flow diagram for a traditional computing engine;

[0051] Figure 2 This is a data flow diagram of a traditional inline computing engine;

[0052] Figure 3 This is a schematic diagram of the data flow principle of a traditional inline computing engine;

[0053] Figure 4 A system block diagram of virtual cache and management in a computing engine in a system for improving computing engine cache utilization according to the present invention;

[0054] Figure 5 A schematic diagram of the structure of memory data delay in a system for improving computing engine cache utilization according to the present invention;

[0055] Figure 6 A schematic diagram of delay statistics in a system for improving computing engine cache utilization according to the present invention;

[0056] Figure 7 A schematic diagram of an embodiment of a computer-readable storage medium for implementing a method for improving cache utilization of a computing engine according to the present invention;

[0057] Figure 8 A schematic diagram of the hardware structure of an embodiment of a computer device for implementing a method for improving cache utilization of a computing engine according to the present invention; DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0059] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are intended to distinguish two non-identical entities or non-identical parameters with the same name. Therefore, "first" and "second" are used for convenience of expression only and should not be understood as limitations on the embodiments of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, other steps or units inherent to a process, method, system, product, or device that includes a series of steps or units.

[0060] To make the purpose, technical solutions and advantages of the present invention more clearly understood, the following is a further detailed description of the embodiments of the present invention in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0061] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0062] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0063] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0064] Because traditional cache management methods require more cache as the latency of external devices increases, the cost of traditional cache management methods will increase significantly in scenarios with high-latency peripherals. The root cause of this shortcoming is that traditional cache management methods do not have high cache utilization.

[0065] In view of this, embodiments of the present invention provide a system and method for improving computing engine cache utilization. Cache utilization can be improved through inline computing engine cache allocation. The main features include:

[0066] Data read commands can be sent to external devices in excess. This means that before a read command is issued, the cache quota allocated for the read command is not based on the actual cache space available, but rather can exceed the actual cache size. In other words, the total amount of unreturned data from read commands sent to the external device can exceed the actual cache size.

[0067] The excess portion beyond the actual cache size is not a fixed value, but a value that is dynamically adjusted according to characteristics such as external device latency and memory read latency.

[0068] Over-allocation may cause a buffer overflow. When an overflow occurs, there are protection measures to redo the calculations involved in the overflowed data.

[0069] Compared with the traditional implementation method, the main advantages of the present invention are:

[0070] The method designed in the present invention introduces an over-allocation mechanism, which can realize a virtual cache larger than the physical cache. Through this part of the virtual cache, the inline computing engine can support more concurrent tasks of external devices, thereby improving cache utilization and the performance of the inline computing engine.

[0071] By dynamically adjusting the virtual cache size through current external device and memory parameters, it is possible to avoid the need to instantiate an overly large physical cache based on the worst-case scenario during chip design. This allows for the use of a smaller physical cache to adapt to various usage scenarios while maintaining the same performance, effectively reducing chip costs.

[0072] The present invention designs a cache allocation method for an inline engine to improve cache utilization.

[0073] In some embodiments of the present invention, see Figure 4 As shown, a system for improving the cache utilization of a computing engine is provided. The system includes an external device, an external device controller, and an inline computing engine. The key parts inside the engine can be abstracted as Figure 4 The structure in .

[0074] The external device is used to send data to the inline computing engine and enter the physical cache; the external device controller is used to send a cache allocation request to the inline computing engine and also to send a read command to the external device; the inline computing engine uses a virtual cache to increase the cache capacity, and the inline computing engine is used to receive data input by the external device for physical caching, and the cached data is output from the physical cache to the computing unit.

[0075] In addition to the physical cache of the traditional inline compute engine, a "virtual" cache is added. This virtual cache simply increases the overall cache capacity without adding additional hardware or changing the data flow path. Data from external devices still enters the physical cache and is ultimately output from the physical cache to the computing unit.

[0076] In some embodiments, the inline computing engine includes a cache allocation module, a cache capacity management module, and an external parameter monitoring module;

[0077] The cache allocation module is used to receive a cache allocation request sent by the external device controller, and perform cache allocation through the cache capacity management module based on the cache allocation request;

[0078] The cache capacity management module is provided with a virtual cache expansion and recycling module, wherein the virtual cache expansion and recycling module is provided with a virtual cache to increase the overall cache capacity of the inline computing engine;

[0079] The external parameter monitoring module is used to connect with the virtual cache expansion and reclaiming module and detect the cache allocation in the inline computing engine based on external parameters.

[0080] In this embodiment, the virtual cache and the physical cache constitute the total cache size of the inline computing engine.

[0081] In some embodiments, in the inline computing engine, the cache allocation module is further configured to query the cache capacity management module for cache status, and determine cache allocation based on the queried total cache size and used cache size.

[0082] A cache capacity management module is added to the inline computing engine. Instead of the traditional cache allocation module allocating all physical caches, the cache allocation module queries the cache capacity management module for cache status and then determines cache allocation based on the total cache size and used cache size. The core reason for this is that the present invention not only presents a virtual cache, but also a virtual cache whose capacity varies with the current I / O environment. Therefore, the cache allocation module needs to see a dynamically sized cache.

[0083] In some embodiments, the virtual cache expansion and reclaim module is used to record the current size of the virtual cache, evaluate the reasonable size of the virtual cache, and perform expansion and reclaim operations on the virtual cache based on the two data.

[0084] In some embodiments, the virtual cache expansion and reclaiming module is further configured to estimate a reasonable size of the virtual cache by collecting parameters of current IO operations.

[0085] The core of the cache capacity management module is the virtual cache expansion and reclamation module, which is responsible for recording the current virtual cache size, evaluating the appropriate virtual cache size, and performing virtual cache expansion and reclamation operations based on these two data sets. Among these functions, evaluating the appropriate virtual cache size is the core function. This function primarily estimates the appropriate virtual cache size by collecting parameters of current I / O operations. I / O parameters primarily include the read latency of current memory data and the frequency of overflow events.

[0086] Among them, the key time periods that affect memory data latency are as follows: Figure 5As shown, t1 comes from the internal processing logic of the inline computing engine. After the computing engine is designed, this delay is a relatively fixed value. t2 is the delay in reading the memory. This delay generally consists of two parts: one is the fixed delay caused by the internal processing logic, and the other is the delay caused by reading the memory particles outside the chip. This part of the delay can be regarded as a random value and will change with conditions such as the number of modules accessing the memory at a certain moment and the access bandwidth. Therefore, the present invention needs to perform statistics and estimation on this random value. t3 is the delay of the memory data being sent to the memory data cache and output to the operation unit. This delay is indirectly affected by the delay of the operation unit writing the result to the memory. It is also a random time, and the present invention will also perform statistical estimation on this time.

[0087] Of all the delays mentioned above, the delay caused by internal logic, which is relatively fixed after the processing logic design is completed, can be calculated based on the longest delay path of the processing logic. This delay is then used to calculate the minimum amount of physical memory required by the computing engine. The calculation method is:

[0088] Minimum physical cache size = fixed delay * designed bandwidth.

[0089] When calculating cache size, traditional methods generally use IO execution delay as the delay data, rather than internal logic delay. The former numerically includes the latter, and the command processing delay of general peripherals is more than ten times the internal logic delay. Therefore, the physical cache size required by the present invention is much smaller than that of traditional methods.

[0090] In some embodiments, the parameters of the IO operation include the read latency of current memory data and the frequency of overflow events.

[0091] Of course, the physical cache calculated by the above method is the minimum value. When implementing the present invention, it is recommended to use a cache larger than the minimum value to reduce the probability of data overflow. The probability of data overflow will be described later. Under normal circumstances, although the read and write delay of the memory chip is a random quantity, it generally fluctuates around a mean value. The mean value is a quantity related to the memory address access pattern. For the same product, the value can be simulated and estimated. If a more accurate value can be obtained, this value can be used as the basis for calculating the physical cache size. At this time, the physical cache size can be set to:

[0092] Physical cache size = (fixed delay + average memory delay) * design bandwidth.

[0093] Delay statistics using Figure 6 In some embodiments, the inline computing engine section performs latency statistics, including the following steps:

[0094] When the read control module sends a read command to the memory, the relevant parameters of the command will be written into the descriptor queue;

[0095] When the memory returns the data to be read to the memory data cache module, the timestamp of the moment when it enters the operation unit from the memory data cache module is recorded and subtracted from the timestamp of the read command to calculate the delay;

[0096] When the first data of each read command is sent to the operation unit, the descriptor queue sends the timestamp of the read command to the delay estimation module, and the delay estimation module uses the delay to estimate and adjust the cache capacity.

[0097] In some embodiments, the read control module sends a read command to the memory, and the relevant parameters of the command are a timestamp when the read command is issued and a data size of the read command.

[0098] In some embodiments, the inline computing engine section performs delay statistics and also includes monitoring the data interface of the memory data cache entering the computing unit, and the monitoring process is based on the read command issued to the memory.

[0099] Therefore, when the read control module sends a read command to the memory, the parameters related to the command will be written into the descriptor queue. The most important parameter is the timestamp when the read command is issued, and also includes the size of the data of the read command.

[0100] When the memory returns the data to be read to the memory data cache and the timestamp of the moment it enters the operation unit from the memory data cache also needs to be recorded, and the delay is calculated by subtracting it from the timestamp of the read command. The data interface of the memory data cache entering the operation unit needs to be monitored, and the monitoring process is based on the read command issued to the memory. For example, assuming that several read commands are issued, the data size of each read command is N0, N1..., the monitoring module first monitors the read command 0 and obtains the moment when the first data of the read command 0 enters the operation unit from the memory data cache; then, it continues to monitor the data outlet. When the data of amount N0 is received by the operation unit, the monitoring of the read command 0 ends and the monitoring of the read command 1 begins. The above N0, N1... are the data sizes in the write descriptor queue.

[0101] When the first data of each read command is sent to the operation unit, the descriptor queue sends the timestamp of the read command to the delay estimation module, and the delay estimation module uses the delay to estimate and adjust the cache capacity.

[0102] If the physical memory size has been designed to take the average memory delay into account, then use this average as the delay module threshold. If the average cannot be estimated, calculate the delay module threshold parameters as follows:

[0103] Delay module threshold = (actual cache size - minimum cache size) ÷ designed bandwidth.

[0104] The role of this threshold is to determine whether the virtual cache needs to be expanded or reduced. The basic criterion is that when the actual delay is greater than the delay module threshold, it means that overflow may occur due to insufficient cache, and the cache utilization needs to be reduced. Therefore, the virtual cache size needs to be reduced. Conversely, the size of the virtual cache can be increased.

[0105] It should be noted that various mature algorithms can be used in implementation. For example, the PID algorithm is a simple and low-cost method, or the virtual cache size can be fine-tuned and locked using the phase-locking principle. This invention does not discuss the advantages and disadvantages of these algorithms, but algorithms designed according to the above principles should be included in the scope of protection of this invention.

[0106] The virtual cache can be increased or decreased according to the following rules:

[0107] If you need to increase the virtual cache, you can do so directly;

[0108] If the virtual cache needs to be reduced, first check whether there is any unallocated virtual cache larger than the amount to be reduced. If so, it can be reduced directly. If not, first deduct all the current free virtual cache space. For the remaining part, deduct the virtual cache that could have been returned to the virtual cache pool after the IO is completed.

[0109] Since the present invention over-configures the cache, data overflow is inevitable. The core reason for the overflow is that there is jitter in the delay of memory access, that is, the delay of memory access at certain moments is significantly greater than the average.

[0110] In the prior art, there are two methods for handling overflow:

[0111] One approach is to mark the task corresponding to the data write to the external device involved in the overflow as invalid and report this to the system scheduling module. The latter will reassign the task to the computing engine and instruct the peripheral management module to resend the data read command to the external device. Furthermore, before reporting this to the system scheduling module, the unprocessed data from the command involved in the overflow data must be processed. A simple approach is to cache this unprocessed data as normal data and count it.

[0112] Another approach is to check the available space in the cache before each data written by an external device is written to it. If there is insufficient space, the data is written to the internal memory instead of the internal cache. At the same time, the data and the corresponding memory location are stored in an overflow queue. When there is free space in the cache, the overflow queue is processed.

[0113] The first of the above methods is more expensive because even a single-byte overflow requires redoing the task; the latter will result in additional write and read operations to the memory. Generally, increased memory load will increase latency and cause more cache overflows. Therefore, although this method can solve the cache overflow problem, it will make the problem worse. If overflows continue to occur, the system will not be able to achieve the designed performance due to excessive memory bandwidth utilization.

[0114] To address this issue, the present invention uses overflow events as another parameter for adjusting the virtual cache size when implementing the aforementioned method. The fundamental principle is that overflow events will result in a reduction in the virtual cache size. This variable can be incorporated into the cache increment / decrement calculation or as a trigger for cache fine-tuning, depending on the specific algorithm.

[0115] The second aspect of the embodiment of the present invention is Figure 5 As shown, a method for improving cache utilization of a computing engine is provided, which is applicable to an inline computing engine. The method includes the following steps:

[0116] Based on the inline computing engine receiving data from external devices, the virtual cache is used to increase the cache capacity. The cache allocation module queries the cache capacity management module for cache status and determines the cache allocation based on the total cache size and used cache size.

[0117] The virtual cache expansion and recycling module records the current virtual cache size, evaluates the reasonable size of the virtual cache, and expands and recycles the virtual cache based on these two data. The reasonable size of the virtual cache is estimated by collecting the parameters of the current IO operation.

[0118] In some embodiments, a "virtual" cache is used instead of just a physical cache for allocation, and a cache capacity management module is added in actual implementation to manage the space.

[0119] In some embodiments, a method for calculating the size of the required physical cache and a regulation criterion for the required virtual cache are provided.

[0120] In some embodiments, the inline computing engine section performs latency statistics, including the following steps:

[0121] When the read control module sends a read command to the memory, the relevant parameters of the command will be written into the descriptor queue;

[0122] When the memory returns the data to be read to the memory data cache module, the timestamp of the moment when it enters the operation unit from the memory data cache module is recorded and subtracted from the timestamp of the read command to calculate the delay;

[0123] When the first data of each read command is sent to the operation unit, the descriptor queue sends the timestamp of the read command to the delay estimation module, and the delay estimation module uses the delay to estimate and adjust the cache capacity.

[0124] In some embodiments, the read control module sends a read command to the memory, and the relevant parameters of the command are a timestamp when the read command is issued and a data size of the read command.

[0125] In some embodiments, the inline computing engine section performs delay statistics and also includes monitoring the data interface of the memory data cache entering the computing unit, and the monitoring process is based on the read command issued to the memory.

[0126] In some embodiments, the size of the physical memory is designed taking into account the average memory delay, and the average memory delay is used as the delay module threshold. The method for calculating the delay module threshold parameter is:

[0127] Delay module threshold = (actual cache size - minimum cache size) ÷ designed bandwidth.

[0128] The method designed in the present invention introduces an over-allocation mechanism, which can realize a virtual cache larger than the physical cache. Through this part of the virtual cache, the inline computing engine can support more concurrent tasks of external devices, thereby improving cache utilization and the performance of the inline computing engine.

[0129] By dynamically adjusting the virtual cache size through current external device and memory parameters, it is possible to avoid the need to instantiate an overly large physical cache based on the worst-case scenario during chip design. This allows for the use of a smaller physical cache to adapt to various usage scenarios while maintaining the same performance, effectively reducing chip costs.

[0130] It should be understood that, although the above is described in a certain order, these steps are not necessarily performed in sequence according to the above order. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0131] A third aspect of the embodiments of the present invention further provides a computer-readable storage medium. Figure 7 Schematic diagram of a computer-readable storage medium for a method of improving computing engine cache utilization provided by an embodiment of the present invention. Figure 7 As shown, the computer-readable storage medium 300 stores computer program instructions 310, which can be executed by a processor. When the computer program instructions 310 are executed, the method of any one of the above embodiments is implemented.

[0132] It should be understood that, to the extent that they do not conflict with each other, all the embodiments, features and advantages described above for the method for improving the computing engine cache utilization according to the present invention are also applicable to the system and storage medium for improving the computing engine cache utilization according to the present invention.

[0133] A fourth aspect of the embodiments of the present invention further provides a computer device 400, including a memory 420 and a processor 410. The memory stores a computer program, and when the computer program is executed by the processor, the method of any one of the above embodiments is implemented, including the following steps:

[0134] Based on the inline computing engine receiving data from external devices, the virtual cache is used to increase the cache capacity. The cache allocation module queries the cache capacity management module for cache status and determines the cache allocation based on the total cache size and used cache size.

[0135] The virtual cache expansion and recycling module records the current virtual cache size, evaluates the reasonable size of the virtual cache, and expands and recycles the virtual cache based on these two data. The reasonable size of the virtual cache is estimated by collecting the parameters of the current IO operation.

[0136] like Figure 8 As shown in FIG, a hardware structure diagram of an embodiment of a computer device for executing a method for improving cache utilization of a computing engine provided by the present invention is shown. Figure 8Taking the computer device 400 shown as an example, the computer device includes a processor 410 and a memory 420, and may also include: an input device 430 and an output device 440. The processor 410, the memory 420, the input device 430 and the output device 440 can be connected via a bus or other means. Figure 8 The example of the bus connection is shown in FIG. The input device 430 can receive input digital or character information and generate signal input related to improving the cache utilization of the computing engine. The output device 440 can include a display device such as a display screen.

[0137] The memory 420, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the method for improving the cache utilization of the computing engine in the embodiment of the present application. The memory 420 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the use of the method for improving the cache utilization of the computing engine, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 420 may optionally include a memory remotely located relative to the processor 410, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0138] The processor 410 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 420, that is, the method for improving the cache utilization of the computing engine of the above method embodiment is implemented, and the steps are as follows:

[0139] Based on the inline computing engine receiving data from external devices, the virtual cache is used to increase the cache capacity. The cache allocation module queries the cache capacity management module for cache status and determines the cache allocation based on the total cache size and used cache size.

[0140] The virtual cache expansion and recycling module records the current virtual cache size, evaluates the reasonable size of the virtual cache, and expands and recycles the virtual cache based on these two data. The reasonable size of the virtual cache is estimated by collecting the parameters of the current IO operation.

[0141] Finally, it should be noted that the computer-readable storage medium (e.g., memory) herein may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. By way of example and not limitation, the non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which may act as an external cache memory. By way of example and not limitation, RAM may be obtained in a variety of forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to include, but are not limited to, these and other suitable types of memory.

[0142] The various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure herein may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and / or any other such configuration.

[0143] The present invention proposes a system and method for improving the cache utilization of a computing engine. The present invention introduces an over-allocation mechanism to achieve a virtual cache that is larger than the physical cache. Through this virtual cache, the inline computing engine can support more concurrent tasks of external devices, thereby improving the cache utilization and the performance of the inline computing engine.

[0144] The present invention dynamically adjusts the virtual cache size through current external device and memory parameters, which can avoid the need to instantiate an overly large physical cache according to the worst case scenario during chip design. It can achieve the same performance while using only a smaller physical cache to adapt to various usage scenarios, which can effectively reduce chip costs.

[0145] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope of the embodiments disclosed in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless expressly limited to the singular.

[0146] It should be understood that, as used herein, the singular form "a" or "an" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the items listed in association. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0147] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples. Within the spirit of the embodiments of the present invention, the technical features of the above embodiments or different embodiments may be combined, and there are many other variations of different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the scope of protection of the embodiments of the present invention.

Claims

1. A system for improving cache utilization of a computing engine, characterized in that: The system includes an external device, an external device controller, and an inline computing engine; The external device is used to send data to the inline computing engine and enter the physical cache; The external device controller is configured to send a cache allocation request to the inline computing engine and to send a read command to the external device; The inline computing engine uses a virtual cache to increase cache capacity. The inline computing engine is used to receive data input from an external device for physical cache, and the cached data is output from the physical cache to the computing unit. The inline computing engine includes a cache allocation module, a cache capacity management module and an external parameter monitoring module; The cache allocation module is used to receive a cache allocation request sent by the external device controller, and perform cache allocation through the cache capacity management module based on the cache allocation request; The cache capacity management module is provided with a virtual cache expansion and recycling module, wherein the virtual cache expansion and recycling module is provided with a virtual cache to increase the overall cache capacity of the inline computing engine; The external parameter monitoring module is used to connect with the virtual cache expansion and recycling module and detect the cache allocation in the inline computing engine based on external parameters; The virtual cache and the physical cache constitute the total cache size of the inline computing engine.

2. The system for improving computing engine cache utilization according to claim 1, characterized in that: In the inline computing engine, the cache allocation module is further configured to query the cache capacity management module for cache status, and determine cache allocation based on the queried total cache size and used cache size.

3. The system for improving computing engine cache utilization according to claim 2, characterized in that: The virtual cache expansion and recycling module is used to record the current size of the virtual cache, evaluate the reasonable size of the virtual cache, and perform expansion and recycling operations on the virtual cache based on these two data; The virtual cache expansion and reclaiming module is further configured to estimate a reasonable size of the virtual cache by collecting parameters of the current IO operation.

4. The system for improving computing engine cache utilization according to claim 3, characterized in that: The parameters of the IO operation include the read latency of the current memory data and the frequency of overflow events.

5. The system for improving computing engine cache utilization according to claim 4, characterized in that: When the inline computing engine section performs physical cache size calculation, the physical cache size is set to: Physical cache size = (fixed latency + average memory latency) * designed bandwidth.

6. The system for improving computing engine cache utilization according to claim 5, characterized in that: The inline computing engine section performs delay statistics, including the following steps: When the read control module sends a read command to the memory, the relevant parameters of the command will be written into the descriptor queue; When the memory returns the data to be read to the memory data cache module, the timestamp of the moment when it enters the operation unit from the memory data cache module is recorded and subtracted from the timestamp of the read command to calculate the delay; When the first data of each read command is sent to the operation unit, the descriptor queue sends the timestamp of the read command to the delay estimation module, and the delay estimation module uses the delay to estimate and adjust the cache capacity.

7. The system for improving computing engine cache utilization according to claim 6, characterized in that: The read control module sends a read command to the memory, wherein the relevant parameters of the command are a timestamp when the read command is issued and a data size of the read command.

8. A method for improving computing engine cache utilization, based on the system for improving computing engine cache utilization according to any one of claims 1 to 7, characterized in that: The method comprises the following steps: Based on the inline computing engine receiving data from external devices, the virtual cache is used to increase the cache capacity. The cache allocation module queries the cache capacity management module for cache status and determines the cache allocation based on the total cache size and used cache size. The virtual cache expansion and recycling module records the current virtual cache size, evaluates the reasonable size of the virtual cache, and expands and recycles the virtual cache based on these two data. The reasonable size of the virtual cache is estimated by collecting the parameters of the current IO operation.

Images