Method and device for caching data in blocks and storage medium

Abstract

The invention discloses a method and a device for caching data in blocks and a storage medium. The method comprises the following steps: after receiving a cache data request of a calculation task, determining target data required by the calculation task and a plurality of data blocks included in the target data; and afterwards, caching only one data block in the cache every time, and after it is confirmed that the calculation unit corresponding to the corresponding data block is executed and a calculation result is obtained, caching the next data block until each data block in the plurality of data blocks is cached. In the process of caching the data block by block, only one data block in the plurality of data blocks is cached each time, so that the requirement on the storage space required for caching the data is relatively low, the existing storage space can be utilized to the greatest extent, the hardware cost is reduced, and the situation that the storage space is not enough is reduced.

Description

technical field [0001] The present invention relates to the field of data processing, in particular to a method, device and storage medium for caching data in blocks. Background technique [0002] As we all know, data caching is an effective way to improve data access speed and is widely used in major data processing systems. In recent years, with the increasing development and popularization of network communication and computer technology, the application of big data has become more and more extensive, which also puts forward higher requirements for data cache space, especially for multiple cache levels. large data processing systems. [0003] The existing big data caching mechanism usually adopts the method of full data loading, which often fails to complete the caching operation due to the large amount of data and insufficient memory space. In addition, with the full data loading method, even if the amount of data is not large, when the memory space or disk space is he...

AI Technical Summary

Problems solved by technology

[0003] The existing big data caching mechanism usually adopts the method of full data loading, which often fails to complete the caching operation due to the large amount of data and insufficient memory space

In addition, using the full data loading method, even if the amount of data is not large, when the memory space or disk space is heavily occupied by some key processes and causes shortage, the caching operation will frequently fail to complete

[0004] In view of the above problems, if we only blindly solve them by adding storage space, it will inevitably lead to an increase in the cost of hardware construction and maintenance

Moreover, for some systems that are not scalable and cannot increase storage space, this method may mean rebuilding, resulting in a great waste of resources

Images