A data caching method and terminal

Abstract

The present invention discloses a data caching method and terminal, which store the serial number and generation time of the data block in the cache, can distinguish the generation time of the cache block, and use the storage location of the cache block as the judgment condition of the cache replacement strategy. Cache blocks with a relatively long generation time are preferentially eliminated, and when the remaining data are all data with a relatively new generation time, they are replaced according to the location of the cache block. In this way, high access efficiency can be maintained for hot message data that needs to be accessed frequently, and in the scenario of temporary intermediate reading of messages, the cache expiration problem can also be solved quickly, ensuring the access efficiency of the latest messages and improving the overall message consumption rate.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and in particular to a data caching method and a terminal. Background Art

[0002] With the rise of the Internet, the number of Internet users has gradually increased, and a large number of user operations will bring tremendous pressure to Internet product service-side products. At present, in order to meet the performance requirements in large-concurrency and high-throughput scenarios, the server-side system will adopt a microservice architecture for application deployment, in which the decoupling between each microservice is carried out by direct http calls or MQ (message queue) message notifications. In the MQ method, in order to ensure that messages are not lost, persistent messages are generally used to store messages. When consumers need to read messages, they will read them from files or databases. In order to improve the reading efficiency of messages, caches are usually used to store temporary messages.

[0003] However, in the usage scenario of MQ, there may be a scenario with a high probability that the MQ consumer may read the message from the end of the file, in which case it will be cached, or it may read the message from the middle of the file, in which case it will also be cached. For most consumers, messages are consumed at the end of the file because the messages are the latest, but there are still some consumers who consume messages from the file header or the middle of the file. At this time, according to the existing cache invalidation methods, the middle part of the cached message will replace the end part. At this time, the data in the cache is only valid for a small number of consumers. For most consumers who need to read the latest messages, the data in the cache cannot be read effectively, and the data needs to be read from the file or storage again and loaded into the cache, which further affects the efficiency of message reading. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data caching method and a terminal, which can ensure the access efficiency of the latest message, thereby improving the overall message consumption rate.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A data caching method comprises the steps of:

[0007] Storing the serial number and generation time of the data block in the cache, wherein the serial number of the data block corresponds to the position of the message file where the data block is located;

[0008] When the cache reaches the storage upper limit, determine whether there is a data block whose generation time and current time difference exceeds a preset value. If so, remove the data block. Otherwise, remove a preset number of data blocks from the cache in ascending order of sequence number.

[0009] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0010] A data cache terminal comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0011] Storing the serial number and generation time of the data block in the cache, wherein the serial number of the data block corresponds to the position of the message file where the data block is located;

[0012] When the cache reaches the storage upper limit, determine whether there is a data block whose generation time and current time difference exceeds a preset value. If so, remove the data block. Otherwise, remove a preset number of data blocks from the cache in ascending order of sequence number.

[0013] The beneficial effects of the present invention are as follows: by storing the serial number and generation time of the data block in the cache, the generation time of the cache block can be distinguished, and the storage location of the cache block is used as the judgment condition for the cache replacement strategy. Cache blocks with a relatively long generation time are preferentially eliminated, and when the remaining data are all data with a relatively new generation time, they are replaced according to the location of the cache block. In this way, high access efficiency can be maintained for hot message data that needs to be accessed frequently, and in the case of temporary intermediate reading of messages, the cache expiration problem can also be solved quickly, ensuring the access efficiency of the latest messages and improving the overall message consumption rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flow chart of a data caching method according to an embodiment of the present invention;

[0015] Figure 2 A schematic diagram of a data cache terminal according to an embodiment of the present invention;

[0016] Description of labels:

[0017] 1. A data cache terminal; 2. A memory; 3. A processor. DETAILED DESCRIPTION

[0018] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in conjunction with the implementation modes and the accompanying drawings.

[0019] Please refer to Figure 1 , an embodiment of the present invention provides a data caching method, comprising the steps of:

[0020] Storing the serial number and generation time of the data block in the cache, wherein the serial number of the data block corresponds to the position of the message file where the data block is located;

[0021] When the cache reaches the storage upper limit, determine whether there is a data block whose generation time and current time difference exceeds a preset value. If so, remove the data block. Otherwise, remove a preset number of data blocks from the cache in ascending order of sequence number.

[0022] From the above description, it can be seen that the beneficial effects of the present invention are: by storing the serial number and generation time of the data block in the cache, the generation time of the cache block can be distinguished, and the storage location of the cache block is used as the judgment condition of the cache replacement strategy. Cache blocks with a relatively long generation time are preferentially eliminated, and when the remaining data are all data with a relatively new generation time, they are replaced according to the location of the cache block. In this way, high access efficiency can be maintained for hot message data that needs to be accessed frequently. For temporary intermediate reading of messages, the cache expiration problem can also be solved quickly, ensuring the access efficiency of the latest messages and improving the overall message consumption rate.

[0023] Furthermore, the sequence number and generation time of the data block stored in the cache also include:

[0024] It is determined whether the difference between the generation time of the data block and the current time exceeds a preset value. If so, the data block is marked as a first partition data block; otherwise, the data block is marked as a second partition data block.

[0025] As can be seen from the above description, the data blocks are divided into the first partition data blocks and the second partition data blocks according to the generation time, so that the expiration processing is directly performed based on the different partition data blocks when the cache expires later, thereby improving the cache efficiency.

[0026] Furthermore, removing a preset number of data blocks from the cache in ascending order of sequence numbers includes:

[0027] Determine whether the sequence number of the newly added data block is less than the preset sequence number. If so, remove a preset number of data blocks from the cache in ascending order of sequence number, and store the newly added data block in the preset number of data blocks;

[0028] Otherwise, before storing each newly added data block, the data block with the smallest sequence number in the cache is removed.

[0029] From the above description, it can be seen that for new data block numbers that are less than the preset number, a preset number of data blocks are removed for storage of the new data blocks; for new data block numbers that are greater than or equal to the preset number, the data block with the smallest number in the cache is removed each time. Therefore, a specific cache space can be provided for the task of temporarily reading intermediate data blocks without affecting the cache of more frequently accessed data blocks.

[0030] Furthermore, the sequence number of the data block corresponds to the position of the message file where the data block is located, including:

[0031] When a new data block is added to the message file, a self-increasing serial number is set for the new data block, and the position of the data block in the message file is determined according to the serial number.

[0032] It can be seen from the above description that when a new data block is added to a message file, a self-incrementing serial number is set for the data block, so that its position in the message file can be accurately understood according to the serial number of the data block.

[0033] Further, the determining whether the sequence number of the newly added data block is less than a preset sequence number, if so, removing a preset number of data blocks from the cache in ascending order of sequence number, and storing the newly added data block in the preset number of data blocks includes:

[0034] Calculate the difference between the maximum sequence number of the data block in the message file and the upper limit of the cache capacity to obtain a preset sequence number;

[0035] The method determines whether the sequence number of the newly added data block is less than the preset sequence number. If so, a preset number of data blocks are removed from the cache in ascending order of sequence number, and the preset number of data blocks are cyclically used to store the newly added data block.

[0036] From the above description, it can be seen that by calculating the difference between the maximum sequence number of the data block in the message file and the upper limit of the cache capacity, the preset sequence number can be obtained, thereby judging whether the newly added data block in the cache is the intermediate data in the temporarily read message file or the data at the end of the frequently read message file; and for the case of temporarily reading the intermediate data, only a preset number of data block spaces are allocated for caching, thereby ensuring that access to the latest message is not affected.

[0037] Please refer to Figure 2 Another embodiment of the present invention provides a data cache terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0038] Storing the serial number and generation time of the data block in the cache, wherein the serial number of the data block corresponds to the position of the message file where the data block is located;

[0039] When the cache reaches the storage upper limit, determine whether there is a data block whose generation time and current time difference exceeds a preset value. If so, remove the data block. Otherwise, remove a preset number of data blocks from the cache in ascending order of sequence number.

[0040] From the above description, it can be seen that by storing the serial number and generation time of the data block in the cache, the generation time of the cache block can be distinguished, and the storage location of the cache block can be used as the judgment condition for the cache replacement strategy. Cache blocks with a relatively long generation time are preferentially eliminated. When the remaining data are all data with a relatively new generation time, they are replaced according to the location of the cache block. In this way, high access efficiency can be maintained for hot message data that needs to be accessed frequently. For temporary intermediate reading scenarios of messages, the cache expiration problem can also be solved quickly, ensuring the access efficiency of the latest messages and improving the overall message consumption rate.

[0041] Furthermore, the sequence number and generation time of the data block stored in the cache also include:

[0042] It is determined whether the difference between the generation time of the data block and the current time exceeds a preset value. If so, the data block is marked as a first partition data block; otherwise, the data block is marked as a second partition data block.

[0043] As can be seen from the above description, the data blocks are divided into the first partition data blocks and the second partition data blocks according to the generation time, so that the expiration processing is directly performed based on the different partition data blocks when the cache expires later, thereby improving the cache efficiency.

[0044] Furthermore, removing a preset number of data blocks from the cache in ascending order of sequence numbers includes:

[0045] Determine whether the sequence number of the newly added data block is less than the preset sequence number. If so, remove a preset number of data blocks from the cache in ascending order of sequence number, and store the newly added data block in the preset number of data blocks;

[0046] Otherwise, before storing each newly added data block, the data block with the smallest sequence number in the cache is removed.

[0047] From the above description, it can be seen that for new data block numbers that are less than the preset number, a preset number of data blocks are removed for storage of the new data blocks; for new data block numbers that are greater than or equal to the preset number, the data block with the smallest number in the cache is removed each time. Therefore, a specific cache space can be provided for the task of temporarily reading intermediate data blocks without affecting the cache of more frequently accessed data blocks.

[0048] Furthermore, the sequence number of the data block corresponds to the position of the message file where the data block is located, including:

[0049] When a new data block is added to the message file, a self-increasing serial number is set for the new data block, and the position of the data block in the message file is determined according to the serial number.

[0050] It can be seen from the above description that when a new data block is added to a message file, a self-incrementing serial number is set for the data block, so that its position in the message file can be accurately understood according to the serial number of the data block.

[0051] Further, the determining whether the sequence number of the newly added data block is less than a preset sequence number, if so, removing a preset number of data blocks from the cache in ascending order of sequence number, and storing the newly added data block in the preset number of data blocks includes:

[0052] Calculate the difference between the maximum sequence number of the data block in the message file and the upper limit of the cache capacity to obtain a preset sequence number;

[0053] The method determines whether the sequence number of the newly added data block is less than the preset sequence number. If so, a preset number of data blocks are removed from the cache in ascending order of sequence number, and the preset number of data blocks are cyclically used to store the newly added data block.

[0054] From the above description, it can be seen that by calculating the difference between the maximum sequence number of the data block in the message file and the upper limit of the cache capacity, the preset sequence number can be obtained, thereby judging whether the newly added data block in the cache is the intermediate data in the temporarily read message file or the data at the end of the frequently read message file; and for the case of temporarily reading the intermediate data, only a preset number of data block spaces are allocated for caching, thereby ensuring that access to the latest message is not affected.

[0055] The data caching method and terminal of the present invention are suitable for maintaining high access efficiency for frequently accessed hot message data in the scenario of message caching, and can also quickly eliminate the cache expiration problem caused by temporarily read intermediate messages in this scenario, thereby ensuring the access efficiency of the latest messages and improving the overall message consumption rate. The following is an explanation through specific implementation methods:

[0056] Embodiment 1

[0057] Please refer to Figure 1 , a data caching method, comprising the steps of:

[0058] S1. Storing the sequence number and generation time of a data block in a cache, wherein the sequence number of the data block corresponds to the position of the message file where the data block is located.

[0059] In this embodiment, most of the messages in the MQ (message queue) are persistently stored in the form of files. Each time a message is read, a batch of coherent message data is read and written into the cache as a cache data block.

[0060] S11. Store the sequence number and generation time of the data block in the cache.

[0061] Specifically, when each cache data block is written into the cache, two pieces of information need to be written synchronously, one is the generation time of the data block, and the other is the sequence number of the data block.

[0062] When a new data block is added to the message file, a self-increasing serial number is set for the new data block, and the position of the data block in the message file is determined according to the serial number.

[0063] Specifically, the sequence number of the data block is a self-increasing sequence number. For example, if a message file has 10 data block sequence numbers, the sequence numbers of the 10 data blocks are incremented in the order of 1-10.

[0064] S12. Determine whether the difference between the generation time of the data block and the current time exceeds a preset value. If so, mark the data block as a first partition data block; otherwise, mark the data block as a second partition data block.

[0065] Specifically, in this embodiment, taking the current time as the standard, assuming that the difference between the generation time of a data block and the current time reaches more than 600 seconds, it is identified as the first partition data block, that is, the old partition data, and the message with a difference within 600 seconds is identified as the second partition data block, that is, the new partition data. In the new partition, the cache data block is distinguished according to the difference between the sequence number where the cache data block is located and the largest sequence number of the cache block.

[0066] S2. When the cache reaches the storage upper limit, determine whether there is a data block whose generation time and current time difference exceeds a preset value. If so, remove the data block. Otherwise, remove a preset number of data blocks from the cache in ascending order of sequence number.

[0067] S21, when the cache reaches the storage limit, the old partition data is removed first and the new partition data is retained. That is, at this time, the cache database blocks generated earlier will be removed according to the generation time of the cache blocks.

[0068] S22. When the cache contains only new partition data, determine whether the sequence number of the newly added data block is less than the preset sequence number. If so, remove a preset number of data blocks from the cache in ascending order of sequence number, and store the newly added data blocks in the preset number of data blocks. Otherwise, remove the data block with the smallest sequence number in the cache before storing each newly added data block.

[0069] Specifically, when a consumer thread reads a message from the middle of a file, it will first expire a small portion of the new partition data. The new partition data will be expired first according to the sequence number of the cached data block.

[0070] In this embodiment, if the message consumed by the consumer thread is only 100 seconds away from the latest message, a cache block with the smallest auto-increment number will be removed first, and the intermediate data will be written into the cache. When a new data block is written again, according to the removal strategy with the smallest auto-increment number in the new partition, only the central data cache block will be removed, and the other newer cache data blocks will not be removed.

[0071] Embodiment 2

[0072] The difference between this embodiment and the first embodiment is that how to perform cache expiration processing of the second partition data block is further defined, specifically:

[0073] Calculate the difference between the maximum sequence number of the data block in the message file and the upper limit of the cache capacity to obtain a preset sequence number;

[0074] The method determines whether the sequence number of the newly added data block is less than the preset sequence number. If so, a preset number of data blocks are removed from the cache in ascending order of sequence number, and the preset number of data blocks are cyclically used to store the newly added data block.

[0075] In this embodiment, the maximum sequence number of the data block in the message file is 100, and the upper limit of the cache capacity is 50, so the preset sequence number is 50;

[0076] At this time, a new consumer thread is added to the message middleware. This thread needs to obtain data from the middle part of the message file. The first starting data obtained at this time has a sequence number of 20. According to the cache replacement rule in this article, only cache blocks with sequence numbers 50 and 51 are removed, and data with sequence numbers 21 and 22 are pre-written.

[0077] When the consumer thread continues to consume messages, it can directly read data from cache blocks 21 and 22, speeding up its own data reading and occupying only two cache blocks. If the existing cache expiration strategy is adopted, this consumer thread will replace the subsequent cache data between blocks 52-100 with the data of cache blocks 23, 24, 25, etc. in sequence, which will cause most consumers who need to read the latest data to read data from the file again and write their own data to the cache again.

[0078] Therefore, this embodiment can prevent a thread that temporarily reads intermediate data from causing the threads that read most of the latest data to spend more time.

[0079] Embodiment 3

[0080] Please refer to Figure 2A data cache terminal 1 includes a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, each step of a data cache method of embodiment one or two is implemented.

[0081] In summary, the present invention provides a data caching method and terminal, which store the serial number and generation time of the data block in the cache, can distinguish the generation time of the cache block, and use the storage location of the cache block as the judgment condition for the cache replacement strategy. Cache blocks with a relatively long generation time are preferentially eliminated, and when the remaining data are all data with a relatively new generation time, they are replaced according to the location of the cache block. In this way, high access efficiency can be maintained for hot message data that needs to be accessed frequently, and in the scenario of temporary intermediate reading of messages, the cache expiration problem can also be solved quickly, ensuring the access efficiency of the latest messages and improving the overall message consumption rate.

[0082] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data caching method, characterized in that: Includes steps: Storing the serial number and generation time of the data block in the cache, wherein the serial number of the data block corresponds to the position of the message file where the data block is located; When the cache reaches the storage limit, determine whether there is a data block whose generation time and current time difference exceeds a preset value. If so, remove the data block. Otherwise, remove a preset number of data blocks from the cache in ascending order of sequence number. Eliminating a preset number of data blocks from the cache in ascending order of sequence numbers includes: determining whether the sequence numbers of the newly added data blocks are less than the preset sequence numbers, if so, eliminating the preset number of data blocks from the cache in ascending order of sequence numbers, and storing the newly added data blocks in the preset number of data blocks; otherwise, before storing each newly added data block, eliminating the data block with the smallest sequence number in the cache; Determining whether the sequence number of the newly added data block is less than the preset sequence number, and if so, removing the preset number of data blocks from the cache in the order of increasing sequence numbers, and storing the newly added data blocks in the preset number of data blocks includes: calculating the difference between the maximum sequence number of the data block in the message file and the upper limit value of the cache capacity to obtain the preset sequence number; determining whether the sequence number of the newly added data block is less than the preset sequence number, and if so, removing the preset number of data blocks from the cache in the order of increasing sequence numbers, and cyclically using the preset number of data blocks to store the newly added data blocks.

2. A data caching method according to claim 1, characterized in that: The sequence number and generation time of the data block stored in the cache also include: It is determined whether the difference between the generation time of the data block and the current time exceeds a preset value. If so, the data block is marked as a first partition data block; otherwise, the data block is marked as a second partition data block.

3. A data caching method according to claim 1, characterized in that: The sequence number of the data block corresponds to the position of the message file where the data block is located, including: When a new data block is added to the message file, a self-increasing serial number is set for the new data block, and the position of the data block in the message file is determined according to the serial number.

4. A data cache terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Storing the serial number and generation time of the data block in the cache, wherein the serial number of the data block corresponds to the position of the message file where the data block is located; When the cache reaches the storage limit, determine whether there is a data block whose generation time and current time difference exceeds a preset value. If so, remove the data block. Otherwise, remove a preset number of data blocks from the cache in ascending order of sequence number. Eliminating a preset number of data blocks from the cache in ascending order of sequence numbers includes: determining whether the sequence numbers of the newly added data blocks are less than the preset sequence numbers, if so, eliminating the preset number of data blocks from the cache in ascending order of sequence numbers, and storing the newly added data blocks in the preset number of data blocks; otherwise, before storing each newly added data block, eliminating the data block with the smallest sequence number in the cache; Determining whether the sequence number of the newly added data block is less than the preset sequence number, and if so, removing the preset number of data blocks from the cache in the order of increasing sequence numbers, and storing the newly added data blocks in the preset number of data blocks includes: calculating the difference between the maximum sequence number of the data block in the message file and the upper limit value of the cache capacity to obtain the preset sequence number; determining whether the sequence number of the newly added data block is less than the preset sequence number, and if so, removing the preset number of data blocks from the cache in the order of increasing sequence numbers, and cyclically using the preset number of data blocks to store the newly added data blocks.

5. A data cache terminal according to claim 4, characterized in that: The sequence number and generation time of the data block stored in the cache also include: It is determined whether the difference between the generation time of the data block and the current time exceeds a preset value. If so, the data block is marked as a first partition data block; otherwise, the data block is marked as a second partition data block.

6. A data cache terminal according to claim 4, characterized in that: The sequence number of the data block corresponds to the position of the message file where the data block is located, including: When a new data block is added to the message file, a self-increasing serial number is set for the new data block, and the position of the data block in the message file is determined according to the serial number.

Images