Table dimension cache refresh method and system

Abstract

The present invention discloses a cache refresh method and system for table dimensions, which belongs to the technical field of table data cache refresh. The technical problem to be solved is how to reduce the development and subsequent maintenance costs of cache refresh. The method comprises the following steps: configuring a cache refresh implementation class corresponding to a target table; configuring the metadata of the target table into a cache configuration table; configuring a cache refresh identifier generator corresponding to the target table, monitoring whether the table data in the target table has changed through the cache identifier generator, and updating the DB data update timestamp of the target table in the configuration table based on the change of the table data in the target table; obtaining the last cache refresh timestamp stored in the cache through a scheduled task, and calling the cache refresh implementation class of the target table to be updated based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table, and performing cache refresh on the table data of the target table to be updated.

Description

Technical Field

[0001] The present invention relates to the technical field of table data cache refresh, and in particular to a method and system for refreshing a table dimension cache. Background Art

[0002] Traditional cache refresh technology generally writes data to the cache (such as Redis) when the database is queried, and retrieves it from Redis the next time it is queried. If the data does not exist, the database is queried again. When the database data is updated or deleted, the cache must be operated accordingly. As the system grows larger, the number of places where the cache is added, deleted, and modified will increase. If a table that is not connected to the cache wants to be connected to the cache, a lot of code modification is required, and vice versa, which greatly increases maintenance costs.

[0003] How to reduce the development and subsequent maintenance costs of cache refresh is a technical problem that needs to be solved. Summary of the Invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a table dimension cache refresh method and system to solve the technical problem of how to reduce the development and subsequent maintenance costs of cache refresh.

[0005] In a first aspect, a cache refresh method for a table dimension of the present invention comprises the following steps:

[0006] For the target table to be accessed into the cache, configure the cache refresh implementation class corresponding to the target table, and the cache refresh implementation class is used to implement cache refresh of the target table data;

[0007] For the target table to be cached, the metadata of the target table is configured in the cache configuration table. The metadata includes the table name corresponding to the target table, the full path of the cache refresh implementation class, the DB data update timestamp, and whether it is enabled;

[0008] For the target table to be accessed by the cache, a cache refresh identifier generator corresponding to the target table is configured. The cache identifier generator is used to monitor whether there are changes in the table data in the target table, and the DB data update timestamp of the target table in the configuration table is updated based on the changes in the table data in the target table.

[0009] The last cache refresh timestamp stored in the cache is obtained through a scheduled task, and based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table, the target table to be updated is found, and the cache refresh implementation class of the target table to be updated is called. The cache refresh implementation class is used to refresh the table data of the target table to be updated.

[0010] Preferably, the metadata further includes extended information, wherein the extended information includes a refresh cache instance ID, an operation type, and a scheduled task execution cycle, wherein the operation type includes adding, modifying, and deleting.

[0011] Preferably, the enabling state includes true and false. The enabling state is true, indicating that the target table is in an enabled state and can be cached and refreshed. The enabling state is false, indicating that the target table is in a stopped state and the use of cache is prohibited.

[0012] Preferably, based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table, searching for the target table to be updated includes the following steps:

[0013] Compare the last cache refresh timestamp with the DB data update timestamp of all enabled target tables configured in the cache configuration table;

[0014] When the DB data update timestamp in the cache configuration table is later than the last cache refresh timestamp stored in the cache, it indicates that the table data of the corresponding target table has changed. Then, the corresponding cache refresh implementation class of the cache configuration table is called to refresh the cache data of the target table, and the last cache refresh timestamp stored in the cache is synchronously updated to the current database time, waiting for the next execution of the scheduled task.

[0015] When the DB data update timestamp in the cache configuration table is earlier than the last cache refresh timestamp stored in the cache, it means that the table data of the corresponding target table in the cache configuration table has not changed, and the cache operation will not continue.

[0016] As a preference, if the scheduled task is executed for the first time, the last cache refresh timestamp is empty, and all cache refresh implementation classes are asynchronously called to fully refresh the cache. If the cache data is subsequently damaged or lost due to force majeure, the last cache refresh timestamp can be cleared and the cache can be fully refreshed.

[0017] More preferably, the following steps are also included:

[0018] The cache configuration table is expanded, and a timed task execution period configuration field is added to the cache configuration table to set a different timed task execution period for each cache refresh of a table using the cache.

[0019] Preferably, the cache refresh implementation class is used to implement cache refresh of the target table data, including implementing conditional query of table data, cache batch deletion and cache batch addition.

[0020] In a second aspect, a table dimension cache refresh system of the present invention is provided, wherein the system is configured to execute the table dimension cache refresh method according to any one of the first aspects. The system includes a plurality of pluggable functional modules, wherein the functional modules include:

[0021] A cache refresh implementation class generation module, for a target table to be accessed by the cache, the cache refresh implementation class generation module is used to configure a cache refresh implementation class corresponding to the target table, the cache refresh implementation class being used to implement cache refresh of the target table data;

[0022] A cache configuration module, which is used to configure a cache configuration table. For a target table to be cached, the cache configuration module is used to configure the metadata of the target table into the cache configuration table. The metadata includes the table name corresponding to the target table, the full path of the cache refresh implementation class, the DB data update timestamp, and whether it is enabled;

[0023] A DB data update timestamp update module, for a target table to be accessed by the cache, configured with a cache refresh identifier generator corresponding to the target table, monitors whether there are changes in the table data in the target table through the cache identifier generator, and updates the DB data update timestamp of the target table in the configuration table based on the changes in the table data in the target table;

[0024] A cache update module, wherein the cache update module is configured with a timer task and is used to obtain the last cache refresh timestamp stored in the cache through the timer task, and based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table, find the target table to be updated, call the cache refresh implementation class of the target table to be updated, and perform cache refresh on the table data of the target table to be updated through the cache refresh implementation class;

[0025] The cache configuration module is further used to expand the cache configuration table, add a timed task execution period configuration field in the cache configuration table, and set a different timed task execution period for each cache refresh of a table using the cache.

[0026] Preferably, the cache configuration table is used to search for a target table to be updated based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table through the following steps:

[0027] Compare the last cache refresh timestamp with the DB data update timestamp of all enabled target tables configured in the cache configuration table;

[0028] When the DB data update timestamp in the cache configuration table is later than the last cache refresh timestamp stored in the cache, it indicates that the table data of the corresponding target table has changed. Then, the corresponding cache refresh implementation class of the cache configuration table is called to refresh the cache data of the target table, and the last cache refresh timestamp stored in the cache is synchronously updated to the current database time, waiting for the next execution of the scheduled task.

[0029] When the DB data update timestamp in the cache configuration table is earlier than the last cache refresh timestamp stored in the cache, it means that the table data of the corresponding target table in the cache configuration table has not changed, and the cache operation will not continue.

[0030] Preferably, the cache configuration table is used to perform the following steps:

[0031] If the scheduled task is executed for the first time, the last cache refresh timestamp is empty, and all cache refresh implementation classes are asynchronously called to fully refresh the cache. If the cache data is subsequently damaged or lost due to force majeure, the last cache refresh timestamp can be cleared and the cache can be fully refreshed.

[0032] The table dimension cache refresh method and system of the present invention have the following advantages:

[0033] 1. Implemented incremental cache refresh based on tables. As long as the corresponding cache refresh generator and cache refresh implementation class are configured, the refresh target, time, and interval can be controlled and configured, which can significantly reduce performance waste.

[0034] 2. Cache refresh is achieved under the multi-table cache system design. The modules are decoupled from each other, easy to scale and plug and play. Systems that are not connected to the cache system can use the present invention to connect to the cache, and systems that are already connected to the cache can also use the present invention to process the cache. There is no intrusion on the original business code, which is in line with the design concept of the microservice architecture.

[0035] 3. Strong scalability: table dimension cache refresh can be further expanded to an instance-accurate cache refresh method without requiring excessive code intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] The present invention will be further described below with reference to the accompanying drawings.

[0038] Figure 1This is a flowchart of the cache refresh method for table dimensions in Example 1;

[0039] Figure 2 This is a schematic diagram of the structure of a cache configuration table instance in the cache refresh method for table dimensions in Example 1;

[0040] Figure 3 This is a block diagram of the working principle of the cache refresh system of the table dimension in Example 2. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0042] Embodiments of the present invention provide a table dimension cache refresh method and system for solving the technical problem of how to reduce the development and subsequent maintenance costs of cache refresh.

[0043] Example 1:

[0044] The cache refresh method of the table dimension of the present invention includes the following steps:

[0045] S100. For a target table to be cached, configure a cache refresh implementation class corresponding to the target table. The cache refresh implementation class is used to implement cache refresh of the target table data.

[0046] S200: For the target table to be cached, configure the metadata of the target table into the cache configuration table. The metadata includes the table name corresponding to the target table, the full path of the cache refresh implementation class, the DB data update timestamp, and whether the status is enabled;

[0047] S300: For a target table to be cached, configure a cache refresh identifier generator corresponding to the target table, monitor whether there are changes in the table data in the target table through the cache identifier generator, and update the DB data update timestamp of the target table in the configuration table based on the changes in the table data in the target table;

[0048] S400. Obtain the last cache refresh timestamp stored in the cache through a scheduled task, and based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table, find the target table to be updated, call the cache refresh implementation class of the target table to be updated, and refresh the cache of the table data of the target table to be updated through the cache refresh implementation class.

[0049] The enabled state includes true and false. If the enabled state is true, it indicates that the target table is in an enabled state and can be cached and refreshed. If the enabled state is false, it indicates that the target table is in a stopped state and the use of cache is prohibited.

[0050] In this embodiment, when a table is to be accessed into the cache, a row of metadata is added to the cache configuration table (refer to Figure 2 Cache configuration table example), the content is the table name of the target table to be connected to the cache, the full path of the cache refresh implementation class, the DB data update timestamp, whether the status is enabled, and other metadata. If more detailed configuration is required, the cache configuration table can be expanded to add refresh cache instance id, operation type (add, modify, delete), scheduled task execution cycle, etc. Cache refresh will only be performed if the target table is configured in the cache configuration table and the enable status is true. If there is no configuration in the cache configuration table or the enable status is false, it will not take effect. That is, the table also serves as a single-table cache refresh enable switch. If you do not want to continue using the cache for a table that has been configured to use the cache, set the enable status of the corresponding metadata to false. If you want to continue later, restore the enable status to true.

[0051] Create a corresponding cache refresh identifier generator for the table you want to access the cache. This cache refresh identifier generator is equivalent to a trigger. The trigger condition is that the target table data has been added, modified, or deleted. The trigger result is to update the db data update timestamp of the target table record in the cache configuration table. This timestamp serves as an identifier to determine whether the cache needs to be refreshed. If refreshing the cache based on the table dimension does not meet actual business needs, the trigger can trigger the recording of the instance ID and operation type in the cache configuration table, that is, accurate to the instance refresh. If there are other business scenarios, continue to expand the cache configuration table and trigger. The trigger example is as follows:

[0052] Taking MySQL as an example, dict_item is the target table to be cached, and db_cache is the cache configuration table:

[0053] create trigger update_dict_item after UPDATE on dict_item

[0054] for each row

[0055] begin

[0056] update db_cache set update_time=now()where table_name='dict_item';

[0057] end;

[0058] The scheduled task specifies the interval to obtain the last cache refresh timestamp stored in the cache (if the scheduled task is executed for the first time, the timestamp will be empty, and all cache refresh implementation classes will be asynchronously called to fully refresh the cache. If the cache data is subsequently damaged or lost due to force majeure, the last cache refresh timestamp can be cleared and the cache can be fully refreshed), and compared with the data update timestamp of all tables configured with the enabled status set to true in the cache configuration table.

[0059] When the table data update timestamp in the cache configuration table is later than the cache refresh time stored in the cache, indicating that the corresponding record's table data has changed, the cache refresh implementation class configured for the corresponding data in the cache configuration table is called to refresh the cache. (Because this is an incremental refresh based on the table dimension, the cache can be set to be permanent. That is, if the database data has not changed after a refresh and the trigger has not executed to change the cache configuration table data update timestamp, no refresh operation will be performed.) The last cache refresh timestamp stored in the cache is also updated to the current database time, waiting for the next scheduled task execution. If the database data update timestamp is earlier than the last cache refresh time stored in the cache, indicating that the corresponding table data in the cache configuration table has not been added, deleted, or modified, that is, the trigger has not executed, no further cache operations will be performed, avoiding performance waste. The cache refresh judgment logic at the instance or other granularity is basically the same as the cache refresh judgment logic based on the table dimension. The scheduled task execution period is also configurable, and scheduled tasks can be set to single or multiple instances according to actual system needs. If you want to dynamically set the scheduled task execution period, you can expand the cache configuration table and add a scheduled task execution period configuration field to set a different scheduled task execution period for each cached table cache refresh.

[0060] This embodiment implements cache refresh based on the table dimension. The structure of each table may vary significantly, so cache refresh can be implemented separately for different tables. This cache refresh implementation class mainly implements conditional query of table data, batch cache deletion, and batch cache addition operations. It is used to execute this class method as a scheduled task to achieve batch refresh of the table cache. For example, if the data volume of a table is not large, you can consider querying all the data of the table, deleting all the cached data of the table, and writing the latest data to the cache to achieve cache refresh.

[0061] This embodiment implements how to refresh cached data based on table dimensions after DB data is added, modified, or deleted. When data in a table connected to the cache is added, modified, or deleted, the cache refresh identifier generator generates a cache refresh identifier for the table. The scheduled task scans the identifier and calls the cache refresh implementation class of the corresponding table to implement cache refresh in the table dimension with zero intrusion into the business code.

[0062] Example 2:

[0063] The cache refresh system of the table dimension of the present invention includes multiple pluggable functional modules, and the functional modules include a cache refresh implementation class generation module, a cache configuration module, a DB data update timestamp update module and a cache update module. For a target table to be connected to the cache, the cache refresh implementation class generation module is used to configure the cache refresh implementation class corresponding to the target table, and the cache refresh implementation class is used to implement cache refresh of the target table data; the cache configuration module is used to configure the cache configuration table. For a target table to be connected to the cache, the cache configuration module is used to configure the metadata of the target table into the cache configuration table. The metadata includes the table name corresponding to the target table, the full path of the cache refresh implementation class, the DB data update timestamp, and whether the status is enabled; the DB data update timestamp update module, for a target table to be connected to the cache, the DB data update timestamp update module is used to configure the target table corresponding to the cache A cache refresh identifier generator monitors whether there are changes in the table data in the target table through the cache identifier generator, and updates the DB data update timestamp of the target table in the configuration table based on the changes in the table data in the target table; the cache update module is configured with a timed task, and the cache update module is used to obtain the last cache refresh timestamp stored in the cache through the timed task, and based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table, find the target table to be updated, call the cache refresh implementation class of the target table to be updated, and refresh the table data of the target table to be updated through the cache refresh implementation class; the cache configuration module is also used to expand the cache configuration table, add a timed task execution period configuration field in the cache configuration table, and set a different timed task execution period for each cache refresh of the table using the cache.

[0064] Among them, the cache configuration table is used to find the target table to be updated based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table through the following steps: compare the last cache refresh timestamp with the DB data update timestamp of all enabled target tables configured in the cache configuration table; when the DB data update timestamp in the cache configuration table is later than the last cache refresh timestamp stored in the cache, it means that the table data of the corresponding target table has changed, then call the corresponding cache refresh implementation class of the cache configuration table to refresh the table data of the target table, and synchronously update the last cache refresh timestamp stored in the cache to the current database time, waiting for the next execution of the scheduled task; when the DB data update timestamp in the cache configuration table is earlier than the last cache refresh timestamp stored in the cache, it means that the table data of the corresponding target table in the cache configuration table has not changed, then no further operation will be performed on the cache.

[0065] If the scheduled task is executed for the first time, the last cache refresh timestamp is empty, and all cache refresh implementation classes are asynchronously called to fully refresh the cache. If the cache data is subsequently damaged or lost due to force majeure, the last cache refresh timestamp can be cleared and the cache can be fully refreshed.

[0066] like Figure 3 As shown, the working principle of this system is: when the data of a table connected to the cache is added, modified, or deleted, the cache refresh identifier generator (trigger) monitors the table data of the corresponding target table and updates its DB data update timestamp in the configuration table, that is, generating a cache refresh identifier for the target table. The scheduled task scans the identifier and compares the identifier with the last data cache refresh timestamp obtained from the cache. If it is determined that the changed target table is updated, the cache refresh implementation class of the corresponding target table is called to implement cache refresh of the table dimension with zero intrusion on the business code.

[0067] The system can execute the cache refresh method for table dimensions disclosed in the embodiment.

[0068] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. The cache refresh method of table dimension is characterized by The steps include: For the target table to be accessed into the cache, configure the cache refresh implementation class corresponding to the target table, and the cache refresh implementation class is used to implement cache refresh of the target table data; For a target table to be cached, the metadata of the target table is configured in the cache configuration table. The metadata includes the table name corresponding to the target table, the full path of the cache refresh implementation class, the DB data update timestamp, and whether the status is enabled. The metadata also includes extended information, including the refresh cache instance ID, the operation type, and the scheduled task execution period. The operation type includes add, modify, and delete. The enable status includes true and false. If the enable status is true, it indicates that the target table is enabled and can be cached. If the enable status is false, it indicates that the target table is in a stopped state and the cache is prohibited. For the target table to be accessed by the cache, a cache refresh identifier generator corresponding to the target table is configured. The cache refresh identifier generator is used to monitor whether there are changes in the table data in the target table, and the DB data update timestamp of the target table in the configuration table is updated based on the changes in the table data in the target table. Obtain the last cache refresh timestamp stored in the cache through a scheduled task, and based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table, find the target table to be updated, call the cache refresh implementation class of the target table to be updated, and use the cache refresh implementation class to perform cache refresh on the table data of the target table to be updated. The cache refresh implementation class is used to implement cache refresh of the table data of the target table, including implementing conditional query of table data, cache batch deletion and cache batch addition; Expand the cache configuration table and add a timed task execution period configuration field in the cache configuration table to set a different timed task execution period for each cache refresh of the table using the cache; Among them, based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table, the target table to be updated is found, including the following steps: comparing the last cache refresh timestamp with the DB data update timestamp of all target tables in the enabled state configured in the cache configuration table; when the DB data update timestamp in the cache configuration table is later than the last cache refresh timestamp stored in the cache, it means that the table data of the corresponding target table has changed, then the corresponding cache refresh implementation class of the cache configuration table is called to refresh the table data of the target table, and synchronously update the last cache refresh timestamp stored in the cache to the current database time, and wait for the next execution of the scheduled task; when the DB data update timestamp in the cache configuration table is earlier than the last cache refresh timestamp stored in the cache, it means that the table data of the corresponding target table in the cache configuration table has not changed, and the cache will not be operated further; If the scheduled task is executed for the first time, the last cache refresh timestamp is empty, and all cache refresh implementation classes are asynchronously called to fully refresh the cache. If the cache data is subsequently damaged or lost due to force majeure, the last cache refresh timestamp can be cleared and the cache can be fully refreshed.

2. The cache refresh system of the table dimension is characterized by The system is used to execute the cache refresh method for table dimensions according to claim 1, and the system includes multiple pluggable functional modules, and the functional modules include: A cache refresh implementation class generation module, for a target table to be accessed by the cache, the cache refresh implementation class generation module is used to configure a cache refresh implementation class corresponding to the target table, the cache refresh implementation class being used to implement cache refresh of the target table data; A cache configuration module, which is used to configure a cache configuration table. For a target table to be cached, the cache configuration module is used to configure the metadata of the target table into the cache configuration table. The metadata includes the table name corresponding to the target table, the full path of the cache refresh implementation class, the DB data update timestamp, and whether it is enabled; A DB data update timestamp update module, for a target table to be accessed by the cache, configured with a cache refresh identifier generator corresponding to the target table, monitors whether there are changes in the table data in the target table through the cache refresh identifier generator, and updates the DB data update timestamp of the target table in the configuration table based on the changes in the table data in the target table; A cache update module, wherein the cache update module is configured with a timer task and is used to obtain the last cache refresh timestamp stored in the cache through the timer task, and based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table, find the target table to be updated, call the cache refresh implementation class of the target table to be updated, and perform cache refresh on the table data of the target table to be updated through the cache refresh implementation class; The cache configuration module is further used to expand the cache configuration table, add a timed task execution period configuration field in the cache configuration table, and set a different timed task execution period for each cache refresh of the table using the cache; The cache configuration table is used to find the target table to be updated based on the enabled status of each target table in the configuration table and the comparison result of the last cache refresh timestamp and the data update timestamp of each target table in the configuration table through the following steps: Compare the last cache refresh timestamp with the DB data update timestamp of all enabled target tables configured in the cache configuration table; When the DB data update timestamp in the cache configuration table is later than the last cache refresh timestamp stored in the cache, it indicates that the table data of the corresponding target table has changed. Then, the corresponding cache refresh implementation class of the cache configuration table is called to refresh the cache data of the target table, and the last cache refresh timestamp stored in the cache is synchronously updated to the current database time, waiting for the next execution of the scheduled task. If the DB data update timestamp in the cache configuration table is earlier than the last cache refresh timestamp stored in the cache, it means that the table data of the corresponding target table in the cache configuration table has not been changed, and the cache operation will not continue; The cache configuration table is used to perform the following steps: If the scheduled task is executed for the first time, the last cache refresh timestamp is empty, and all cache refresh implementation classes are asynchronously called to fully refresh the cache. If the cache data is subsequently damaged or lost due to force majeure, the last cache refresh timestamp can be cleared and the cache can be fully refreshed.

Images