A data caching method and apparatus

Abstract

The application discloses a data caching method and device, and relates to the technical field of computers. A specific implementation of the method comprises the following steps: listening to each node of a distributed scheduling center; wherein, a directory of the node is in one-to-one correspondence with each data table in each database, and node information saved by the node comprises a data change timestamp of the data table corresponding to the node; if it is found that the node information saved by the node has changed, generating a cache identifier and a Bloom filter according to the directory of the node and the changed node information; associating the cache identifier with the Bloom filter, and storing the cache identifier and the Bloom filter corresponding to the cache identifier into a filter set. The implementation can solve the technical problems of great development difficulty, uneven memory, blocked query or inconsistent data.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data caching method and apparatus. Background Technology

[0002] In big data analytics scenarios, ClickHouse (column-based storage engine) is favored by data analysts for its excellent underlying design and good query performance, and is currently widely used in the field of online analytical processing.

[0003] However, while ClickHouse provides excellent data analysis capabilities, it also heavily utilizes CPU resources to maintain high-performance query capabilities. This makes ClickHouse unsuitable for handling particularly high-frequency operations, such as write, query, and modify operations. Users often face the problem of limited query QPS (queries per second) when using ClickHouse.

[0004] When faced with scenarios that require increased concurrent database queries, without considering complex underlying storage engine upgrades, an effective solution is to provide a pre-caching solution to enable data to be used multiple times from a single query.

[0005] However, commonly used caching solutions currently have the following technical problems:

[0006] When querying large result sets, there is a risk of uneven memory distribution and blocked queries. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a data caching method and apparatus to solve the technical problems of uneven memory allocation and blocked queries.

[0008] To achieve the above objectives, according to one aspect of the present invention, a data caching method is provided, applied to a server, comprising:

[0009] Monitor each node of the distributed scheduling center; wherein, the directory of the node corresponds one-to-one with each data table in each database, and the node information stored by the node includes the data change timestamp of the data table corresponding to the node;

[0010] If a change is detected in the node information stored by the node, a cache identifier and a Bloom filter are generated based on the node's directory and the changed node information.

[0011] The cache identifier is associated with the Bloom filter, and the cache identifier and its corresponding Bloom filter are stored in the filter set.

[0012] Optionally, the cache identifier includes the directory of the node and the data change timestamp of the data table corresponding to the node.

[0013] Optionally, the distributed scheduling center retrieves messages from the message queue, parses the messages to obtain database information, data table information, and data change timestamps, and thereby changes the node information stored by the nodes corresponding to the database and the data table.

[0014] The message is generated by the data storage engine and written into the message queue.

[0015] Optionally, after storing the cache identifier and its corresponding Bloom filter in the filter set, the method further includes:

[0016] Configure the failure timestamp of the Bloom filter; wherein the failure timestamp is the current timestamp plus the failure duration, and the failure duration is greater than the batch write cycle of the data storage engine.

[0017] Furthermore, according to another aspect of the present invention, a data caching method is provided, applied to a client, comprising:

[0018] Monitor the target node of the distributed scheduling center; wherein, the directory of the node of the distributed scheduling center corresponds one-to-one with each data table in each database, and the node information stored by the target node includes the data change timestamp of the data table corresponding to the target node;

[0019] If a change is detected in the node information stored by the target node, the changed node information is retrieved.

[0020] Optionally, it also includes:

[0021] Receive a data query request; wherein the data query request carries database information, data table information and input parameters;

[0022] The data change timestamp is matched with the database information and the data table information to generate a cache identifier;

[0023] Retrieve the Bloom filter corresponding to the cache identifier from the filter set;

[0024] The query data corresponding to the data query request is obtained based on the input parameters and the Bloom filter.

[0025] Optionally, the cache identifier includes the database information, the data table information, and the data change timestamp.

[0026] Optionally, obtaining the query data corresponding to the data query request based on the input parameters and the Bloom filter includes:

[0027] Generate a string identifier based on the class path, method name, and input parameters of the interface provided by the client;

[0028] The string identifier is calculated using a hash function to obtain the identifier bit;

[0029] Based on the identifier, determine whether the Bloom filter has a hit; if yes, retrieve the query data corresponding to the string identifier from the cache; if no, retrieve the query data corresponding to the data query request from the data table in the database.

[0030] Optionally, a string identifier is generated based on the class path, method name, and input parameters of the interface provided by the client, including:

[0031] The class path, method name, and input parameters of the interface provided by the client are encapsulated into strings;

[0032] The string is encrypted using a message digest algorithm to obtain a string identifier.

[0033] Optionally, retrieving the query data corresponding to the string identifier from the cache includes:

[0034] Determine whether the cache hit is based on the string identifier;

[0035] If so, retrieve the query data corresponding to the string identifier from the cache;

[0036] If not, then retrieve the query data corresponding to the data query request from the data table in the database.

[0037] Optionally, after obtaining the query data corresponding to the data query request from the data table in the database, the method further includes:

[0038] Store the string identifier and the query data corresponding to the data query request in the cache, and update the value of the Bloom filter based on the identifier bit.

[0039] Optionally, storing the string identifier and the query data corresponding to the data query request in a cache includes:

[0040] Determine whether the size of the dataset corresponding to the data query request is greater than or equal to the threshold;

[0041] If so, the query data is stored in HDFS, and the string identifier and the storage path of the query data in HDFS are stored in HBase;

[0042] If not, the string identifier and the query data are stored in HBase.

[0043] Furthermore, according to another aspect of the present invention, a data caching device is provided, disposed on a server, comprising:

[0044] The first monitoring module is used to monitor each node of the distributed scheduling center; wherein, the directory of the node corresponds one-to-one with each data table in each database, and the node information stored by the node includes the data change timestamp of the data table corresponding to the node;

[0045] The generation module is used to generate a cache identifier and a Bloom filter based on the directory of the node and the changed node information if a change is detected in the node information stored by the node.

[0046] A storage module is used to associate the cache identifier with the Bloom filter and store the cache identifier and its corresponding Bloom filter in a filter set.

[0047] Optionally, the cache identifier includes the directory of the node and the data change timestamp of the data table corresponding to the node.

[0048] Optionally, the distributed scheduling center retrieves messages from the message queue, parses the messages to obtain database information, data table information, and data change timestamps, and thereby changes the node information stored by the nodes corresponding to the database and the data table.

[0049] The message is generated by the data storage engine and written into the message queue.

[0050] Optionally, the storage module is further configured to:

[0051] After storing the cache identifier and its corresponding Bloom filter in the filter set, configure the expiration timestamp of the Bloom filter; wherein, the expiration timestamp is the current timestamp plus the expiration duration, and the expiration duration is greater than the batch write cycle of the data storage engine.

[0052] Furthermore, according to another aspect of the present invention, a data caching device is provided, disposed on a client, comprising:

[0053] The second monitoring module is used to monitor the target nodes of the distributed scheduling center; wherein, the directory of the nodes of the distributed scheduling center corresponds one-to-one with each data table in each database, and the node information stored by the target node includes the data change timestamp of the data table corresponding to the target node;

[0054] The pull module is used to pull the changed node information if it detects that the node information saved by the target node has changed.

[0055] Optionally, it also includes a processing module for:

[0056] Receive a data query request; wherein the data query request carries database information, data table information and input parameters;

[0057] The data change timestamp is matched with the database information and the data table information to generate a cache identifier;

[0058] Retrieve the Bloom filter corresponding to the cache identifier from the filter set;

[0059] The query data corresponding to the data query request is obtained based on the input parameters and the Bloom filter.

[0060] Optionally, the cache identifier includes the database information, the data table information, and the data change timestamp.

[0061] Optionally, the processing module is further configured to:

[0062] Generate a string identifier based on the class path, method name, and input parameters of the interface provided by the client;

[0063] The string identifier is calculated using a hash function to obtain the identifier bit;

[0064] Based on the identifier, determine whether the Bloom filter has a hit; if yes, retrieve the query data corresponding to the string identifier from the cache; if no, retrieve the query data corresponding to the data query request from the data table in the database.

[0065] Optionally, the processing module is further configured to:

[0066] The class path, method name, and input parameters of the interface provided by the client are encapsulated into strings;

[0067] The string is encrypted using a message digest algorithm to obtain a string identifier.

[0068] Optionally, the processing module is further configured to:

[0069] Determine whether the cache hit is based on the string identifier;

[0070] If so, retrieve the query data corresponding to the string identifier from the cache;

[0071] If not, then retrieve the query data corresponding to the data query request from the data table in the database.

[0072] Optionally, the processing module is further configured to:

[0073] After obtaining the query data corresponding to the data query request from the data table in the database, the string identifier and the query data corresponding to the data query request are stored in the cache, and the value of the Bloom filter is updated based on the identifier bit.

[0074] Optionally, the processing module is further configured to:

[0075] Determine whether the size of the dataset corresponding to the data query request is greater than or equal to the threshold;

[0076] If so, the query data is stored in HDFS, and the string identifier and the storage path of the query data in HDFS are stored in HBase;

[0077] If not, the string identifier and the query data are stored in HBase.

[0078] According to another aspect of the present invention, an electronic device is also provided, comprising:

[0079] One or more processors;

[0080] Storage device for storing one or more programs.

[0081] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0082] According to another aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0083] One embodiment of the above invention has the following advantages or beneficial effects: If a change is detected in the node information stored by the target node, the changed node information is retrieved, and a cache identifier is generated based on the database table information and data change timestamp. This allows for the acquisition of the Bloom filter corresponding to the cache identifier, thus overcoming the technical problems of high development difficulty, uneven memory distribution, blocked queries, or inconsistent data in the prior art. This invention's embodiment stores cached results based on Bloom filters and a caching scheme, ensuring the storage of large result sets while improving concurrent query capabilities through pre-filter matching. The client, by monitoring nodes in the distributed scheduling, can achieve real-time synchronization of the latest cache identifier, obtaining the latest Bloom filter and ensuring consistency between the cached query results and the database data, thereby achieving lossless cache query capabilities. Furthermore, the client only needs to import a client component as a plug-in to enhance caching capabilities, thus reducing development difficulty.

[0084] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0085] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0086] Figure 1 This is a schematic diagram of the main flow of a data caching method according to an embodiment of the present invention;

[0087] Figure 2 This is a schematic diagram of the server and client architecture according to an embodiment of the present invention;

[0088] Figure 3 This is a schematic diagram of the main flow of a data caching method according to another embodiment of the present invention;

[0089] Figure 4 This is a schematic diagram of the client architecture according to an embodiment of the present invention;

[0090] Figure 5 This is a schematic diagram of the main flow of a data caching method according to a possible embodiment of the present invention;

[0091] Figure 6 This is a schematic diagram of the main modules of a data caching device according to an embodiment of the present invention;

[0092] Figure 7 This is a schematic diagram of the main modules of a data caching device according to another embodiment of the present invention;

[0093] Figure 8 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0094] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0095] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0096] Since high-concurrency query capabilities are essential for users, and the ClickHouse engine lacks a universal solution for handling such high-frequency operations, improving ClickHouse's concurrent query capabilities is a key concern. In scenarios requiring increased database concurrent queries, and without considering complex underlying storage engine upgrades, an effective solution is to provide a pre-caching mechanism to enable data to be retrieved once and used multiple times.

[0097] Currently, the two most common caching solutions are as follows:

[0098] 1) Caching capabilities can be added to the client side. Result sets can be cached based on specific caching rules, thereby improving concurrent access capabilities by reducing the traffic to the database. A common caching solution is Redis.

[0099] 2) Provide a proxy layer on the server side to increase caching capabilities, perform unified SQL parsing, and control the traffic accessing the database.

[0100] The solutions above are common approaches to providing caching capabilities, but these solutions have the following problems:

[0101] 1) The development and risk control are quite difficult: For example, in the second caching solution, the server directly provides a proxy layer, which requires some SQL AST (Abstract Syntax Tree) parsing work, which puts a lot of pressure on the development team and is easy to affect upstream and downstream businesses during the upgrade process, so the risk is high;

[0102] 2) Redis large key performance issues: Conventional caching solutions rely more on Redis to cache result sets. However, the data query scenarios for ClickHouse are quite diverse. In addition to simple result sets, there may also be large result set queries (such as detailed queries). When dealing with value data exceeding 10k, Redis cache storage has the problem of large keys, which may lead to uneven memory distribution and blocked queries.

[0103] 3) Problem of lossy cached result sets: Existing Redis caching storage solutions rely on the TTL (Time to Live) expiration mechanism of keys to manage cache invalidation. Although this avoids database pressure, it only caches data within a certain period of the TTL period. This causes inconsistency between cached data and real-time data sources, i.e., the temporary and lossy nature of cached results.

[0104] To address the above technical issues, this invention proposes a data caching method that can improve ClickHouse's concurrent query capabilities. This method enables client-side plug-in upgrades, large object storage, and lossless cache query capabilities.

[0105] Figure 1 This is a schematic diagram of the main flow of a data caching method according to an embodiment of the present invention. As one embodiment of the present invention, such as... Figure 1 As shown, the data caching method is applied to the server side and may include:

[0106] Step 101: Monitor each node of the distributed scheduling center.

[0107] The architecture diagram of this invention embodiment is as follows: Figure 2 As shown, the overall architecture is divided into two parts: the server and the client. The core function of the server is to provide real-time computing capabilities. By monitoring changes in database tables during batch writes, it updates the cache identifiers of ClickHouse's database tables in the distributed scheduling center (such as Zookeeper) nodes, thus synchronizing the cache identifiers with the database change information. The directories of the nodes in the distributed scheduling center correspond one-to-one with the data tables in each database, and the node information stored includes the data change timestamps of the data tables corresponding to that node.

[0108] Optionally, the distributed scheduling center retrieves messages from the message queue, parses the messages to obtain database information, data table information, and data change timestamps, thereby changing the node information stored by the nodes corresponding to the database and the data tables; wherein, the messages are generated by the data storage engine and written to the message queue. Figure 2 As shown in the diagram, the identifiers are explained as follows: {db} represents the database that sent the data change, {table} represents the table whose data changed, {time} represents the timestamp of the data change, and {batch_write_period} represents the batch write period. The entire server includes components such as a message queue (MQ), a distributed scheduling center (Zookeeper), and Redis. After the real-time computation update ClickHouse service writes data in batches to ClickHouse, it sends an asynchronous message and writes the asynchronous message to the message queue. The message contains information about the database that changed data (which can be the database name or identifier), the table information (which can be the table name or identifier), and the timestamp of the data change. The server uses message listening to implement updates from the distributed scheduling center. Specifically, the distributed scheduling center retrieves messages from the message queue, parses out the database and table information that changed data, and then updates temporary nodes in the {db} / {table} directory of the distributed scheduling center to update the ClickHouse data cache identifier. The node stores the latest timestamp {time} of the table update.

[0109] Step 102: If a change is detected in the node information stored by the node, a cache identifier and a Bloom filter are generated based on the directory of the node and the changed node information.

[0110] The server monitors whether the node information in the distributed scheduling center has changed. When it detects that the node information stored in the distributed scheduling center has changed, it initializes and generates a Bloom filter with the latest timestamp {time}.

[0111] Optionally, the cache identifier includes the directory of the node and the data change timestamp of the data table corresponding to the node. Since each data table in each database corresponds one-to-one, the cache identifier is {db}_{table}_{time}. The latest Bloom filter can be matched using the cache identifier. In this embodiment of the invention, the Bloom filter pool is updated according to the cache identifier, and the latest Bloom filter is initialized to prepare for subsequent matching queries by the client.

[0112] Step 103: Associate the cache identifier with the Bloom filter, and store the cache identifier and its corresponding Bloom filter in the filter set.

[0113] The server associates the cache identifier with the Bloom filter and stores the cache identifier and its corresponding Bloom filter in a filter set, which can be stored in Redis. This embodiment of the invention uses Redis's bit structure (SETBIT) to implement the Bloom filter function. The cache identifier is {db}_{table}_{time}, and all identifiers in the Bloom filter are set to 0.

[0114] Optionally, after step 103, the method further includes: configuring the failure timestamp of the Bloom filter; wherein the failure timestamp is the current timestamp plus the failure duration, and the failure duration is greater than the batch write cycle of the data storage engine. For example, the failure duration can be configured to be twice the ClickHouse batch write cycle, i.e., 2*{batch_write_period}, which facilitates the automatic recycling of older Bloom filters.

[0115] By following the steps above, the server can monitor updates to data in different database tables within ClickHouse and update cache identifiers in real time, thereby ensuring the generation of the latest Bloom filters.

[0116] Because ClickHouse is characterized by batch writes, it can monitor writes at specific times. This embodiment of the invention uses ClickHouse to monitor updated data and update the latest cache identifier. On the server side, it implements real-time updates of the Bloom filter and automatic invalidation of old filters, so that the client can query the latest Bloom filter, thereby achieving lossless cache query capability.

[0117] Based on the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problems of high development difficulty, uneven memory distribution, blocked queries, or inconsistent data in the prior art by generating a cache identifier and a Bloom filter based on the node's directory and the changed node information if a change in the node information stored in the monitored node is detected. The node's directory corresponds one-to-one with each data table in each database, and the node information includes the data table's data change timestamp. The embodiments of the present invention improve concurrent query capabilities through pre-filter matching and can also achieve real-time updates of the Bloom filter and automatic invalidation of old filters, so that the client can query the latest Bloom filter, thereby achieving lossless caching query capabilities.

[0118] Figure 3 This is a schematic diagram of the main flow of a data caching method according to another embodiment of the present invention. As yet another embodiment of the present invention, such as... Figure 3 As shown, the data caching method is applied to the client and may include:

[0119] Step 301: Monitor the target node of the distributed scheduling center.

[0120] like Figure 2 As shown, similar to the server, the client also monitors whether the node information of the distributed scheduling center (such as Zookeeper) has changed. The directories of the nodes in the distributed scheduling center correspond one-to-one with the data tables in each database, and the node information stored includes the data change timestamps of the corresponding data tables. Optionally, the client caching service can be encapsulated as an SPI plugin extension service. After the user introduces the plugin service, the client caching service starts along with the user service, connects to the distributed scheduling center, and monitors the nodes of the distributed scheduling center.

[0121] The ClickHouse database and table information to be monitored by the client can be pre-configured, which refers to the target nodes being monitored. The node information stored in the target node includes the data change timestamps of the data tables corresponding to the target node. Each client monitors different database and table information, so different clients can monitor different target nodes. Each client can monitor one target node or multiple target nodes; this embodiment of the invention does not impose any restrictions on this. It should be noted that different clients are equivalent to different business systems, and each business system monitors different databases and tables. Therefore, the ClickHouse database and table information to be monitored by each client needs to be pre-configured.

[0122] Step 302: If a change is detected in the node information stored by the target node, the changed node information is retrieved.

[0123] The client monitors for changes in node information stored in the distributed scheduling center. When a change is detected, the client retrieves the updated node information (i.e., the latest timestamp {time} of the updated data table) to its local machine. Based on the target node configured on the client, the client retrieves the latest node information from the remote distributed scheduling center at path {db} / {table}, forming a JVM caching capability with cache identifiers. This plugin-based approach significantly reduces system coupling and minimizes code intrusion into existing projects.

[0124] In an embodiment of the present invention, the client uses a watcher monitoring mechanism to synchronize information with the remote distributed scheduling center. When the node information under the directory {db} / {table} of the distributed scheduling center changes, the client can pull the latest node information to its local machine in real time and update the latest {time} information of the relevant tables.

[0125] Step 303: Receive a data query request. The data query request carries database information, data table information, and input parameters.

[0126] like Figure 4 As shown, AOP proxy can be used to intercept API requests. By configuring the API annotation of the aspect to access ClickHouse's database table information, query requests to access that database table can be intercepted.

[0127] Step 304: Match the data change timestamps based on the database information and the data table information to generate a cache identifier.

[0128] Since the client locally stores the latest timestamp {time} corresponding to the database table information, the latest timestamp {time} can be matched with the database table information accessed in the request to generate a cache identifier.

[0129] Optionally, the cache identifier includes the database information, the data table information, and the data change timestamp, which facilitates the client to pull the latest Bloom filter from the filter set.

[0130] Step 305: Obtain the Bloom filter corresponding to the cache identifier from the filter set.

[0131] The client obtains the latest Bloom filter through the cache identifier, thereby synchronizing the cache identifier with the database change information and achieving lossless cache query capability.

[0132] Step 306: Obtain the query data corresponding to the data query request based on the input parameters and the Bloom filter.

[0133] After obtaining the latest Bloom filter, the client retrieves the query data corresponding to the data query request from the cache or ClickHouse database based on the input parameters carried in the request and the value of the Bloom filter.

[0134] Optionally, step 306 may include: generating a string identifier based on the class path, method name, and input parameters of the interface provided by the client; calculating the identifier bit using a hash function; determining whether the Bloom filter has a hit based on the identifier bit; if yes, retrieving the query data corresponding to the string identifier from the cache; if no, retrieving the query data corresponding to the data query request from the data table in the database. In an embodiment of the present invention, the client generates a string identifier based on the class path, method name, and sequential input parameters of the interface it provides, then calculates the identifier bit of the string identifier using a hash function, and finally determines whether the latest Bloom filter has a hit based on the identifier bit. If it has a hit, it means that the query data may exist in the cache, and the query data can be directly retrieved from the cache; if it has a hit, it means that the query data does not exist in the cache, and the query data needs to be retrieved from the ClickHouse database.

[0135] Optionally, such as Figure 4 As shown, after generating a globally unique string identifier key, you can further generate identifier bits (i.e., bit arrays) using three different hash methods. Finally, determine whether each generated identifier bit is 1 in the Bloom filter. If it is, the Bloom filter has hit and the query data may exist in the cache. If not, the Bloom filter has not hit and the query data does not exist in the cache.

[0136] Optionally, you can query the values ​​of Redis Bloom filters in batches via a pipeline, such as using the command GETBITkey offset. If the query fails to find the value, it means that the data is not cached. If the query succeeds, it may have been cached, and you need to query the cache (e.g., HBase).

[0137] Optionally, generating a string identifier based on the class path, method name, and input parameters of the interface provided by the client includes: encapsulating the class path, method name, and input parameters of the interface provided by the client into a string; and encrypting the string using a message digest algorithm to obtain the string identifier. Figure 4 As shown, the client intercepts user access through a unified proxy, encapsulates the provided interface's class path, method name, and sequential parameters into a string `{app}_{class name}_{method name}_{sequential parameters}`, and further uses a message digest algorithm (such as MD5) to calculate a globally unique access string identifier as the key. The corresponding query data (value) can then be retrieved from the cache using this string identifier.

[0138] Optionally, retrieving the query data corresponding to the string identifier from the cache includes: determining whether the cache has been hit based on the string identifier; if so, retrieving the query data corresponding to the string identifier from the cache; if not, retrieving the query data corresponding to the data query request from the data table in the database.

[0139] In an embodiment of the present invention, the client's subsequent logic regarding whether a match has occurred is as follows:

[0140] 1) If the Bloom filter hits, continue to query the cached data based on the string identifier key calculated by the message digest algorithm. If found, return the query result directly.

[0141] 2) If the Bloom filter hits but the cache misses, it indicates that the Bloom filter is ineffective, so proceed to step 3).

[0142] 3) If the Bloom filter misses, continue using Spring reflection to invoke the ClickHouse query based on the client interface and retrieve the query result set. Store the value of the query result set from the reflection call in the cache and synchronously update the latest Bloom filter value. Optionally, if the result set is very large (e.g., >10M), store it in HDFS, with HBase only storing the HDFS path information. If the result set is small (e.g., ≤10M), store it in HBase according to the value (hbase.client.keyvalue.maxsize defaults to 10M).

[0143] By following the steps above, the client can enhance its ClickHouse service queries, intercept external accesses through a client-side cache proxy, and improve the high concurrency capabilities for ClickHouse services. This invention is suitable for storing large result sets and accelerates query performance through a Bloom filter set. Furthermore, this invention is highly versatile and applicable to API queries, large-scale offline detailed caching, etc., and is particularly suitable for cache invalidation management scenarios involving batch database changes.

[0144] It should be noted that the core function of the client is to implement the unified proxy caching service capability. The client-side SPI enhanced service capability can be provided to users in the form of a JAR plugin. The proxy function is implemented through the plugin capability. The latest Bloom filter is identified through interceptors and cache flag listeners. The Bloom filter is queried for whether it is hit through various hash mapping methods. Then, the cache hit is matched, and the first query result is stored in the cache as a pre-cache for subsequent queries.

[0145] Therefore, the embodiments of the present invention can support the storage of large result sets, and realize ClickHouse's lossless caching capability and high concurrency access capability.

[0146] Based on the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problems of high development difficulty, uneven memory distribution, blocked queries, or inconsistent data in the prior art by using a technical means that, if a change is detected in the node information stored by the target node, the changed node information is retrieved, a cache identifier is generated based on the database table information and the data change timestamp, and the corresponding Bloom filter is obtained. The embodiments of the present invention store cached results based on Bloom filters and caching schemes, which can not only ensure the storage of large result sets, but also improve concurrent query capabilities through pre-filter matching. The client, by monitoring nodes in the distributed scheduling, can achieve real-time synchronization of the latest cache identifier, obtain the latest Bloom filter, and ensure consistency between the cached query results and the database data, thereby achieving lossless cache query capabilities. Moreover, the client only needs to introduce a client component as a plug-in to enhance caching capabilities, thus reducing development difficulty.

[0147] Figure 5 This is a schematic diagram of the main flow of a data caching method according to a possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 5 As shown, the data caching method is applied to the client and may include:

[0148] Step 501: Monitor the target node of the distributed scheduling center.

[0149] The client monitors whether the node information of the distributed scheduling center (such as Zookeeper) has changed. The directory of the nodes in the distributed scheduling center corresponds one-to-one with each data table in each database. The node information stored by the target node includes the data change timestamp of the data table corresponding to the target node.

[0150] The ClickHouse database table information that the client listens to can be pre-configured, which is the target node to listen to. The node information stored in the target node includes the data change timestamp of the data table corresponding to the target node.

[0151] Step 502: If a change is detected in the node information stored by the target node, the changed node information is retrieved.

[0152] The client monitors for changes in node information stored in the distributed scheduling center. When a change is detected, the client retrieves the updated node information (i.e., the latest timestamp {time} of the updated data table) to its local machine. Based on the target node configured on the client, the client retrieves the latest timestamp {time} from the remote distributed scheduling center at path {db} / {table}.

[0153] Step 503: Receive a data query request. The data query request carries database information, data table information, and input parameters.

[0154] The client implements interface interception capabilities through AOP proxy, configuring access to ClickHouse's database table information in the interface annotation of the aspect, thereby intercepting query requests to access that database table.

[0155] Step 504: Match the data change timestamps with the database information and the data table information to generate a cache identifier.

[0156] Since the client locally stores the latest timestamp {time} corresponding to the database and table information, the latest timestamp {time} can be matched based on the database and table information accessed in the request to ClickHouse, thereby generating a cache identifier. The cache identifier includes the database information, the table information, and the data change timestamp.

[0157] Step 505: Obtain the Bloom filter corresponding to the cache identifier from the filter set.

[0158] The client obtains the latest Bloom filter through the cache identifier, thereby synchronizing the cache identifier with the database change information and achieving lossless cache query capability.

[0159] Step 506: Encapsulate the class path, method name, and input parameters of the interface provided by the client into a string.

[0160] Step 507: Encrypt the string using a message digest algorithm to obtain a string identifier.

[0161] The string can be encrypted using MD5 to obtain a globally unique string identifier key.

[0162] Step 508: Calculate the identifier of the string using a hash function to obtain the identifier bit.

[0163] Various hash functions can be used to calculate the identifier bits of the string to improve the accuracy of the Bloom filter.

[0164] Step 509: Determine whether the Bloom filter is hit based on the flag bit; if yes, proceed to step 510; if no, proceed to step 511.

[0165] Step 510: Retrieve the query data corresponding to the string identifier from the cache.

[0166] If the Bloom filter hits, the query data may exist in the cache, requiring further cache lookup. Optionally, step 510 may include: determining whether the cache has been hit based on the string identifier; if so, retrieving the query data corresponding to the string identifier from the cache; if not, retrieving the query data corresponding to the data query request from the data table in the database.

[0167] Step 511: Obtain the query data corresponding to the data query request from the data table in the database.

[0168] If the Bloom filter misses, the query data is not in the cache, and the query data corresponding to the data query request needs to be retrieved from the data table in the database.

[0169] Step 512: Store the string identifier and the query data corresponding to the data query request in the cache, and update the value of the Bloom filter based on the identifier bit.

[0170] By using Spring reflection via invoke, the system queries ClickHouse based on the client interface and retrieves the query result set. The value of the query result set invoked by reflection is stored in the cache, and the latest Bloom filter value is updated synchronously.

[0171] Optionally, step 512 may include: determining whether the dataset size of the query data corresponding to the data query request is greater than or equal to a threshold; if yes, storing the query data in HDFS and storing the string identifier and the storage path of the query data in HDFS in HBase; if no, storing the string identifier and the query data in HBase.

[0172] Optionally, if the result set is very large (e.g., >20M), it is stored in HDFS, and HBase only stores the HDFS path information. If the result set is not large (e.g., ≤20M), it is stored in HBase according to the value.

[0173] Furthermore, the specific implementation details of the data caching method in one of the reference embodiments of the present invention have been described in detail in the data caching method described above, so the details will not be repeated here.

[0174] Figure 6 This is a schematic diagram of the main modules of a data caching device according to an embodiment of the present invention. Figure 6 As shown, the data caching device 600 includes a first monitoring module 601, a generation module 602, and a storage module 603. The first monitoring module 601 monitors each node of the distributed scheduling center. Each node's directory corresponds one-to-one with each data table in each database, and the node information stored by each node includes the data change timestamp of the corresponding data table. The generation module 602, if it detects a change in the node information stored by the node, generates a cache identifier and a Bloom filter based on the node's directory and the changed node information. The storage module 603 associates the cache identifier with the Bloom filter and stores the cache identifier and its corresponding Bloom filter in a filter set.

[0175] Optionally, the cache identifier includes the directory of the node and the data change timestamp of the data table corresponding to the node.

[0176] Optionally, the distributed scheduling center retrieves messages from the message queue, parses the messages to obtain database information, data table information, and data change timestamps, and thereby changes the node information stored by the nodes corresponding to the database and the data table.

[0177] The message is generated by the data storage engine and written into the message queue.

[0178] Optionally, the storage module 603 is further configured to:

[0179] After storing the cache identifier and its corresponding Bloom filter in the filter set, configure the expiration timestamp of the Bloom filter; wherein, the expiration timestamp is the current timestamp plus the expiration duration, and the expiration duration is greater than the batch write cycle of the data storage engine.

[0180] It should be noted that the specific implementation details of the data caching device described in this invention have been described in detail in the data caching method described above, so the details will not be repeated here.

[0181] Figure 7 This is a schematic diagram of the main modules of a data caching device according to an embodiment of the present invention. Figure 7 As shown, the data caching device 700 includes a second monitoring module 701 and a pull module 702; wherein, the second monitoring module 701 is used to monitor the target node of the distributed scheduling center; wherein, the directory of the node of the distributed scheduling center corresponds one-to-one with each data table in each database, and the node information stored by the target node includes the data change timestamp of the data table corresponding to the target node; the pull module 702 is used to pull the changed node information if it detects that the node information stored by the target node has changed.

[0182] Optionally, it also includes a processing module for:

[0183] Receive a data query request; wherein the data query request carries database information, data table information and input parameters;

[0184] The data change timestamp is matched with the database information and the data table information to generate a cache identifier;

[0185] Retrieve the Bloom filter corresponding to the cache identifier from the filter set;

[0186] The query data corresponding to the data query request is obtained based on the input parameters and the Bloom filter.

[0187] Optionally, the cache identifier includes the database information, the data table information, and the data change timestamp.

[0188] Optionally, the processing module is further configured to:

[0189] Generate a string identifier based on the class path, method name, and input parameters of the interface provided by the client;

[0190] The string identifier is calculated using a hash function to obtain the identifier bit;

[0191] Based on the identifier, determine whether the Bloom filter has a hit; if yes, retrieve the query data corresponding to the string identifier from the cache; if no, retrieve the query data corresponding to the data query request from the data table in the database.

[0192] Optionally, the processing module is further configured to:

[0193] The class path, method name, and input parameters of the interface provided by the client are encapsulated into strings;

[0194] The string is encrypted using a message digest algorithm to obtain a string identifier.

[0195] Optionally, the processing module is further configured to:

[0196] Determine whether the cache hit is based on the string identifier;

[0197] If so, retrieve the query data corresponding to the string identifier from the cache;

[0198] If not, then retrieve the query data corresponding to the data query request from the data table in the database.

[0199] Optionally, the processing module is further configured to:

[0200] After obtaining the query data corresponding to the data query request from the data table in the database, the string identifier and the query data corresponding to the data query request are stored in the cache, and the value of the Bloom filter is updated based on the identifier bit.

[0201] Optionally, the processing module is further configured to:

[0202] Determine whether the size of the dataset corresponding to the data query request is greater than or equal to the threshold;

[0203] If so, the query data is stored in HDFS, and the string identifier and the storage path of the query data in HDFS are stored in HBase;

[0204] If not, the string identifier and the query data are stored in HBase.

[0205] It should be noted that the specific implementation details of the data caching device described in this invention have been described in detail in the data caching method described above, so the details will not be repeated here.

[0206] Figure 8 An exemplary system architecture 800 is shown where the data caching method or data caching device of the present invention can be applied.

[0207] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, and 803, a network 804, and a server 805. Network 804 serves as the medium for providing communication links between terminal devices 801, 802, and 803 and server 805. Network 804 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0208] Users can use terminal devices 801, 802, and 803 to interact with server 805 via network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0209] Terminal devices 801, 802, and 803 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0210] Server 805 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 801, 802, and 803 (for example only). The backend management server can analyze and process data such as received item information query requests, and feed back the processing results (such as target push information and item information—for example only) to the terminal devices.

[0211] It should be noted that the data caching method provided in this embodiment of the invention is generally executed by server 805, and correspondingly, the data caching device is generally installed in server 805. The data caching method provided in this embodiment of the invention can also be executed by terminal devices 801, 802, and 803, and correspondingly, the data caching device can be installed in terminal devices 801, 802, and 803.

[0212] It should be understood that Figure 8 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0213] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 900 suitable for implementing a terminal device of the present invention. Figure 9 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0214] like Figure 9As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded into random access memory (RAM) 903 from storage section 908. The RAM 903 also stores various programs and data required for the operation of the system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0215] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0216] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this invention.

[0217] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0218] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer programs according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0219] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor; for example, a processor can be described as including a first listening module, a generation module, and a storage module, wherein the names of these modules do not necessarily limit the module itself.

[0220] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor; for example, a processor can be described as including a second listening module, a pull module, and a processing module, wherein the names of these modules do not necessarily limit the module itself.

[0221] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: monitoring each node of a distributed scheduling center; wherein the directory of the node corresponds one-to-one with each data table in each database, and the node information stored by the node includes the data change timestamp of the data table corresponding to the node; if a change in the node information stored by the node is detected, a cache identifier and a Bloom filter are generated based on the directory of the node and the changed node information; the cache identifier is associated with the Bloom filter, and the cache identifier and its corresponding Bloom filter are stored in a filter set.

[0222] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: monitoring a target node of a distributed scheduling center; wherein the directory of the nodes in the distributed scheduling center corresponds one-to-one with each data table in each database, and the node information stored by the target node includes the data change timestamp of the data table corresponding to the target node; if a change is detected in the node information stored by the target node, the changed node information is retrieved.

[0223] According to the technical solution of this invention, by employing the method of retrieving the changed node information if a change is detected in the node information stored by the target node, generating a cache identifier based on the database table information and data change timestamp, and then obtaining the Bloom filter corresponding to the cache identifier, the technical problems of high development difficulty, uneven memory distribution, blocked queries, or inconsistent data in the prior art are overcome. This invention, based on Bloom filters and a caching scheme, stores cached results, ensuring the storage of large result sets and improving concurrent query capabilities through pre-filter matching. The client, by monitoring nodes in the distributed scheduling, can achieve real-time synchronization of the latest cache identifier, obtaining the latest Bloom filter and ensuring consistency between the cached query results and the database data, thus achieving lossless cache query capabilities. Furthermore, the client only needs to import a client component as a plug-in to enhance caching capabilities, thereby reducing development difficulty.

[0224] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data caching method, characterized in that, Applied to the server side, including: The system monitors each node of the distributed scheduling center; wherein the directory of each node corresponds one-to-one with each data table in each database, and the node information stored by each node includes the data change timestamp of the data table corresponding to the node; the distributed scheduling center obtains messages from the message queue, the messages containing database information and data table information where data changes have occurred, and the timestamp of the data change, and parses the messages to obtain the database information, data table information, and data change timestamp, thereby changing the node information stored by the nodes corresponding to the database and the data table; If a change is detected in the node information stored by the node, a cache identifier and a Bloom filter are generated based on the node's directory and the changed node information. Associate the cache identifier with the Bloom filter, and store the cache identifier and its corresponding Bloom filter in the filter set; Configure the failure timestamp of the Bloom filter; wherein the failure timestamp is the current timestamp plus the failure duration, and the failure duration is greater than the batch write cycle of the data storage engine.

2. The method according to claim 1, characterized in that, The cache identifier includes the directory of the node and the data change timestamp of the data table corresponding to the node.

3. The method according to claim 1, characterized in that, The message is generated by the data storage engine and written to the message queue.

4. A data caching method, characterized in that, Applied to the client side, including: The system monitors the target node of the distributed scheduling center. The directory of the nodes in the distributed scheduling center corresponds one-to-one with each data table in each database. The node information stored by the target node includes the data change timestamp of the data table corresponding to the target node. The distributed scheduling center retrieves a message from a message queue. The message contains database and data table information where data changes have occurred, as well as the timestamp of the data change. The system parses the message to obtain the database information, data table information, and data change timestamp, thereby updating the node information stored by the nodes corresponding to the database and the data table. If a change is detected in the node information stored by the target node, the changed node information is retrieved. Receive a data query request; wherein the data query request carries database information, data table information and input parameters; The data change timestamp is matched with the database information and the data table information to generate a cache identifier; Retrieve the Bloom filter corresponding to the cache identifier from the filter set; The query data corresponding to the data query request is obtained based on the input parameters and the Bloom filter.

5. The method according to claim 4, characterized in that, The cache identifier includes the database information, the data table information, and the data change timestamp.

6. The method according to claim 4, characterized in that, Based on the input parameters and the Bloom filter, the query data corresponding to the data query request is obtained, including: Generate a string identifier based on the class path, method name, and input parameters of the interface provided by the client; The string identifier is calculated using a hash function to obtain the identifier bit; Based on the identifier, determine whether the Bloom filter has a hit; if yes, retrieve the query data corresponding to the string identifier from the cache; if no, retrieve the query data corresponding to the data query request from the data table in the database.

7. The method according to claim 6, characterized in that, Generate a string identifier based on the class path, method name, and input parameters of the interface provided by the client, including: The class path, method name, and input parameters of the interface provided by the client are encapsulated into strings; The string is encrypted using a message digest algorithm to obtain a string identifier.

8. The method according to claim 6, characterized in that, Retrieving the query data corresponding to the string identifier from the cache includes: Determine whether the cache hit is based on the string identifier; If so, retrieve the query data corresponding to the string identifier from the cache; If not, then retrieve the query data corresponding to the data query request from the data table in the database.

9. The method according to claim 6, characterized in that, After retrieving the query data corresponding to the data query request from the data table in the database, the method further includes: Store the string identifier and the query data corresponding to the data query request in the cache, and update the value of the Bloom filter based on the identifier bit.

10. The method according to claim 9, characterized in that, Storing the string identifier and the query data corresponding to the data query request in the cache includes: Determine whether the size of the dataset corresponding to the data query request is greater than or equal to the threshold; If so, the query data is stored in HDFS, and the string identifier and the storage path of the query data in HDFS are stored in HBase; If not, the string identifier and the query data are stored in HBase.

11. A data caching device, characterized in that, Configured on the server side, including: The first monitoring module is used to monitor each node of the distributed scheduling center. Each node's directory corresponds one-to-one with each data table in each database. The node information stored by each node includes the data change timestamp of the corresponding data table. The distributed scheduling center retrieves messages from a message queue. These messages contain database and data table information about the data change, as well as the timestamp of the data change. The center parses the messages to obtain the database information, data table information, and data change timestamp, thereby updating the node information stored by the nodes corresponding to the database and data table. The generation module is used to generate a cache identifier and a Bloom filter based on the directory of the node and the changed node information if a change is detected in the node information stored by the node. A storage module is used to associate the cache identifier with the Bloom filter and store the cache identifier and its corresponding Bloom filter in a filter set; configure the expiration timestamp of the Bloom filter; wherein the expiration timestamp is the current timestamp plus the expiration duration, and the expiration duration is greater than the batch write cycle of the data storage engine.

12. A data caching device, characterized in that, Settings on the client side include: The second monitoring module is used to monitor the target nodes of the distributed scheduling center. The directories of the nodes in the distributed scheduling center correspond one-to-one with the data tables in each database. The node information stored by the target node includes the data change timestamps of the data tables corresponding to the target node. The distributed scheduling center retrieves messages from the message queue. These messages contain database and data table information about the data changes, as well as the timestamps of the data changes. The center parses the messages to obtain the database information, data table information, and data change timestamps, thereby modifying the node information stored by the nodes corresponding to the databases and data tables. The pull module is used to pull the changed node information if it detects that the node information saved by the target node has changed. The processing module is used to receive data query requests; wherein the data query request carries database information, data table information and input parameters; match the data change timestamps according to the database information and the data table information to generate a cache identifier; obtain the Bloom filter corresponding to the cache identifier from the filter set; and obtain the query data corresponding to the data query request based on the input parameters and the Bloom filter.

13. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

14. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images