Data query management method and device based on dynamic cache optimization

Abstract

The invention relates to the technical field of information management methods, and discloses a data query management method based on dynamic cache optimization. Comprising the steps of constructing a dynamic aging adjustment engine, and automatically adjusting cache time based on data change frequency; real-time monitoring is carried out through a database trigger and log analysis; the method comprises the following steps: acquiring a historical query record, predicting query types and time possibly initiated by a user in the future through a machine learning model, before the query time is predicted, loading predicted query type data to a cache, dynamically marking hotspot data according to an access frequency, preferentially allocating cache resources to the hotspot data, and prolonging the time efficiency; constructing a multi-level cache architecture based on a local cache and a distributed cache, and acquiring data by adopting a cache degradation strategy; an enhanced Kafka message processing mechanism is introduced; the method has the advantages of improving data query efficiency and ensuring data consistency.

Description

Technical Field

[0001] This invention relates to the field of information management methods, and in particular to a data query management method and apparatus based on dynamic caching optimization. Background Technology

[0002] The current workflow query function has several significant issues. Firstly, data query efficiency is low. For example, personal workflow queries (such as lists of processed, initiated, and pending approval workflows, along with alert statistics) involve data from across the entire network, and these queries are frequently accessed on the PC homepage and in the app. Each query directly accesses the database, resulting in numerous database I / O operations, long response times, and a negative impact on user experience. Secondly, data consistency is difficult to guarantee. When approval nodes in a workflow change (such as initiating a new workflow or conducting workflow approval), the lack of an effective cache clearing mechanism may lead to inconsistencies between cached data and the actual data in the database. Users still see the old cached data, affecting the accuracy of business decisions and operations.

[0003] Therefore, a data query management method and device based on dynamic caching optimization are provided to solve the above problems. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problems in the prior art, such as low efficiency of traditional data query, poor data consistency, rigid caching strategies, and insufficient reliability.

[0005] The first aspect of this invention provides a data query management method based on dynamic caching optimization, the data query management method based on dynamic caching optimization includes: Build a dynamic timeliness adjustment engine to automatically adjust cache time based on data change frequency; perform real-time monitoring through database triggers and log analysis; Obtain historical query records, use machine learning models to predict the types and times of queries that users may initiate in the future, load the predicted query type data into the cache before predicting the query time, dynamically mark hot data according to access frequency, prioritize the allocation of cache resources for hot data and extend its validity period; Build a multi-level caching architecture based on local and distributed caching, and use a cache degradation strategy to retrieve data; An enhanced Kafka message processing mechanism was introduced.

[0006] Optionally, the construction of a dynamic timeliness adjustment engine to automatically adjust the cache time based on the data change frequency includes: Obtain the number of times data is updated per unit of time, and divide the data into high-frequency updates, medium-frequency updates, and low-frequency updates; When the data is determined to be frequently updated, the cache time is shortened to 10 minutes; If the data is determined to be low-frequency updated data, the cache time will be extended to 24 hours. When the data is determined to be medium-frequency updated data, the cache time is kept at 1 hour.

[0007] Optionally, the real-time monitoring via database triggers and log analysis includes: Create a trigger on the table where data update operations occur. When data is inserted, updated, or deleted, the trigger immediately captures the change time and sends a change notification. Obtain log information, analyze the log information, extract the time information of data insertion, update or deletion operations in the log information, and send change notifications.

[0008] Optionally, the step of obtaining historical query records and predicting the type and time of future queries that users may initiate through machine learning models includes: A user behavior analysis platform is built to collect users' historical query records, which include query time information, query type information, and query condition information. A user behavior prediction model is constructed using a long short-term memory network. The format of historical query records is converted, and the user behavior prediction model is trained using the user's historical query records; Using historical query records as input, predict the types and times of queries that users may initiate in the near future.

[0009] Optionally, loading the predicted query type data into the cache before predicting the query time includes: Based on the query type and query time predicted by the user behavior prediction model, a preloading task is started before the predicted query time. The preloading task queries relevant data from the database and loads it into the cache; The method of dynamically marking hot data based on access frequency, prioritizing the allocation of caching resources for hot data and extending its validity period includes: Establish a hotspot data identification module to statistically analyze the access frequency of data in real time; The sliding window algorithm is used to calculate the number of times each data is accessed per unit time; when the number of accesses exceeds a set threshold, the data is marked as hot data. For frequently accessed data, prioritize allocating cache resources to extend cache validity.

[0010] Optionally, the construction of a multi-level caching architecture based on local caching and distributed caching, and the use of a cache degradation strategy to retrieve data, includes: Build a local cache, using Redis as the in-memory database; build a distributed cache using a consistent hashing algorithm, using Memcached or RedisCluster as the distributed cache solution, and construct a multi-level cache architecture based on local and distributed caches. A cache degradation strategy is employed to retrieve data. Specifically, when the local cache expires, the system automatically retrieves data from the distributed cache. If the required data is also not found in the distributed cache, a further query is performed on the database, and the query results are sequentially returned to both the distributed and local caches, while simultaneously updating the caches.

[0011] Optionally, the introduction of the enhanced Kafka message processing mechanism includes: An ACK mechanism is introduced on the Kafka consumer side. After successfully processing a message, the consumer sends an acknowledgment message to the Kafka cluster. If an exception occurs or the consumer fails to send an acknowledgment during message processing, Kafka puts the message back into the retry queue. To ensure that cache clearing messages for the same process are processed in order, Kafka's partitioning mechanism is used; when producing messages, the process ID is used as the partition key to send messages for the same process to the same partition. Set up a scheduled task to periodically scan the cache; check for any residual cache data after message processing failures, and if any uncleared cache is found, perform a clearing operation.

[0012] A second aspect of the present invention provides a data query management device based on dynamic caching optimization, the data query management device based on dynamic caching optimization comprising: The dynamic timeliness adjustment module is used to build a dynamic timeliness adjustment engine, which automatically adjusts the cache time based on the frequency of data changes; and performs real-time monitoring through database triggers and log analysis. The user behavior prediction module is used to obtain historical query records, predict the types and times of queries that users may initiate in the future through machine learning models, load the predicted query type data into the cache before predicting the query time, dynamically mark hot data according to access frequency, prioritize the allocation of cache resources for hot data and extend the validity period. The multi-level caching architecture building module is used to build a multi-level caching architecture based on local caching and distributed caching, and to retrieve data using a cache degradation strategy. The message processing mechanism establishment module is used to introduce the enhanced Kafka message processing mechanism.

[0013] A third aspect of the present invention provides an electronic device, the electronic device comprising a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the electronic device to perform the various steps of the data query management method based on dynamic cache optimization as described above.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the data query management method based on dynamic caching optimization as described above.

[0015] In the technical solution of this invention, by setting up a cache, frequent database query operations are avoided, thus improving data query efficiency; data can be quickly retrieved from the cache, greatly shortening the query response time and improving the user experience; the cache of relevant personnel can be cleared in a timely manner, so that users can obtain the latest business data when they query again; decision-making errors and operational mistakes caused by outdated cached data are avoided, thus improving the accuracy and reliability of office business processing. Attached Figure Description

[0016] Figure 1 A flowchart of a data query management method based on dynamic caching optimization provided in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a data query management device based on dynamic caching optimization provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] This invention provides a data query management method based on dynamic caching optimization, including: constructing a dynamic timeliness adjustment engine to automatically adjust cache time based on data change frequency; real-time monitoring through database triggers and log analysis; acquiring historical query records, predicting the types and times of queries that users may initiate in the future through machine learning models, loading the predicted query type data into the cache before the predicted query time, dynamically marking hot data according to access frequency, prioritizing the allocation of cache resources and extending the timeliness of hot data; constructing a multi-level caching architecture based on local caching and distributed caching, and adopting a cache degradation strategy to obtain data; and introducing an enhanced Kafka message processing mechanism. This invention solves the technical problems of low efficiency, poor data consistency, rigid caching strategies, and insufficient reliability in existing technologies.

[0018] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the data query management method based on dynamic caching optimization in this invention includes: Build a dynamic timeliness adjustment engine to automatically adjust cache time based on data change frequency; perform real-time monitoring through database triggers and log analysis. The construction of a dynamic timeliness adjustment engine, which automatically adjusts the cache time based on the frequency of data changes, includes: Obtain the number of times data is updated per unit of time, and divide the data into high-frequency updates, medium-frequency updates, and low-frequency updates; When data is determined to be frequently updated, the cache time is shortened to 10 minutes. For frequently updated data, such as data in pending approval processes, the cache time is automatically shortened to 10 minutes when its update frequency exceeds once per minute. This ensures that the data obtained by users is always up-to-date, avoiding inconsistencies in approval information caused by expired cache.

[0020] When data is determined to be infrequently updated, the cache time is extended to 24 hours. For infrequently updated data, such as historical process statistics, if the update frequency is less than once a day, the cache time is also extended to 24 hours. This reduces frequent database access, improves overall system performance, and also reduces the database load. In implementation, the mapping relationship between data types and update frequencies, as well as the corresponding cache time, is defined through a configuration file.

[0021] When the data is determined to be medium-frequency updated data, the cache time is kept at 1 hour.

[0022] The real-time monitoring via database triggers and log analysis includes: Create a trigger on the table where data update operations occur. When data is inserted, updated, or deleted, the trigger immediately captures the change time and sends a change notification. Obtain log information, analyze the log information, extract the time information of data insertion, update or deletion operations in the log information, and send change notifications.

[0023] The databases mentioned above are generally MySQL and Oracle databases.

[0024] Obtain historical query records, use machine learning models to predict the types and times of queries that users may initiate in the future, load the predicted query type data into the cache before predicting the query time, dynamically mark hot data according to access frequency, prioritize the allocation of cache resources for hot data and extend its validity period; Build a multi-level caching architecture based on local and distributed caching, and use a cache degradation strategy to retrieve data; An enhanced Kafka message processing mechanism was introduced.

[0025] Please see Figure 1 The second embodiment of the data query management method based on dynamic caching optimization in this invention includes: Build a dynamic timeliness adjustment engine to automatically adjust cache time based on data change frequency; perform real-time monitoring through database triggers and log analysis; The construction of a dynamic timeliness adjustment engine, which automatically adjusts the cache time based on the frequency of data changes, includes: Obtain the number of times data is updated per unit of time, and divide the data into high-frequency updates, medium-frequency updates, and low-frequency updates; When data is determined to be frequently updated, the cache time is shortened to 10 minutes. For frequently updated data, such as data in pending approval processes, the cache time is automatically shortened to 10 minutes when its update frequency exceeds once per minute. This ensures that the data obtained by users is always up-to-date, avoiding inconsistencies in approval information caused by expired cache.

[0026] When data is determined to be infrequently updated, the cache time is extended to 24 hours. For infrequently updated data, such as historical process statistics, if the update frequency is less than once a day, the cache time is also extended to 24 hours. This reduces frequent database access, improves overall system performance, and also reduces the database load. In implementation, the mapping relationship between data types and update frequencies, as well as the corresponding cache time, is defined through a configuration file.

[0027] When the data is determined to be medium-frequency updated data, the cache time is kept at 1 hour.

[0028] The real-time monitoring via database triggers and log analysis includes: Create a trigger on the table where data update operations occur. When data is inserted, updated, or deleted, the trigger immediately captures the change time and sends a change notification. Obtain log information, analyze the log information, extract the time information of data insertion, update or deletion operations in the log information, and send change notifications.

[0029] The databases mentioned above are generally MySQL and Oracle databases.

[0030] Obtain historical query records, use machine learning models to predict the types and times of queries that users may initiate in the future, load the predicted query type data into the cache before predicting the query time, dynamically mark hot data according to access frequency, prioritize the allocation of cache resources for hot data and extend its validity period; This includes retrieving historical query records and using machine learning models to predict the types and timing of future queries that users may initiate, including: A user behavior analysis platform is built to collect users' historical query records, which include query time information, query type information, and query condition information. A user behavior prediction model is constructed using a Long Short-Term Memory (LSTM) network. LSTM networks can effectively process time-series data and capture the long-term dependencies of user query behavior. The historical query records are formatted, such as converting query time into timestamps and encoding query types; the user's historical query records are used to train the user behavior prediction model; Using historical query records as input, predict the types and times of queries that users may initiate in the near future.

[0031] The step of loading the predicted query type data into the cache before predicting the query time includes: Based on the query type and query time predicted by the user behavior prediction model, a preloading task is started before the predicted query time. The preloading task queries relevant data from the database and loads it into the cache; for example, if it is predicted that a user will query a certain type of data in 10 minutes, the preloading operation will begin 9 minutes in advance. To avoid putting excessive pressure on the database during preloading, asynchronous loading and rate limiting strategies are adopted. The concurrency of preloading tasks is controlled through a thread pool, and a maximum query limit is set for each second.

[0032] The method of dynamically marking hot data based on access frequency, prioritizing the allocation of caching resources for hot data and extending its validity period includes: Establish a hotspot data identification module to statistically analyze the access frequency of data in real time; The sliding window algorithm is used to calculate the number of times each data is accessed per unit time; when the number of accesses exceeds a set threshold, the data is marked as hot data. For frequently accessed data, prioritize allocating more cache resources, such as increasing cache memory space and extending its cache expiration time. Simultaneously, in the cache eviction policy, reduce the priority of evicting frequently accessed data.

[0033] Build a multi-level caching architecture based on local and distributed caching, and use a cache degradation strategy to retrieve data; An enhanced Kafka message processing mechanism was introduced.

[0034] Please see Figure 1 The third embodiment of the data query management method based on dynamic caching optimization in this invention includes: Build a dynamic timeliness adjustment engine to automatically adjust cache time based on data change frequency; perform real-time monitoring through database triggers and log analysis; The construction of a dynamic timeliness adjustment engine, which automatically adjusts the cache time based on the frequency of data changes, includes: Obtain the number of times data is updated per unit of time, and divide the data into high-frequency updates, medium-frequency updates, and low-frequency updates; When data is determined to be frequently updated, the cache time is shortened to 10 minutes. For frequently updated data, such as data in pending approval processes, the cache time is automatically shortened to 10 minutes when its update frequency exceeds once per minute. This ensures that the data obtained by users is always up-to-date, avoiding inconsistencies in approval information caused by expired cache.

[0035] When data is determined to be infrequently updated, the cache time is extended to 24 hours. For infrequently updated data, such as historical process statistics, if the update frequency is less than once a day, the cache time is also extended to 24 hours. This reduces frequent database access, improves overall system performance, and also reduces the database load. In implementation, the mapping relationship between data types and update frequencies, as well as the corresponding cache time, is defined through a configuration file.

[0036] When the data is determined to be medium-frequency updated data, the cache time is kept at 1 hour.

[0037] The real-time monitoring via database triggers and log analysis includes: Create a trigger on the table where data update operations occur. When data is inserted, updated, or deleted, the trigger immediately captures the change time and sends a change notification. Obtain log information, analyze the log information, extract the time information of data insertion, update or deletion operations in the log information, and send change notifications.

[0038] The databases mentioned above are generally MySQL and Oracle databases.

[0039] Obtain historical query records, use machine learning models to predict the types and times of queries that users may initiate in the future, load the predicted query type data into the cache before predicting the query time, dynamically mark hot data according to access frequency, prioritize the allocation of cache resources for hot data and extend its validity period; This includes retrieving historical query records and using machine learning models to predict the types and timing of future queries that users may initiate, including: A user behavior analysis platform is built to collect users' historical query records, which include query time information, query type information, and query condition information. A user behavior prediction model is constructed using a Long Short-Term Memory (LSTM) network. LSTM networks can effectively process time-series data and capture the long-term dependencies of user query behavior. Format conversion of historical query records, such as converting query time to timestamp and encoding query type; use user historical query records to train user behavior prediction models; Using historical query records as input, predict the types and times of queries that users may initiate in the near future.

[0040] The step of loading the predicted query type data into the cache before predicting the query time includes: Based on the query type and query time predicted by the user behavior prediction model, a preloading task is started before the predicted query time. The preloading task queries relevant data from the database and loads it into the cache; for example, if it is predicted that a user will query a certain type of data in 10 minutes, the preloading operation will begin 9 minutes in advance. To avoid putting excessive pressure on the database during preloading, asynchronous loading and rate limiting strategies are adopted. The concurrency of preloading tasks is controlled through a thread pool, and a maximum query limit is set for each second.

[0041] The method of dynamically marking hot data based on access frequency, prioritizing the allocation of caching resources for hot data and extending its validity period includes: Establish a hotspot data identification module to statistically analyze the access frequency of data in real time; The sliding window algorithm is used to calculate the number of times each data is accessed per unit time; when the number of accesses exceeds a set threshold, the data is marked as hot data. For frequently accessed data, prioritize allocating more cache resources, such as increasing cache memory space and extending its cache expiration time. Simultaneously, in the cache eviction policy, reduce the priority of evicting frequently accessed data.

[0042] Build a multi-level caching architecture based on local and distributed caching, and use a cache degradation strategy to retrieve data; Specifically, this includes: A local cache is built, using Redis as the in-memory database, leveraging its high performance and rich data structures. The local cache stores personalized query results, such as user-specific approval process progress and customized report data. Redis's response time can be controlled within ≤10ms, meeting the requirement for fast response. To improve cache hit rate, an LRU (Least Recently Used) cache eviction strategy is adopted. In terms of code implementation, Redis's Python client library (such as redis-py) is used for data operations. A distributed cache is built using the consistent hashing algorithm, with Memcached or RedisCluster chosen as the distributed caching solution. Taking RedisCluster as an example, data is distributed across different nodes based on hash values, achieving data sharding and load balancing. When a client performs data read or write operations, the hash value is calculated based on the data's key to determine the node it resides on, thus enabling access. Build a multi-level caching architecture based on local caching and distributed caching; A cache degradation strategy is employed to retrieve data. Specifically, when the local cache expires, the system automatically retrieves data from the distributed cache. If the required data is also not found in the distributed cache, a further query is performed on the database, and the query results are sequentially returned to the distributed cache and the local cache, while simultaneously updating the caches. To ensure service continuity, reasonable timeout periods and retry mechanisms are set during the cache degradation process. If retrieving data from the distributed cache or the database times out, the old data in the cache (if it exists) is returned, and the exception information is recorded for subsequent troubleshooting.

[0043] An enhanced Kafka message processing mechanism was introduced.

[0044] Specifically, it includes: Message acknowledgment and retry: An ACK (acknowledgment) mechanism is introduced on the Kafka consumer side. After successfully processing a message, the consumer sends an acknowledgment message to the Kafka cluster. If an exception occurs or a timeout occurs during message processing and no acknowledgment is sent, Kafka puts the message back into the retry queue (DeadLetterQueue).

[0045] Ordering Guarantee: To ensure that cache clearing messages within the same process are processed sequentially, Kafka's partitioning mechanism is utilized. When producing messages, messages from the same process are sent to the same partition based on the process ID as the partition key. Since Kafka guarantees the order of messages within a partition, sequential processing of cache clearing messages is achieved.

[0046] Redundancy Removal Command: Set up a scheduled task to periodically scan the cache. Check for any residual cached data after message processing failures; if any uncleared cache is found, perform a removal operation. The execution frequency of the scheduled task can be adjusted according to actual needs, such as once per hour.

[0047] The data query management method based on dynamic caching optimization in the embodiments of the present invention has been described above. The data query management device based on dynamic caching optimization in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 2 The data query management device based on dynamic caching optimization in this embodiment of the invention includes, for the above embodiments: The dynamic timeliness adjustment module 201 is used to build a dynamic timeliness adjustment engine, which automatically adjusts the cache time based on the frequency of data changes; and performs real-time monitoring through database triggers and log analysis. The user behavior prediction module 202 is used to obtain historical query records, predict the types and times of queries that users may initiate in the future through machine learning models, load the predicted query type data into the cache before predicting the query time, dynamically mark hot data according to access frequency, prioritize the allocation of cache resources for hot data and extend the validity period. Multi-level cache architecture building module 203 is used to build a multi-level cache architecture based on local cache and distributed cache, and to obtain data using a cache degradation strategy; The message processing mechanism establishment module 204 is used to introduce an enhanced Kafka message processing mechanism.

[0048] above Figure 2 The data query management device based on dynamic caching optimization in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The electronic device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0049] Figure 3This is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present invention. The electronic device 700 can vary significantly due to differences in configuration or performance. It may include one or more processors 710 (e.g., one or more processors) and a memory 720, and one or more storage media 730 (e.g., one or more storage devices, including RAM, FLASH, etc.) for storing application programs 733 or data 732. The memory 720 and storage media 730 can be temporary or persistent storage. The program stored in the storage media 730 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the electronic device 700. Furthermore, the processor 710 may be configured to communicate with the storage media 730 and execute the series of instruction operations in the storage media 730 on the electronic device 700.

[0050] The electronic device 700 may also include one or more power supplies 740, one or more input / output interfaces 750, and / or one or more operating systems 731, such as FreeRTOS, Android, etc. Those skilled in the art will understand that... Figure 3 The illustrated electronic device structure does not constitute a limitation on electronic devices and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0051] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a data query management method based on dynamic caching optimization.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0053] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, mobile device, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data query management method based on dynamic caching optimization, characterized in that, The data query management method based on dynamic caching optimization includes: Build a dynamic timeliness adjustment engine to automatically adjust cache time based on data change frequency; perform real-time monitoring through database triggers and log analysis; Obtain historical query records, use machine learning models to predict the types and times of queries that users may initiate in the future, load the predicted query type data into the cache before predicting the query time, dynamically mark hot data according to access frequency, prioritize the allocation of cache resources for hot data and extend its validity period; Build a multi-level caching architecture based on local and distributed caching, and use a cache degradation strategy to retrieve data; An enhanced Kafka message processing mechanism was introduced.

2. The data query management method based on dynamic caching optimization according to claim 1, characterized in that, The construction of a dynamic timeliness adjustment engine, which automatically adjusts the cache time based on the frequency of data changes, includes: Obtain the number of times data is updated per unit of time, and divide the data into high-frequency updates, medium-frequency updates, and low-frequency updates; When the data is determined to be frequently updated, the cache time is shortened to 10 minutes; If the data is determined to be low-frequency updated data, the cache time will be extended to 24 hours. When the data is determined to be medium-frequency updated data, the cache time is kept at 1 hour.

3. The data query management method based on dynamic caching optimization according to claim 1, characterized in that, The real-time monitoring through database triggers and log analysis includes: Create a trigger on the table where data update operations occur. When data is inserted, updated, or deleted, the trigger immediately captures the change time and sends a change notification. Obtain log information, analyze the log information, extract the time information of data insertion, update or deletion operations in the log information, and send change notifications.

4. The data query management method based on dynamic caching optimization according to claim 1, characterized in that, The process of obtaining historical query records and using machine learning models to predict the types and times of future queries that users may initiate includes: A user behavior analysis platform is built to collect users' historical query records, which include query time information, query type information, and query condition information. A user behavior prediction model is constructed using a long short-term memory network. The format of historical query records is converted, and the user behavior prediction model is trained using the user's historical query records; Using historical query records as input, predict the types and times of queries that users may initiate in the near future.

5. The data query management method based on dynamic caching optimization according to claim 4, characterized in that, Loading the predicted query type data into the cache before predicting the query time includes: Based on the query type and query time predicted by the user behavior prediction model, a preloading task is started before the predicted query time. The preloading task queries relevant data from the database and loads it into the cache; The method of dynamically marking hot data based on access frequency, prioritizing the allocation of caching resources for hot data and extending its validity period includes: Establish a hotspot data identification module to statistically analyze the access frequency of data in real time; The sliding window algorithm is used to calculate the number of times each data is accessed per unit time; when the number of accesses exceeds a set threshold, the data is marked as hot data. For frequently accessed data, prioritize allocating cache resources to extend cache validity.

6. The data query management method based on dynamic caching optimization according to claim 1, characterized in that, The construction of a multi-level caching architecture based on local and distributed caching, and the use of a cache degradation strategy to retrieve data, includes: Build a local cache, using Redis as the in-memory database; build a distributed cache using a consistent hashing algorithm, using Memcached or RedisCluster as the distributed cache solution, and construct a multi-level cache architecture based on local and distributed caches. A cache degradation strategy is employed to retrieve data. Specifically, when the local cache expires, the system automatically retrieves data from the distributed cache. If the required data is also not found in the distributed cache, a further query is performed on the database, and the query results are sequentially returned to both the distributed and local caches, while simultaneously updating the caches.

7. The data query management method based on dynamic caching optimization according to claim 1, characterized in that, The introduced enhanced Kafka message processing mechanism includes: An ACK mechanism is introduced on the Kafka consumer side. After successfully processing a message, the consumer sends an acknowledgment message to the Kafka cluster. If an exception occurs or the consumer fails to send an acknowledgment during message processing, Kafka puts the message back into the retry queue. To ensure that cache clearing messages for the same process are processed in order, Kafka's partitioning mechanism is used; when producing messages, the process ID is used as the partition key to send messages for the same process to the same partition. Set up a scheduled task to periodically scan the cache; check for any residual cache data after message processing failures, and if any uncleared cache is found, perform a clearing operation.

8. A data query management device based on dynamic caching optimization, used in the data query management method based on dynamic caching optimization as described in any one of claims 1-7, characterized in that, include: The dynamic timeliness adjustment module is used to build a dynamic timeliness adjustment engine, which automatically adjusts the cache time based on the frequency of data changes; Real-time monitoring is achieved through database triggers and log analysis; The user behavior prediction module is used to obtain historical query records, predict the types and times of queries that users may initiate in the future through machine learning models, load the predicted query type data into the cache before predicting the query time, dynamically mark hot data according to access frequency, prioritize the allocation of cache resources for hot data and extend the validity period. The multi-level caching architecture building module is used to build a multi-level caching architecture based on local caching and distributed caching, and to retrieve data using a cache degradation strategy. The message processing mechanism establishment module is used to introduce the enhanced Kafka message processing mechanism.

9. An electronic device comprising a memory and at least one processor, wherein the memory stores instructions; characterized in that, The at least one processor invokes the instructions in the memory to cause the electronic device to perform the steps of the data query management method based on dynamic cache optimization as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the data query management method based on dynamic cache optimization as described in any one of claims 1-7.

Images