Real-time data caching method and system for real-time database
By preloading commonly used data in real-time database, generating data change event messages, implementing cache partitions and sharding, dynamically adjusting cache policies, the problem of insufficient query delay and response speed during high concurrent access and large data processing in real-time databases is solved. , achieving efficient performance and data consistency.
Patent Information
- Application Number
- CN202410839523.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-06-26
AI Technical Summary
The existing real-time database data caching methods have problems of query latency and insufficient response speed during high concurrent access and large data volume processing, and it is difficult to maintain efficient performance and data consistency.
Adopt real-time data caching method, including starting the cache server to preload common data, generating data change event messages, updating cache server data, implementing cache partitions and sharding, and dynamically adjusting cache policies to ensure consistency between cached data and database data and High efficiency performance.
By accurately judging common and extremely useful data, optimize the data query process, reduce cache miss rate, improve the system's initial query efficiency, reduce system load, ensure the timeliness and consistency of cached data, and improve query performance and response speed.
Smart Images

Figure CN118964414A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer data storage, and in particular to a real-time data caching method and system for a real-time database. Background Art
[0002] With the development of information technology and the rapid growth of data volume, real-time databases have become particularly important in processing large-scale, high-frequency data queries and updates. However, traditional real-time database systems often face problems of query delays and insufficient response speed when processing high concurrent access and large amounts of data. In order to solve these problems, data caching technology is widely used in real-time databases to reduce the frequency of direct access to the database, thereby improving the overall performance of the system.
[0003] Existing caching methods mainly rely on simple caching strategies, such as Least Recently Used (LRU) and Least Frequently Used (LFU), which can be effective in some cases, but are often insufficient when dealing with complex real-time data changes and high-frequency data access. In addition, how to maintain consistency between cached data and database data, and how to dynamically adjust caching strategies to adapt to changing data access patterns are also challenges faced by existing technologies. Summary of the invention
[0004] In view of the above-mentioned problems, the present invention is proposed.
[0005] Therefore, the technical problem solved by the present invention is: the existing real-time database data caching method has the problems of query delay and insufficient response speed, and how to maintain high efficiency and data consistency during high concurrent access and large data volume processing.
[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a real-time data caching method of a real-time database, comprising:
[0007] Start the cache server, preload commonly used data, initiate a data query request, and check whether the data to be queried exists in the cache server;
[0008] When the real-time database data changes, a data change event message is generated and sent to the message queue;
[0009] Monitor data change events in the message queue and update the data in the cache server according to the event type;
[0010] Determine the cache update strategy, keep the cache data consistent with the database data, implement cache partitioning and sharding, and dynamically adjust the cache strategy.
[0011] As a preferred solution of the real-time data caching method of the real-time database of the present invention, wherein: the cache server includes a first cache server and a second cache server;
[0012] The preloading of common data includes collecting access logs of each data item and counting the access frequency f i and creation time t i ,Build a common data evaluation model to determine the data type, expressed as,
[0013]
[0014]
[0015]
[0016]
[0017]
[0018]
[0019]
[0020] Among them, i represents the i-th data item, P i represents the logistic regression classification probability of the i-th data item, F i represents the result of the access frequency ratio of the i-th data item combined with the classification probability, C i represents the result of the fusion probability and frequency of the i-th data item, S i represents the inverse proportional function value of the survival time of the ith data item, D i represents the result of the time decay function of the ith data item combined with the survival time, E i represents the comprehensive time and frequency fusion result of the i-th data item, A i represents the comprehensive judgment and cumulative effect result of the i-th data item, β0 and β1 represent the regression coefficients of the logistic regression model, and x i represents the independent variable of the ith data item, f i represents the access frequency of the i-th data item, represents the sum of the access frequencies of all data items, t i represents the creation time of the i-th data item, T represents the survival time length of the data item, D0 represents the initial value of the time decay function, λ represents the time decay constant, γ0 and γ1 represent the parameters of the cumulative effect function, δ0 and δ1 represent the parameters of the comprehensive judgment function, τ represents the time variable, k represents the discrete time point, and m represents the number of time points;
[0021] If C iIf it is greater than 0.4, it is considered as common data. i Less than or equal to 0.4, calculate E i , if E i If it is greater than 0.3, it is considered as common data. i Less than or equal to 0.3, calculate A i , if A i If it is greater than 0.2, it is considered as commonly used data, otherwise it is determined as uncommon data.
[0022] As a preferred solution of the real-time data caching method of the real-time database of the present invention, wherein: the first cache server stores preloaded commonly used data; the second cache server stores uncommonly used data;
[0023] The initiating of the data query request includes receiving a unique identifier of the data, searching for the data in the first cache server using the unique identifier, and if the data exists in the first cache server and has not expired, determining it as a cache hit, and directly returning the data;
[0024] If the data does not exist in the first cache server or has expired, continue to check the second cache server;
[0025] If the data exists in the second cache server and has not expired, it is determined as a cache hit, the data is directly returned, and the data is updated to the first cache server;
[0026] If the data exists in the second cache server but has expired or does not exist, it is determined as a cache miss, and the data is obtained from the real-time database; the obtained data is written to the cache server.
[0027] As a preferred solution of the real-time data caching method of the real-time database described in the present invention, the method comprises: generating a data change event message and sending it to a message queue comprises: when a cache miss is detected, using a unique identifier to query and obtain required data from the real-time database; temporarily storing the data obtained from the real-time database in a buffer zone, and setting an expiration time for the data in the buffer zone; treating the operation of writing data into the buffer zone as a data change, generating an event message including a unique identifier of the data item, a change type, a timestamp and data content; and sending the generated data change event message to the message queue.
[0028] As a preferred solution of the real-time data caching method of the real-time database described in the present invention, wherein: the updating of data in the cache server according to the event type includes monitoring the message queue, receiving and parsing the data change event message, extracting the unique identifier, change type, timestamp and data content of the data item; judging the importance and access frequency of the data according to historical access data or real-time statistical data; determining the storage location of the data according to the importance and access frequency of the data, and setting an expiration time.
[0029] As a preferred solution of the real-time data caching method of the real-time database of the present invention, wherein: the cache update strategy determined includes a write-through strategy and a write-back strategy;
[0030] The write-through strategy is to write data to the real-time database immediately when the data is written to the cache server. When the write-through strategy is selected, each time the data is written to the cache server, it is also written to the real-time database simultaneously.
[0031] The write-back strategy is to write the data to the cache server first, and then write it to the real-time database in batches after reaching a predetermined time interval or a cache data volume threshold. When the write-back strategy is selected, the data is temporarily stored in the cache server after being written to the cache server, and is written to the real-time database in batches after waiting for a predetermined time interval or reaching a cache data volume threshold.
[0032] As a preferred solution of the real-time data caching method of the real-time database of the present invention, wherein: the implementation of cache partitioning and sharding includes analyzing the data characteristics in the cache server, selecting a range partitioning strategy, and distributing the data to different cache nodes and shards;
[0033] The dynamic adjustment of the cache strategy includes collecting the cache hit rate, access frequency and data expiration rate for each partition and shard, determining each partition and shard as a high-frequency access or low-frequency access, and setting a survival time accordingly.
[0034] Another object of the present invention is to provide a real-time data cache system for a real-time database, which can solve the performance bottleneck and data consistency problems in the existing real-time database system by constructing the real-time data cache system.
[0035] To solve the above technical problems, the present invention provides the following technical solutions: a real-time data caching system for a real-time database, comprising: a data query module, a data transmission module, a data update module and a data cache module; the data query module is used to start a cache server, preload commonly used data, initiate a data query request, and check whether there is data to be queried in the cache server; the data transmission module is used to generate a data change event message when the real-time database data changes, and send it to a message queue; the data update module is used to monitor the data change events in the message queue, and update the data in the cache server according to the event type; the data cache module is used to determine the cache update strategy, keep the cache data consistent with the database data, implement cache partitioning and sharding, and dynamically adjust the cache strategy.
[0036] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the real-time data caching method of the real-time database when executing the computer program.
[0037] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the real-time data caching method of the real-time database.
[0038] Beneficial effects of the present invention: The real-time data caching method for a real-time database provided by the present invention improves the query performance and response speed of the real-time database by preloading commonly used data, optimizing data query processes, processing data change events in real time, dynamically adjusting cache strategies, and adopting a flexible cache update mechanism.
[0039] By accurately determining commonly used data and uncommon data, the cache miss rate during cold start is reduced, and the initial query efficiency of the system is improved; data is searched in the cache first, reducing direct access to the real-time database and reducing system load; cache data is updated in real time to ensure the consistency and timeliness of cache data and database data; through cache partitioning and sharding management, the survival time of high-frequency and low-frequency access data is optimized to improve cache hit rate and data update efficiency; two cache update schemes, write-through strategy and write-back strategy, are provided to achieve real-time synchronous or asynchronous batch update of data, reduce database write pressure, and further improve system performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0041] Figure 1 An overall flow chart of a real-time data caching method for a real-time database provided by an embodiment of the present invention.
[0042] Figure 2 The overall structure diagram of a real-time data cache system of a real-time database provided by the second embodiment of the present invention DETAILED DESCRIPTION
[0043] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0045] Example 1
[0046] Reference Figure 1 , as an embodiment of the present invention, provides a real-time data caching method for a real-time database, comprising:
[0047] Start the cache server, preload commonly used data, initiate a data query request, and check whether the data to be queried exists in the cache server;
[0048] When the real-time database data changes, a data change event message is generated and sent to the message queue;
[0049] Monitor data change events in the message queue and update the data in the cache server according to the event type;
[0050] Determine the cache update strategy, keep the cache data consistent with the database data, implement cache partitioning and sharding, and dynamically adjust the cache strategy.
[0051] The cache server includes a first cache server and a second cache server;
[0052] The preloading of common data includes collecting access logs of each data item and counting the access frequency f i and creation time t i ,Build a common data evaluation model to determine the data type, expressed as,
[0053]
[0054]
[0055]
[0056]
[0057]
[0058]
[0059]
[0060] Among them, i represents the i-th data item, P i represents the logistic regression classification probability of the i-th data item, F i represents the result of the access frequency ratio of the i-th data item combined with the classification probability, C i represents the result of the fusion probability and frequency of the i-th data item, S i represents the inverse proportional function value of the survival time of the ith data item, D i represents the result of the time decay function of the ith data item combined with the survival time, E i represents the comprehensive time and frequency fusion result of the i-th data item, A i represents the comprehensive judgment and cumulative effect result of the i-th data item, β0 and β1 represent the regression coefficients of the logistic regression model, and x i represents the independent variable of the ith data item, f i represents the access frequency of the i-th data item, represents the sum of the access frequencies of all data items, t i represents the creation time of the i-th data item, T represents the survival time length of the data item, D0 represents the initial value of the time decay function, λ represents the time decay constant, γ0 and γ1 represent the parameters of the cumulative effect function, δ0 and δ1 represent the parameters of the comprehensive judgment function, τ represents the time variable, k represents the discrete time point, and m represents the number of time points;
[0061] If C i If it is greater than 0.4, it is considered as common data. i Less than or equal to 0.4, calculate E i , if E i If it is greater than 0.3, it is considered as common data. i Less than or equal to 0.3, calculate A i , if A i If it is greater than 0.2, it is considered as commonly used data, otherwise it is determined as uncommon data.
[0062] The first cache server stores preloaded frequently used data; the second cache server stores infrequently used data;
[0063] The initiating of the data query request includes receiving a unique identifier of the data, searching for the data in the first cache server using the unique identifier, and if the data exists in the first cache server and has not expired, determining it as a cache hit, and directly returning the data;
[0064] If the data does not exist in the first cache server or has expired, continue to check the second cache server;
[0065] If the data exists in the second cache server and has not expired, it is determined as a cache hit, the data is directly returned, and the data is updated to the first cache server;
[0066] If the data exists in the second cache server but has expired or does not exist, it is determined as a cache miss, and the data is obtained from the real-time database; the obtained data is written to the cache server.
[0067] The method of generating a data change event message and sending it to a message queue includes, when a cache miss is detected, using a unique identifier to query and obtain required data from a real-time database; temporarily storing the data obtained from the real-time database in a buffer and setting an expiration time for the data in the buffer; treating the operation of writing data to the buffer as a data change, generating an event message including a data item unique identifier, a change type, a timestamp and data content; and sending the generated data change event message to a message queue.
[0068] The updating of data in the cache server according to the event type includes monitoring the message queue, receiving and parsing the data change event message, extracting the unique identifier, change type, timestamp and data content of the data item; judging the importance and access frequency of the data according to the historical access data or real-time statistical data; determining the storage location of the data according to the importance and access frequency of the data, and setting an expiration time.
[0069] The cache update strategy includes a write-through strategy and a write-back strategy;
[0070] The determination rule of cache update strategy is expressed as,
[0071] S=αD+βF+γE
[0072]
[0073]
[0074]
[0075]
[0076] Among them, S represents the comprehensive evaluation score, D represents the data consistency evaluation function, F represents the write operation frequency evaluation function, E represents the delay requirement evaluation function, C represents the data consistency requirement coefficient, ranging from 0 to 1, W represents the standardized write operation frequency, ranging from 0 to 1, L represents the delay requirement coefficient, ranging from 0 to 1, α, β, γ represent weight coefficients, corresponding to the importance of data consistency requirements, write operation frequency and delay requirements, respectively, and satisfying α+β+γ=1, T represents the scoring threshold, μ, λ, σ represent adjustment coefficients, which are used to control the relationship between different parameters, and n represents the number of discrete time points.
[0077] The write-through strategy is to write data to the real-time database immediately when the data is written to the cache server. When the write-through strategy is selected, each time the data is written to the cache server, it is also written to the real-time database simultaneously.
[0078] The write-back strategy is to write the data to the cache server first, and then write it to the real-time database in batches after reaching a predetermined time interval or a cache data volume threshold. When the write-back strategy is selected, the data is temporarily stored in the cache server after being written to the cache server, and is written to the real-time database in batches after waiting for a predetermined time interval or reaching a cache data volume threshold.
[0079] The system performs consistency checks regularly, sets scheduled tasks, and regularly compares data in the cache server and real-time database. If inconsistent data is found, it will be updated synchronously.
[0080] The implementation of cache partitioning and sharding includes analyzing the data characteristics in the cache server, selecting a range partitioning strategy, distributing the data to different cache nodes and shards, each partition and shard storing a specific range of data, and managing the data according to a preset strategy;
[0081] The dynamic adjustment of the cache strategy includes collecting the cache hit rate, access frequency and data expiration rate for each partition and shard, determining each partition and shard as a high-frequency access or low-frequency access, and setting a survival time accordingly.
[0082] Example 2
[0083] Reference Figure 2 , as an embodiment of the present invention, provides a real-time data caching system of a real-time database, comprising:
[0084] Data query module, data transmission module, data update module and data cache module;
[0085] The data query module is used to start the cache server, preload common data, initiate a data query request, and check whether the data to be queried exists in the cache server;
[0086] The data transmission module is used to generate a data change event message when the real-time database data changes, and send it to the message queue;
[0087] The data update module is used to monitor data change events in the message queue and update the data in the cache server according to the event type;
[0088] The data cache module is used to determine the cache update strategy, keep the cache data consistent with the database data, implement cache partitioning and sharding, and dynamically adjust the cache strategy.
[0089] Example 3
[0090] An embodiment of the present invention is different from the first two embodiments in that:
[0091] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0092] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0093] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0094] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0095] Example 4
[0096] As an embodiment of the present invention, a real-time data caching method of a real-time database is provided. In order to verify the beneficial effects of the present invention, a scientific demonstration is carried out through simulation experiments.
[0097] The experiment was conducted under the same hardware and data environment, using the method of the present invention and the traditional method respectively, and adopting the LRU cache strategy for data query and cache management. The experiment focused on the performance of the system when processing high concurrent access and large amounts of data, including indicators such as query response time, cache hit rate, and system load. The experimental environment is shown in Table 1.
[0098] Table 1 Experimental environment configuration table
[0099]
[0100] Deploy a MySQL database and import 10 million test data. Configure a Redis cache server. Use Apache JMeter to configure a load generator and simulate 1,000 concurrent users to perform data query operations.
[0101] The traditional method configures the Redis cache server to adopt the LRU strategy. Run the load generator and record the query response time, cache hit rate, and system load.
[0102] The method of the present invention configures the Redis cache server to adopt the method of the present invention, including preloading common data, dynamically adjusting cache strategy, cache partitioning and sharding, and asynchronous update mechanism. Run the load generator to record the query response time, cache hit rate and system load.
[0103] The query response time, cache hit rate and system load of the two methods are compared. The experimental results are shown in Table 2.
[0104] Table 2 Comparison of experimental results
[0105] index Traditional method (LRU) Method of the present invention Average query response time (ms) 150 80 Cache hit rate (%) 70 95 System CPU usage (%) 85 60 System memory usage (%) 80 65
[0106] The traditional LRU cache strategy frequently eliminates the least recently used data when facing high concurrent access, which may cause frequently accessed data to be constantly removed from the cache, resulting in frequent cache misses. In this case, the system needs to frequently access the database, which increases the query response time. The LRU strategy only considers the most recent access time of the data, but fails to manage the cache in combination with the access frequency and importance of the data. If frequently accessed data has not been accessed for a period of time, it may be eliminated by the LRU strategy, thereby reducing the cache hit rate. When the cache hit rate is low, the traditional method increases direct access to the database, resulting in increased consumption of system CPU and memory resources. Frequent database access not only increases the system load, but may also cause database performance bottlenecks.
[0107] The method of the present invention ensures a high cache hit rate at the beginning of system startup by preloading commonly used data, thereby reducing the number of cache misses during cold start. By dynamically adjusting the cache strategy and adjusting the time to live (TTL) according to the access frequency and importance of the data, frequently accessed data will be stored in the cache for a longer time, reducing the probability of cache misses, thereby significantly reducing the query response time.
[0108] The method of the present invention combines multiple factors such as logistic regression model and access frequency to construct a common data evaluation model, which can more accurately determine which data should be stored in the cache. Through cache partitioning and sharding strategies, data is distributed to different cache nodes and shards according to specific rules, so that high-frequency access data can be more effectively managed and stored, significantly improving the cache hit rate. A cache update mechanism combining write-through strategy and write-back strategy is adopted, and synchronous or asynchronous update mode is flexibly selected, which reduces frequent write operations to the database and reduces the CPU and memory load of the system. Through the asynchronous update mechanism, the write operation of cache data is processed in batches, which effectively reduces the write pressure of the database and further improves the overall performance and resource utilization efficiency of the system.
[0109] The present invention is more intelligent and efficient in cache management by preloading commonly used data and dynamically adjusting the cache strategy, so that high-frequency access data can be retained first, avoiding the defect of the traditional LRU strategy of frequently eliminating important data. By generating data change event messages and processing these events in real time, the consistency of cache data and database data is ensured, and the problem of traditional methods having difficulty in data synchronization under high concurrency is solved. The method of the present invention uses scientific cache partitioning and sharding strategies to reasonably allocate cache resources, further optimize the performance and scalability of the system, and ensure that the system can still maintain efficient operation under large data volumes and high concurrent access conditions.
[0110] The present invention overcomes the shortcomings of the traditional LRU cache strategy in processing high concurrent access and large data volumes, significantly improves the query performance and response speed of the real-time database, and ensures efficient use of system resources and data consistency.
[0111] Example 5
[0112] As an embodiment of the present invention, a real-time data caching method for a real-time database is provided. In order to verify the beneficial effects achieved by the preferred threshold of the present invention, a scientific demonstration is carried out through simulation experiments.
[0113] Collect the access logs of each data item in the experimental environment and count the access frequency f i and creation time t i ,Build a common data evaluation model to determine the data type, expressed as,
[0114]
[0115]
[0116]
[0117]
[0118]
[0119]
[0120]
[0121] Among them, i represents the i-th data item, P i represents the logistic regression classification probability of the i-th data item, F i represents the result of the access frequency ratio of the i-th data item combined with the classification probability, C i represents the result of the fusion probability and frequency of the i-th data item, S i represents the inverse proportional function value of the survival time of the ith data item, Di represents the result of the time decay function of the ith data item combined with the survival time, E i represents the comprehensive time and frequency fusion result of the i-th data item, A i represents the comprehensive judgment and cumulative effect result of the i-th data item, β0 and β1 represent the regression coefficients of the logistic regression model, and x i represents the independent variable of the ith data item, f i represents the access frequency of the i-th data item, represents the sum of the access frequencies of all data items, t i represents the creation time of the i-th data item, T represents the survival time length of the data item, D0 represents the initial value of the time decay function, λ represents the time decay constant, γ0 and γ1 represent the parameters of the cumulative effect function, δ0 and δ1 represent the parameters of the comprehensive judgment function, τ represents the time variable, k represents the discrete time point, and m represents the number of time points;
[0122] The initial thresholds are set to C=0.5, E=0.4, and A=0.3.
[0123] The simulation environment is as follows: cache server capacity: 1000 data items; real-time database capacity: 10000 data items.
[0124] Access frequency: randomly generated, the access frequency of each data item is between 1 and 1()() times / second.
[0125] Data creation time: randomly generated, the creation time of each data item is between 0 and 1000 seconds.
[0126] Generate an access log containing 10,000 data items, each of which contains information such as access frequency and creation time.
[0127] The simulated cache system includes a first cache server and a second cache server. Parameters such as storage capacity and expiration time of the cache server are set.
[0128] Step 1: Evaluate based on initial parameter thresholds
[0129] All data items are evaluated using an initial threshold, and the data items are classified into commonly used data and uncommonly used data, which are stored in a first cache server and a second cache server, respectively.
[0130] Step 2: Simulate data access process
[0131] The system data access process within a period of time (set as 24 hours in the present invention) is simulated, and performance indicators such as cache hit rate, cache miss number, and data access delay are recorded.
[0132] Adjust the threshold parameters and try different threshold combinations, including:
[0133] Combination 1: C = 0.4, E = 0.3, A = 0.2
[0134] Combination 2: C = 0.3, E = 0.2, A = 0.1
[0135] Combination 3: C = 0.5, E = 0.4, A = 0.3
[0136] Combination 4: C = 0.6, E = 0.5, A = 0.4
[0137] Combination 5: C = 0.35, E = 0.25, A = 0.15
[0138] Repeat steps 1 and 2 for each combination, and record the performance indicators after each adjustment. The experimental results are shown in Table 3.
[0139] Table 3 Threshold experimental data comparison table
[0140] Threshold combination Cache hit rate Cache miss count Average data access latency (ms) Combination 1 85% 1500 200 Combination 2 95% 500 100 Combination 3 90% 1000 150 Combination 4 80% 1800 250 Combination 5 92% 700 120
[0141] Combination 1 performed best. This combination had the highest cache hit rate of 95%, the lowest cache misses of 500, and the lowest average data access latency of 100 milliseconds.
[0142] Although the cache hit rate of combination 2 has improved to 90%, the number of cache misses is still high, at 1,000 times, and the data access latency is moderate, at 150 milliseconds.
[0143] Under the initial threshold combination of combination 3, the cache hit rate is low, which is 85%, the number of cache misses is high, which is 1500 times, and the data access latency is high, which is 200 milliseconds.
[0144] The performance of combination 4 degrades further, with the cache hit rate decreasing to 80%, the number of cache misses increasing to 1800, and the data access latency increasing to 250 milliseconds.
[0145] Combination 5 performs better, but still not as well as combination 1, with a cache hit rate of 92%, 700 cache misses, and a data access latency of 120 milliseconds.
[0146] Through experimental data comparison and analysis, the combination of thresholds C = 0.4, E = 0.3, and A = 0.2 performs best in improving cache hit rate, reducing cache misses, and reducing data access latency. The experimental results verify the effectiveness and rationality of this combination in cache server performance optimization.
[0147] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A real-time data caching method for a real-time database, characterized in that: include: Start the cache server, preload commonly used data, initiate a data query request, and check whether the data to be queried exists in the cache server; When the real-time database data changes, a data change event message is generated and sent to the message queue; Monitor data change events in the message queue and update the data in the cache server according to the event type; Determine the cache update strategy, keep the cache data consistent with the database data, implement cache partitioning and sharding, and dynamically adjust the cache strategy.
2. The real-time data caching method of a real-time database according to claim 1, characterized in that: The cache server includes a first cache server and a second cache server; The preloading of common data includes collecting access logs of each data item and counting the access frequency f i and creation time t i ,Build a common data evaluation model to determine the data type, expressed as, Among them, i represents the i-th data item, P i represents the logistic regression classification probability of the i-th data item, F i represents the result of the access frequency ratio of the i-th data item combined with the classification probability, C i represents the result of the fusion probability and frequency of the i-th data item, S i represents the inverse proportional function value of the survival time of the ith data item, D i represents the result of the time decay function of the ith data item combined with the survival time, E i represents the comprehensive time and frequency fusion result of the i-th data item, A i represents the comprehensive judgment and cumulative effect result of the i-th data item, β0 and β1 represent the regression coefficients of the logistic regression model, and x i represents the independent variable of the ith data item, f i represents the access frequency of the i-th data item, represents the sum of the access frequencies of all data items, t i represents the creation time of the i-th data item, T represents the survival time length of the data item, D0 represents the initial value of the time decay function, λ represents the time decay constant, γ0 and γ1 represent the parameters of the cumulative effect function, δ0 and δ1 represent the parameters of the comprehensive judgment function, τ represents the time variable, k represents the discrete time point, and m represents the number of time points; If C i If it is greater than 0.4, it is considered as common data. i Less than or equal to 0.4, calculate E i , if E i If it is greater than 0.3, it is considered as common data. i Less than or equal to 0.3, calculate A i , if A i If it is greater than 0.2, it is considered as commonly used data, otherwise it is determined as uncommon data.
3. The real-time data caching method of the real-time database according to claim 2, characterized in that: The first cache server stores preloaded frequently used data; the second cache server stores infrequently used data; The initiating of the data query request includes receiving a unique identifier of the data, searching for the data in the first cache server using the unique identifier, and if the data exists in the first cache server and has not expired, determining it as a cache hit, and directly returning the data; If the data does not exist in the first cache server or has expired, continue to check the second cache server; If the data exists in the second cache server and has not expired, it is determined as a cache hit, the data is directly returned, and the data is updated to the first cache server; If the data exists in the second cache server but has expired or does not exist, it is determined as a cache miss, and the data is obtained from the real-time database; the obtained data is written to the cache server.
4. The real-time data caching method of the real-time database according to claim 3, characterized in that: The method of generating a data change event message and sending it to a message queue includes, when a cache miss is detected, using a unique identifier to query and obtain required data from a real-time database; temporarily storing the data obtained from the real-time database in a buffer and setting an expiration time for the data in the buffer; treating the operation of writing data to the buffer as a data change, generating an event message including a data item unique identifier, a change type, a timestamp and data content; and sending the generated data change event message to a message queue.
5. The real-time data caching method of the real-time database according to claim 4, characterized in that: The updating of data in the cache server according to the event type includes monitoring the message queue, receiving and parsing the data change event message, extracting the unique identifier, change type, timestamp and data content of the data item; judging the importance and access frequency of the data according to the historical access data or real-time statistical data; determining the storage location of the data according to the importance and access frequency of the data, and setting an expiration time.
6. The real-time data caching method of a real-time database as claimed in claim 5, characterized in that: The cache update strategy includes a write-through strategy and a write-back strategy; The write-through strategy is to write data to the real-time database immediately when the data is written to the cache server. When the write-through strategy is selected, each time the data is written to the cache server, it is also written to the real-time database simultaneously. The write-back strategy is to write the data to the cache server first, and then write it to the real-time database in batches after reaching a predetermined time interval or a cache data volume threshold. When the write-back strategy is selected, the data is temporarily stored in the cache server after being written to the cache server, and is written to the real-time database in batches after waiting for a predetermined time interval or reaching a cache data volume threshold.
7. The real-time data caching method of a real-time database as claimed in claim 6, characterized in that: The implementation of cache partitioning and sharding includes analyzing the data characteristics in the cache server, selecting a partitioning strategy by range, and distributing the data to different cache nodes and shards; The dynamic adjustment of the cache strategy includes collecting the cache hit rate, access frequency and data expiration rate for each partition and shard, determining each partition and shard as a high-frequency access or low-frequency access, and setting a survival time accordingly.
8. A system using the real-time data caching method of a real-time database according to any one of claims 1 to 7, characterized in that: include: Data query module, data transmission module, data update module and data cache module; The data query module is used to start the cache server, preload common data, initiate a data query request, and check whether the data to be queried exists in the cache server; The data transmission module is used to generate a data change event message when the real-time database data changes, and send it to the message queue; The data update module is used to monitor data change events in the message queue and update the data in the cache server according to the event type; The data cache module is used to determine the cache update strategy, keep the cache data consistent with the database data, implement cache partitioning and sharding, and dynamically adjust the cache strategy.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the real-time data caching method of the real-time database according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the real-time data caching method of a real-time database according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Cache data management method and device
CN107291779A
Data caching processing method and device, computer device and storage medium
CN110795457A
Method for updating database hotspot data cache in real time
CN114138807A
Cache updating method and device and computer equipment
CN116775708A
Caching System for Eventually Consistent Services
US20190004969A1
Cited By
Dynamic medical data anonymization processing system and method based on message monitoring
CN119475438A
A dynamic medical data anonymization processing system and method based on message monitoring
CN119475438B
Strategy adjustment method and device and storage medium
CN119743394A
Data management method based on real-time data warehouse
CN120407565A
Tax form intelligent asynchronous processing and real-time query method based on dynamic partition
CN120492470A
