Method for accelerating database hotspot data information

Abstract

This invention relates to a method for accelerating the processing of hot data information in a database, comprising the following steps: collecting user-related feature data, including static and dynamic features; identifying hot data in the database based on global query logs; calculating the matching degree between user feature data and the hot data, and triggering preloading of the hot data to the user's local cache based on the matching degree result; managing the data in the local cache, including data updates and data eviction; and upon receiving a user query request, prioritizing matching data from the local cache and outputting it, and if the cache misses, initiating a query to the database. By combining user static and dynamic features to calculate the matching degree, indiscriminate preloading is avoided, thereby improving the cache hit rate and reducing server-side system resource consumption while improving query response speed.

Description

Technical Field

[0001] This invention relates to the field of data information technology, specifically to a method for accelerating the processing of hot data information in a database. Background Technology

[0002] With the rapid development of internet technology, user data query requests on e-commerce, news, and government platforms have exploded. The query efficiency of high-frequency, hot data directly affects user experience and system stability. Traditional database query methods follow a passive response model of "user request - database retrieval - data return." When a large number of users simultaneously initiate high-frequency queries, it can cause excessive load on the database server, leading to increased query response latency and even system lag.

[0003] To address these issues, existing technologies often employ a global caching strategy, which involves caching frequently queried data from the database to the server. However, this approach has significant drawbacks: First, global caching lacks user-specificity, resulting in low matching between cached data and actual user needs, leading to a low cache hit rate. Second, blindly preloading large amounts of data into the cache consumes excessive storage resources, and the synchronization cost during data updates is high. Third, it fails to incorporate personalized features such as user identity and historical behavior, making accurate preloading decisions impossible.

[0004] Therefore, there is an urgent need for an acceleration method that can accurately preload hot data based on user characteristics, so as to improve query response speed while reducing system resource consumption. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] This invention provides a method for accelerating the processing of hot data information in a database, which solves the problems of slow response to high-frequency database queries, low cache hit rate, and resource waste in the prior art.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for accelerating hot data information in a database, comprising the following steps:

[0009] Step 1: Collect user-related feature data, including static and dynamic features;

[0010] Step 2: Based on the global query log of the database, identify hot data in the database;

[0011] Step 3: Calculate the matching degree between user feature data and the hot data, and trigger the preloading of hot data to the user's local cache based on the matching degree result;

[0012] Step 4: Manage the data in the local cache, including data updating and data eviction;

[0013] Step 5: After receiving the user's query request, first match the data from the local cache and output it. If the cache is not hit, then initiate a query to the database.

[0014] Preferably, the static features include user identity-related information, and the dynamic features include user browsing history-related information. The user identity-related information includes at least one of user level, permission tags, and group affiliation; the user browsing history-related information includes at least one of historical query keywords, click records, data dwell time, collection behavior, and sharing behavior.

[0015] In a further preferred embodiment, the dimensions for identifying the hot data include at least one of data query frequency, query time concentration, and data correlation. By quantifying each dimension of the hot data, data that meets the preset standard can be identified as hot data.

[0016] In a further preferred embodiment, when calculating the matching degree between user feature data and the hot data, a matching degree quantification result is formed based on the similarity between the user's historical query keywords and hot data tags, the user's historical behavior on similar data, and the priority corresponding to the user's identity. When the matching degree quantification result reaches a preset triggering standard, a preloading operation of hot data to the local cache is performed.

[0017] In a further preferred embodiment, the data update includes synchronously updating the corresponding data in the local cache when the hot data content in the database changes; the data eviction is based on the degree of data hotness and the idle time of the cache, prioritizing the eviction of data with low hotness and long idle time.

[0018] In a further preferred approach, after initiating a query to the database, the keywords and data corresponding to this query are fed back to the user's relevant feature data collection and hotspot data identification, which are used to update the feature data and hotspot data identification results.

[0019] (III) Beneficial Effects

[0020] Compared with existing technologies, the present invention provides a method for accelerating the processing of hot data information in a database, which has the following beneficial effects:

[0021] In this invention, by combining user static and dynamic features to calculate the matching degree, indiscriminate preloading is avoided, which can improve the cache hit rate, thereby increasing query response speed while reducing server-side system resource consumption.

[0022] In this invention, by adopting an eviction strategy that combines popularity and time decay, high-value cached data can be prioritized for retention, reducing invalid storage and update costs, and lowering user-end system resource consumption. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a method for accelerating database hotspot data information according to the implementation plan. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see Figure 1 A method for accelerating the processing of hot data information in a database includes the following steps:

[0026] First, user-related feature data is collected, including static and dynamic features. Static features include user identity-related information, such as user level, permission tags, and group affiliation (at least one). Dynamic features include user browsing history-related information, such as historical query keywords, click records, data dwell time, favorites behavior, and sharing behavior (at least one). After collecting the user-related feature data, it is normalized to form a user feature vector.

[0027] Next, based on the global query logs of the database, hot data in the database is identified. For example, it can be quantified from three dimensions: data query frequency, query time concentration, and data correlation. The degree of hot data is confirmed based on the quantified data of each dimension, and data that meets preset standards can be identified as hot data. The specific calculation formula is as follows:

[0028] ;

[0029] In the formula, For data The popularity score is used to measure the popularity of data. The higher the value, the more likely the data will be included in the hot data set. This is the frequency weighting coefficient, with a value range of (0,1), used to adjust the proportion of the influence of query frequency on the popularity value; For data The relative query frequency, specifically the data The ratio of the number of queries to the global average number of queries; The time decay factor is calculated using the following formula: ,in, The attenuation coefficient has a value range of (0, 0.1]. For current time and data The time difference of the last query, in hours. This is used to reduce the impact of long-term query behavior on the popularity value, reflecting the principle of prioritizing recent high-frequency queries; This is the correlation weighting coefficient, with a value range of (0,1), used to adjust the proportion of the influence of data correlation on the popularity value; For data The correlation popularity, specifically with data The average popularity value of other data in the linked query. When the data popularity value... When the data is greater than or equal to a preset popularity threshold (e.g., initially set to 50), the data is identified as hot data and a hot data set is formed.

[0030] Then, the matching degree between user feature data and the hot data is calculated, and the preloading of hot data to the user's local cache is triggered based on the matching degree result. When calculating the matching degree between user feature data and the hot data, a quantitative result of the matching degree can be formed based on the similarity between the user's historical query keywords and hot data tags, the user's historical behavior on similar data, and the priority corresponding to the user's identity. The specific calculation formula is as follows:

[0031] ;

[0032] In the formula, For users With data The matching degree, with a value range of (0,100], the higher the value, the higher the preloading priority; α is the keyword similarity weight, with a value range of (0,1); For users' historical query keywords and data The cosine similarity of the tag keywords ranges from [0,1]; β is the user behavior weight, which ranges from (0,1). Weighted values ​​are assigned to user behavior, specifically the weighted sum of a user's historical clicks, dwell times, and favorites on similar data. ,in The weighting coefficients for different behaviors satisfy... , γ is the normalized result of the number of user actions, with a value range of [0,1]; γ is the user identity weight, with a value range of (0,1); This is a user priority coefficient, determined based on user identity. For example, a VIP user might have a value of 1, while a regular user might have a value of 0.5. It only applies when the match is satisfactory. When the threshold value is greater than or equal to the preset threshold, perform a preloading operation on hot data to the user's local cache and generate a list of preloaded data.

[0033] Next, the data in the local cache is managed, including data updates and data eviction. During data updates, when hot data content in the database changes, the corresponding data in the local cache is updated synchronously. During data eviction, a comprehensive judgment logic based on the data's hotness and the cache's idle time is used, with the specific calculation formula as follows:

[0034] ;

[0035] In the formula, For data The cache retention priority is set, with a value range of (0, 100]. The lower the value, the higher the priority for eviction. For data The cache idle time is expressed in hours; m is the idle decay coefficient, ranging from (0, 0.05], used to control the rate at which idle time affects retention priority. When the cache capacity reaches the threshold, priority is given to eviction. The smallest data point.

[0036] Finally, when a user query request is received, data is first matched and output from the local cache. If the cache is not hit, a query is initiated to the database. The keywords and data corresponding to this query are fed back to the feature collection stage and the hot data identification stage to update the feature data and hot data identification results.

[0037] In the above process, cache hit rate and preloading cost can also be calculated. The formulas for calculating both are as follows:

[0038] ;

[0039] ;

[0040] In the formula, For cache hit rate, To cache the number of queries that are hit, This represents the total number of queries.

[0041] For preloading costs, This refers to the preloaded data list, which is the entire dataset preloaded into the local cache after the model makes its decision. For data Storage size (in KB, MB). Cost per unit of data storage (e.g., cost of storage resources per MB of data). For data The update frequency (e.g., the number of updates per day). Cost per unit of data update (such as computing power and time cost of data synchronization). This represents the storage cost per piece of data. The larger the data volume, the higher the unit price of storage, and the higher the storage cost. This represents the cost of updating a single piece of data. The more frequently the data is updated, the higher the cost per update.

[0042] The above formulas for calculating cache hit rate and preloading cost aim to achieve a dual-objective optimization: maximizing cache hit rate and minimizing the cost of invalid preloading. Cache hit rate is the core metric for measuring the effectiveness of the preloading strategy. A higher hit rate indicates that the preloaded data better matches user needs, a higher proportion of user queries retrieve data from the local cache, lower database query frequency, and a more significant acceleration effect. However, focusing solely on hit rate while ignoring cost may result in a large amount of useless data being preloaded, thus consuming resources. Preloading is not cost-free; its costs include storage costs (local storage space resources occupied by cached data) and update costs (computing power and time costs for synchronously updating cached data when hot database data is updated). "Invalid preloading" refers to cached data that has not been accessed by users for a long time after preloading and does not match user needs; this type of data wastes resources. The model minimizes the cost of invalid preloading while ensuring a high hit rate by setting a matching threshold and a cache eviction policy. Through processes such as "matching degree threshold filtering" (preloading only data with a matching degree exceeding the threshold) and "heat-time decay elimination" (eliminating low-value cached data), the ultimate goal is to reduce the calculation result of this formula.

[0043] In this embodiment, unlike the traditional "global indiscriminate caching" mode, the data preloading decision does not rely on the global popularity of data, but rather anchors to the user's static characteristics (identity, level, permissions) and dynamic characteristics (historical queries, clicks, favorites behavior) to construct a user demand profile. The preloaded data is hot data that highly matches the user profile, rather than all global hot data, fundamentally improving the fit between cached data and user needs. In addition, hot data is not passively retrieved after the user initiates a query, but actively identifies hot data in the database and loads highly matched hot data into the user's local cache in advance based on the user's characteristic matching degree. When the user initiates a query, the data can be read directly from the local cache, bypassing the time-consuming path of database query and achieving a "request-to-response" acceleration effect. The purpose of the database hot data information acceleration method provided in this embodiment is to solve the problems of high response latency and high server load in traditional database high-frequency query scenarios. Through the dual optimization of "precise preloading + local caching", it simultaneously achieves the dual acceleration value of "improved user query speed" and "reduced database pressure".

[0044] In the description of this invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0045] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for accelerating database hotspot data information, characterized in that, Includes the following steps: Step 1: Collect user-related feature data, including static and dynamic features; Step 2: Based on the global query log of the database, identify hot data in the database; Step 3: Calculate the matching degree between user feature data and the hot data, and trigger the preloading of hot data to the user's local cache based on the matching degree result; Step 4: Manage the data in the local cache, including data updating and data eviction; Step 5: After receiving the user's query request, first match the data from the local cache and output it. If the cache is not hit, then initiate a query to the database.

2. The method of claim 1, wherein: The static features include user identity-related information, and the dynamic features include user browsing history-related information.

3. The method of claim 2, wherein: The user identity-related information includes at least one of user level, permission tags, and group affiliation; the user browsing history-related information includes at least one of historical query keywords, click records, data dwell time, collection behavior, and sharing behavior.

4. The method of claim 1, wherein: The dimensions for identifying hot data include at least one of data query frequency, query time concentration, and data correlation. By quantifying each dimension of hot data, data that meets the preset standards can be identified as hot data.

5. The method of claim 4, wherein: When calculating the matching degree between user feature data and the hot data, a quantitative result of the matching degree is formed based on the similarity between the user's historical query keywords and the hot data tags, the user's historical behavior on similar data, and the priority corresponding to the user's identity.

6. The method of claim 5, wherein: When the matching degree quantification result reaches the preset trigger standard, the hot data is preloaded to the local cache.

7. The method of claim 1-6, wherein: The data update includes synchronously updating the corresponding data in the local cache when the hot data content in the database changes.

8. The method of claim 7, wherein: The data eviction process is based on the data's popularity and the length of time the cache has been idle, prioritizing the eviction of data with low popularity and long idle time.

9. The method of claim 1, wherein: After a query is sent to the database, the keywords and data corresponding to this query are fed back to the user's relevant feature data collection and hotspot data identification, which are used to update the feature data and hotspot data identification results.

Images