A multi-level data node storage index method and system

CN118626685BActive Publication Date: 2024-11-05HANGZHOU NEW WINDOWS INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411088530.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-05
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The traditional single-layer index structure is difficult to meet the needs of high efficiency and dynamic expansion in big data processing and storage scenarios, resulting in slow index updates, long query response time, and serious resource waste, which fails to effectively optimize the data access mode.

Method used

Using a multi-level data node storage index method, predict the query frequency of data nodes, calculate the score, divide the data nodes into memory and external memory, and use a hash table to establish a global index, dynamically adjust the data distribution, and optimize the memory and external memory. Resource utilization.

Benefits of technology

It improves search efficiency, reduces I/O operations, significantly improves system response speed, optimizes resource utilization, flexibly adapts to data changes, is suitable for large-scale data processing and storage scenarios, and improves the overall performance of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118626685B_ABST
    Figure CN118626685B_ABST
Patent Text Reader

Abstract

The present invention proposes a multi-level data node storage index method and system. The method includes: predicting the query frequency of data nodes in the future time period; calculating the score of the data node according to the predicted query frequency of the data node, the most recent access time and the total access frequency, dividing the data node into memory data nodes and external storage data nodes according to the score, and establishing a global index according to whether it is stored in the memory or the external storage. Through this method and the corresponding system, efficient data node management and search can be achieved, the resource utilization of the memory and external storage can be optimized, and the overall performance of the query system can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention proposes a multi-level data node storage index method and system, and relates to the field of data processing. Background Art

[0002] In modern scenarios of big data processing and storage, effective indexing strategies are essential to improve data access speed and reduce system load. With the substantial increase in data volume, traditional single-layer index structures often fail to meet the requirements of high efficiency and dynamic expansion. Therefore, multi-level data processing and storage indexing methods have begun to receive more and more attention, which aims to improve indexing performance and storage efficiency through structural optimization and intelligent data allocation. In a single-layer index structure, as the amount of data increases, the size and maintenance cost of the index itself also increase dramatically, which may make the index a bottleneck; for large-scale data sets, a single index structure may lead to slow index updates and long query response times, especially in environments with frequent updates; traditional indexing techniques may waste a lot of computing and storage resources on unnecessary data retrieval because they are not optimized for data access patterns. Summary of the invention

[0003] The present invention provides a multi-level data node storage index method and system to solve the above-mentioned problems:

[0004] The present invention proposes a multi-level data node storage index method, the method comprising:

[0005] Predict the query frequency of data nodes in the future time period;

[0006] The score of the data node is calculated based on the predicted query frequency, recent access time and total access frequency of the data node. The data nodes are divided into memory data nodes and external storage data nodes based on the score, and a global index is established based on whether the data is stored in memory or external storage.

[0007] Furthermore, a multi-level data node storage index method predicts the query frequency of the data node in the future time period, including:

[0008] Collecting historical records about queries on the data node, the historical records including a timestamp of the query, a query type, and an identifier of the data node, the query type including reading, updating, and deleting the data node;

[0009] Seasonal activities, special events and emergency events are added to the historical records. These information are marked as binary features. If there is a special event on a certain day, the feature is 1, otherwise it is 0;

[0010] Extracting date features from the timestamp of the query, the date features including month, day of the week and hour, creating new derived variables from the date features, the new derived variables including the interval from the last query time to the current query time;

[0011] constructing the historical records, seasonal activity information, special event information, emergency event information, date features and derived variables into a data set;

[0012] Handle missing values ​​and outliers in the data set, and standardize the data;

[0013] The data set is divided into a training set and a test set, with 80% of the data used as the training set and the remaining 20% ​​as the test set;

[0014] Train a random forest model on the training set and use cross-validation to optimize model parameters and avoid overfitting;

[0015] Generate a feature data set containing a future time period, the feature data set including date features, seasonality and event features, and historical features, the date feature is to generate the date of the future time period and extract the date features of the future time period, the seasonality and time features are to mark special events and seasonal activities in the future time period according to expectations, and the historical features are characterized by the number of queries for data in the past time period;

[0016] Use the trained random forest model to make predictions on a feature dataset containing future time periods;

[0017] Convert the query probabilities output by the random forest model into query frequencies.

[0018] Furthermore, a multi-level data node storage index method calculates the score of the data node according to the predicted query frequency, the most recent access time and the total access frequency of the data node, divides the data node into memory data nodes and external storage data nodes according to the score, and establishes a global index according to whether the data node is stored in memory or external storage, including:

[0019] The data node score is calculated based on the predicted query frequency, the most recent access time, and the total access frequency of the data node through the heat model;

[0020] Obtain the memory storage space and the size of each data node, and determine the number of data nodes that can be accommodated based on the memory storage space;

[0021] Sort all data nodes from high to low according to their scores, and select the corresponding number of data nodes in the front of the sorted data nodes according to the number of nodes that can be accommodated, and store the remaining nodes in external storage;

[0022] Build an in-memory AVL tree for the data nodes in the memory, build an external memory AVL tree for the data nodes in the external memory, and establish a top-level index for the external memory AVL tree;

[0023] A global index is established according to whether the data is stored in the memory or in the external memory. The global index uses a hash table as its data structure. The hash table uses a sub-hash function for mapping. Specifically, the sub-hash function is:

[0024]

[0025] Where NH is the calculated new hash value, H is the original hash value calculated based on the key value, i.e. the score, f is the frequency adjustment factor, F is the access frequency, Fc is the access frequency constant, Tc is the current time, T l is the time of the last access, d is the time decay factor, S is the size of the data node, S C is a space constant and m is the size of the hash table.

[0026] Furthermore, a multi-level data node storage index method calculates the data node score according to the predicted query frequency, the most recent access time and the total access frequency of the data node through a heat model, including:

[0027] Specifically, the heat model is:

[0028]

[0029] Where f is the frequency at which the predicted data node is queried, and f c is the predicted frequency constant, L is the length of time since the last visit, and L C is the time constant, is the access frequency at time t, t0 is the current time, λ is the time decay factor used to adjust the impact of past accesses on the current time, and t represents all time points from the past to the current time point t0.

[0030] Furthermore, a multi-level data node storage index method includes:

[0031] Define the structure of data nodes. Each data node includes: key value, stored data, left child node, right child node and height;

[0032] According to the AVL tree insertion algorithm, insert each memory data node into the AVL tree and keep the tree balanced. Perform insertion operations on all memory data nodes to build a memory AVL tree.

[0033] Divide the external memory data nodes into several small blocks, build an AVL tree for each small block of data nodes, and store these AVL trees in the external memory;

[0034] A top-level index is established for all external storage trees, and searches are performed in the in-memory AVL tree. For data that is not in the memory, the top-level index is used to locate the corresponding external storage tree for search.

[0035] The present invention proposes a multi-level data node storage index system, the system comprising:

[0036] Prediction module, used to predict the query frequency of data nodes in the future time period;

[0037] The partitioning and indexing module is used to calculate the score of the data node according to the predicted query frequency, the most recent access time and the total access frequency of the data node, divide the data node into memory data node and external storage data node according to the score, and establish a global index according to whether it is stored in memory or external storage.

[0038] Furthermore, in a multi-level data node storage index system, the prediction module includes:

[0039] A data collection module, used to collect historical records about data nodes being queried, wherein the historical records include a timestamp of the query, a query type, and an identifier of the data node, wherein the query type includes reading, updating, and deleting the data node;

[0040] Add a feature module to add seasonal activities, special events and emergency event information to the historical records. These information are marked as binary features. If there is a special event on a certain day, the feature is 1, otherwise it is 0;

[0041] A feature extraction module, configured to extract date features from the query timestamp, the date features including month, day of the week and hour, and create new derived variables from the date features, the new derived variables including the interval from the last query time to the current query time;

[0042] A data set building module is used to build the historical records, seasonal activity information, special event information, emergency event information, date features and derived variables into a data set;

[0043] The cleaning and preprocessing module is used to handle missing values ​​and outliers in the data set, as well as to standardize the data;

[0044] The data set division module is used to divide the data set into a training set and a test set, with 80% of the data used as the training set and the remaining 20% ​​as the test set;

[0045] The training model module is used to train the random forest model on the training set and use cross-validation to optimize model parameters and avoid overfitting;

[0046] Generate future feature data set module, used to generate feature data set containing future time period, the feature data set includes date feature, seasonality and event feature and historical feature, the date feature is to generate the date of future time period and extract the date feature of future time period, the seasonality and time feature is to mark special events and seasonal activities in future time period according to expectation, the historical feature is to take the query times of data in past time period as feature;

[0047] The future time period prediction module is used to use the trained random forest model to predict the feature data set containing the future time period;

[0048] The output query frequency module is used to convert the query probability output by the random forest model into query frequency.

[0049] Furthermore, in a multi-level data node storage index system, the partitioning and indexing module includes:

[0050] The score calculation module is used to calculate the data node score based on the predicted query frequency, the most recent access time and the total access frequency of the data node through the heat model;

[0051] A module for determining the number of data nodes stored in memory is used to obtain the memory storage space and the size of each data node, and determine the number of data nodes that can be accommodated according to the memory storage space;

[0052] The sorting and storage module is used to sort all data nodes from high to low according to the scores, and select the corresponding number of data nodes in the front of the sorted data nodes according to the number of nodes that can be accommodated, and the remaining nodes are stored in the external memory;

[0053] The AVL tree construction module is used to construct an in-memory AVL tree for the data nodes in the memory, to construct an external memory AVL tree for the data nodes in the external memory, and to establish a top-level index for the external memory AVL tree;

[0054] The hash mapping indexing module is used to establish a global index according to whether it is stored in the memory or in the external memory. The global index uses a hash table as its data structure. The hash table uses a sub-hash function for mapping. Specifically, the sub-hash function is:

[0055]

[0056] Where NH is the calculated new hash value, H is the original hash value calculated based on the key value, i.e. the score, f is the frequency adjustment factor, F is the access frequency, Fc is the access frequency constant, Tc is the current time, T l is the time of the last access, d is the time decay factor, S is the size of the data node, S Cis a space constant and m is the size of the hash table.

[0057] Furthermore, in a multi-level data node storage index system, the score calculation module includes a storage heat model module,

[0058] Specifically, the heat model is:

[0059]

[0060] Where f is the frequency at which the predicted data node is queried, and f c is the predicted frequency constant, L is the length of time since the last visit, and L C is the time constant, is the access frequency at time t, t0 is the current time, λ is the time decay factor used to adjust the impact of past accesses on the current time, and t represents all time points from the past to the current time point t0.

[0061] Furthermore, in a multi-level data node storage index system, the AVL tree construction module includes:

[0062] Definition module, used to define the structure of data nodes. Each data node includes: key value, stored data, left child node, right child node and height;

[0063] The node insertion module is used to insert each memory data node into the AVL tree according to the AVL tree insertion algorithm, and keep the tree balanced, perform insertion operations on all memory data nodes, and build a memory AVL tree;

[0064] The external memory data node division module is used to divide the external memory data node into several small blocks, build an AVL tree for each small block of data nodes, and store these AVL trees in the external memory;

[0065] A top-level index module is established to establish a top-level index for all external storage treelets, and to search in the internal memory AVL tree. For data that is not in the internal memory, the top-level index is used to locate the corresponding external storage AVL treelet for search.

[0066] The invention has the following beneficial effects: improving search efficiency, by predicting query frequency and calculating scores, giving priority to storing high-frequency accessed data nodes in memory, reducing I / O operations; the access speed of memory data nodes is much higher than that of external memory data nodes, which can significantly improve the response speed of the system; optimizing resource utilization, dynamically adjusting the data distribution of memory and external memory according to the scores of data nodes, optimizing the resource utilization of memory and external memory; memory space is limited, by reasonably allocating memory and external memory data nodes, it is possible to maximize the use of memory resources while ensuring search efficiency; flexibly adapting to data changes, the prediction model and score calculation can be dynamically adjusted according to data changes, and flexibly adapting to data changes According to changes in access patterns; for example, some data nodes are accessed frequently in a specific time period, and their scores can be adjusted to temporarily store them in the memory and then moved back to the external memory; the index structure can quickly locate the storage layer where the data node is located, reduce the search range, and improve the search efficiency; the index is used to quickly locate the memory or external memory, reducing unnecessary I / O operations; through the above technical solutions, efficient data node management and search can be achieved, the resource utilization of memory and external memory can be optimized, and the overall performance of the system can be improved; this method is particularly suitable for large-scale data processing and storage scenarios, and can optimize resource utilization while ensuring search efficiency, thereby improving the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of a multi-level data node storage index method according to the present invention;

[0068] Figure 2 A schematic diagram of a multi-level data node storage index structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0070] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. The embodiments described are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0072] One embodiment of the present invention provides a multi-level data node storage index method, the method comprising:

[0073] Predict the query frequency of data nodes in the future time period;

[0074] The score of the data node is calculated based on the predicted query frequency, recent access time and total access frequency of the data node. The data nodes are divided into memory data nodes and external storage data nodes based on the score, and a global index is established based on whether the data is stored in memory or external storage.

[0075] The working principle and effect of the above technical solution are: use historical data and prediction models (such as time series analysis, machine learning models, etc.) to predict the query frequency of each data node in the future time period. The prediction model can consider multiple factors, such as historical access frequency, time trend, seasonal changes, etc.; calculate the score of each data node based on the predicted query frequency, the most recent access time and the total access frequency, and divide the data nodes into memory data nodes and external storage data nodes based on the calculated score. Data nodes with higher scores are stored in memory first to improve access speed; data nodes with lower scores are stored in external storage to save memory space; establish a global index based on the storage location of the data node (memory or external storage). Improve search efficiency. By predicting query frequency and calculating scores, high-frequency access data nodes are stored in memory first, reducing I / O operations. The access speed of memory data nodes is much higher than that of external storage data nodes, which can significantly improve the response speed of the system. Optimize resource utilization. Dynamically adjust the data distribution of memory and external storage according to the scores of data nodes to optimize the resource utilization of memory and external storage. The memory space is limited. By reasonably allocating data nodes in memory and external storage, the memory resources can be maximized while ensuring search efficiency. Flexible adaptation to data changes. The prediction model and score calculation can be dynamically adjusted according to data changes, and can flexibly adapt to changes in data access patterns. For example, some data nodes have a high access frequency in a specific time period. They can be temporarily stored in memory by adjusting the scores and then moved back to external storage. The index structure can quickly locate the storage layer where the data node is located, reduce the search range, and improve the search efficiency. The index is used to quickly locate memory or external storage to reduce unnecessary I / O operations. Through the above technical solution, efficient data node management and search can be achieved, the resource utilization of memory and external storage can be optimized, and the overall performance of the system can be improved. This method is particularly suitable for large-scale data processing and storage scenarios, and can optimize resource utilization while ensuring search efficiency, thereby improving the overall performance of the system.

[0076] An embodiment of the present invention provides a multi-level data node storage index method for predicting the query frequency of a data node in a future time period, including:

[0077] Collecting historical records about queries on the data node, the historical records including a timestamp of the query, a query type, and an identifier of the data node, the query type including reading, updating, and deleting the data node;

[0078] Seasonal activities, special events and emergency events are added to the historical records. These information are marked as binary features. If there is a special event on a certain day, the feature is 1, otherwise it is 0;

[0079] Extracting date features from the timestamp of the query, the date features including month, day of the week and hour, creating new derived variables from the date features, the new derived variables including the interval from the last query time to the current query time;

[0080] constructing the historical records, seasonal activity information, special event information, emergency event information, date features and derived variables into a data set;

[0081] Handle missing values ​​and outliers in the data set, and standardize the data;

[0082] The data set is divided into a training set and a test set, with 80% of the data used as the training set and the remaining 20% ​​as the test set;

[0083] Train a random forest model on the training set and use cross-validation to optimize model parameters and avoid overfitting;

[0084] Generate a feature data set containing a future time period, the feature data set including date features, seasonality and event features, and historical features, the date feature is to generate the date of the future time period and extract the date features of the future time period, the seasonality and time features are to mark special events and seasonal activities in the future time period according to expectations, and the historical features are characterized by the number of queries for data in the past time period;

[0085] Use the trained random forest model to make predictions on a feature dataset containing future time periods;

[0086] Convert the query probabilities output by the random forest model into query frequencies.

[0087] The working principle and effect of the above technical solution are as follows: collect historical records about data node queries, which include query timestamps, query types (read, update, delete), and identifiers of data nodes; integrate information about seasonal activities, special events, and emergency events into historical records in the form of binary features, where the feature is 1 if there is a special event, otherwise it is 0; extract date features such as month, day of the week, and hour from the query timestamp, and create derived variables, such as the time interval from the last query to the current query; handle missing values ​​and outliers in the data set, and standardize the data to meet the model input requirements; divide the sorted data set into 80% training sets and 20% test sets, use the random forest model to train on the training set, and optimize the model parameters through cross-validation to improve the generalization ability of the model and avoid overfitting; based on the trained random forest model, construct a feature data set containing future time periods, which includes date features, expected seasonality and event features, and features based on the number of historical queries; use the model to predict the query frequency of data nodes in the future time period; convert the query probability output by the model into a specific query frequency for further application in data management and optimization strategies. Improve prediction accuracy. By integrating historical query data, seasonality, and special event information, the random forest model is provided with richer features to learn, so as to more accurately predict future query frequencies. Dynamic data management. According to the predicted query frequency information, data nodes can be more intelligently allocated to memory or external storage to optimize the efficiency and cost of data access. Response performance improvement. Accurate prediction of expected query demand and corresponding data scheduling can improve the overall response performance of the system, especially during high load or peak data access periods. Resource optimization. The model can help determine the best use strategy for memory and external storage, reduce unnecessary resource waste, and ensure a balance between performance and cost. This technical solution uses data science methods and machine learning technology to make data storage and access more intelligent and efficient, especially for scenarios that need to process large-scale data and complex query patterns. By providing accurate data access predictions, the system performance and user experience can be significantly improved.

[0088] An embodiment of the present invention is a multi-level data node storage index method, which calculates the score of the data node according to the predicted query frequency, the most recent access time and the total access frequency of the data node, divides the data node into memory data nodes and external storage data nodes according to the score, and establishes a global index according to whether the data is stored in the memory or the external storage, including:

[0089] The data node score is calculated based on the predicted query frequency, the most recent access time, and the total access frequency of the data node through the heat model;

[0090] Obtain the memory storage space and the size of each data node, and determine the number of data nodes that can be accommodated based on the memory storage space;

[0091] Sort all data nodes from high to low according to their scores, and select the corresponding number of data nodes in the front of the sorted data nodes according to the number of nodes that can be accommodated, and store the remaining nodes in external storage;

[0092] Build an in-memory AVL tree for the data nodes in the memory, build an external memory AVL tree for the data nodes in the external memory, and establish a top-level index for the external memory AVL tree;

[0093] A global index is established according to whether the data is stored in the memory or in the external memory. The global index uses a hash table as its data structure. The hash table uses a sub-hash function for mapping. Specifically, the sub-hash function is:

[0094]

[0095] Where NH is the calculated new hash value, H is the original hash value calculated based on the key value, i.e. the score, f is the frequency adjustment factor, F is the access frequency, Fc is the access frequency constant, Tc is the current time, T l is the time of the last access, d is the time decay factor, S is the size of the data node, S C is a space constant and m is the size of the hash table.

[0096] The working principle and effect of the above technical solution are as follows: use the heat model to calculate the score of each data node, taking into account factors including the predicted query frequency, the most recent access time and the total query frequency; determine the number of data nodes that can be stored in the memory based on the size of each data node and the total capacity of the memory; sort all data nodes according to the scores, select a part of the nodes with the highest scores to store in the memory, and store the remaining nodes in the external memory; construct an in-memory AVL tree of the nodes in the memory and an external memory AVL tree of the nodes in the external memory; establish a top-level index on the external memory AVL tree; the global index uses a hash table as the data structure, and uses a sub-hash function for mapping to determine the location of the node index; the hash mapping uses a sub-hash function. Resource optimization: By accurately calculating and sorting the heat scores of data nodes, memory can be used more efficiently, and the most frequently accessed data nodes can be kept in fast-access memory, while the less frequently accessed nodes are stored in external memory. Access efficiency is improved: through the hierarchical storage structure of memory and external memory, combined with an effective indexing system, data access speed and overall system performance can be significantly improved. Dynamic data management: data nodes are dynamically migrated according to the heat scores of real-time or periodic evaluations to ensure that the system adapts to changes in data access patterns and optimizes data access strategies. Query positioning optimization: using sub-hash functions to consider factors such as access frequency, time changes, and size, so that hash tables can be more adaptable to various load conditions, significantly reduce hash conflicts, and improve query efficiency. Through the above technical solutions, intelligent management of large-scale data storage and access can be achieved, storage efficiency and access speed can be improved, system operating costs can be reduced, and user experience can be optimized. It is particularly suitable for environments that need to handle data that is frequently accessed and has large dynamic changes, such as cloud services and big data analysis platforms.

[0097] The XOR operation in the sub-hash function is used to introduce another form of variation, so that even if two data items have the same original hash value H, the results after XOR processing may still be different, depending on the subsequent adjustment parameters, which helps to reduce hash collisions. The access frequency is introduced in the hash function for frequently accessed data items. Its hash value should be more randomized by combining items related to the access frequency, thereby reducing hash conflicts of hot data, making hot data scattered in different locations in the hash table, reducing the probability of multiple high-frequency access items falling in the same location, and increasing the access efficiency of the hash table;

[0098] Considering the time factor is used to incorporate the timeliness of data items into the calculation. A data item that has not been accessed for a long time should have a different hash value from a data item that has been frequently accessed recently. This helps to make old data and fresh data more evenly distributed in the hash table. It may also naturally "forget" data that is no longer frequently used due to the time factor, thereby providing faster access speed for frequently used data.

[0099] The introduction of data size may also affect the storage and access patterns of data. For example, large data items may require more frequent IO operations or more computing resources than small data items. This method optimizes hash value calculation by considering data size, which may make large objects relatively evenly distributed and avoid overcrowding of a hash bucket that affects performance. This design makes the hash function not only static or based only on the value of the data key, but dynamically adjusted to match the usage pattern and characteristics of the actual data. Such dynamic and adaptable nature improves the efficiency of data access, especially for frequently accessed data; reduces hash conflicts, enhances the overall performance of the hash table, and makes the performance of the hash table more adaptable to the needs of specific application scenarios, such as different strategies may be suitable for different markets or data processing needs.

[0100] An embodiment of the present invention is a multi-level data node storage index method, which calculates the data node score according to the predicted query frequency, the most recent access time and the total access frequency of the data node through a heat model, including:

[0101] Specifically, the heat model is:

[0102]

[0103] Where f is the frequency at which the predicted data node is queried, and f c is the predicted frequency constant, L is the length of time since the last visit, and L C is the time constant, is the access frequency at time t, t0 is the current time, λ is the time decay factor used to adjust the impact of past accesses on the current time, and t represents all time points from the past to the current time point t0.

[0104] The working principle and effect of the above technical solution are as follows: although past behavior can be used as an indicator of future behavior, its relevance decays over time. Old visits should have less impact on current heat evaluation, while recent visits should have a greater impact. The introduction of a time decay factor can simulate this natural decay process. The model can distinguish between short-term and long-term access trends, and use time as a variable to dynamically adjust the heat score of the data. Such a design can smoothly process the weight of the data and ensure that the evaluated heat can sensitively reflect the recent data access trend. By adjusting the time decay factor λ, the model can adapt to different usage scenarios. For example, in a system with rapid dynamic changes, λ can be set larger to reflect the latest changes more quickly; in a relatively stable system, λ can be set larger to reflect the latest changes more quickly. It can be set smaller to maintain a certain historical continuity. Without introducing time decay, the model may rely too much on historical data, giving too much weight to data nodes that have existed for a long time but are no longer popular or important, while emerging and potentially more important data nodes may not receive sufficient attention. Using exponentially decayed weighted average is a common processing method in time series analysis. It can handle the time effect of access frequency more smoothly and avoid fluctuations in heat scores due to extreme values ​​or outliers. While ensuring that long-term access patterns are reasonably reflected, the model can retain information on recent access activities, so that the heat score has both real-time response and long-term reference.

[0105] The time decay factor can be adjusted according to different business needs, which is flexible; practical, the model can better reflect the changes in user behavior in the real world and more accurately predict the popularity of data nodes in the short term; dynamically adjust the heat score according to time, making the cache or other resource optimization strategies more efficient and adaptable; avoid historical data overload and prevent the historical data accumulated over a long period of time from having a disproportionate impact on the current heat evaluation; suitable for large-scale, dynamic and constantly updated systems, such as recommendation systems, search engines or social networks; the heat evaluation model can be used as a core component in data indexing, caching operations and content distribution networks, and is easy to expand and integrate; by combining time factors and access frequency to evaluate heat, the provided model is particularly suitable for modern big data applications, which can better reflect the real-time changes in user needs and content popularity, help managers reasonably allocate resources, and improve the overall performance of the system.

[0106] An embodiment of the present invention provides a multi-level data node storage index method, comprising:

[0107] Define the structure of data nodes. Each data node includes: key value, stored data, left child node, right child node and height;

[0108] According to the AVL tree insertion algorithm, insert each memory data node into the AVL tree and keep the tree balanced. Perform insertion operations on all memory data nodes to build a memory AVL tree.

[0109] Divide the external memory data nodes into several small blocks, build an AVL tree for each small block of data nodes, and store these AVL trees in the external memory;

[0110] A top-level index is established for all external storage trees, and searches are performed in the in-memory AVL tree. For data that is not in the memory, the top-level index is used to locate the corresponding external storage tree for search.

[0111] The working principle and effect of the above technical solution are as follows: Data node structure definition, each data node contains the following attributes:

[0112] Key value: used to uniquely identify the data node; stored data: the actual stored data content; left child node: a pointer to the left child node; right child node: a pointer to the right child node; height: the height of the node in the AVL tree, used to keep the tree balanced; According to the insertion algorithm of the AVL tree, each memory data node is inserted into the AVL tree. During the insertion process, the balance of the AVL tree is maintained, that is, the height and structure of the tree are checked and adjusted after each insertion to ensure that the height difference of the left and right subtrees does not exceed 1; all memory data nodes are inserted, and finally a balanced memory AVL tree is constructed; the external memory data nodes are divided into several small blocks, each of which contains a certain number of data nodes; an AVL tree is constructed for the data nodes of each small block to ensure that each tree is balanced internally; these AVL trees are stored in the external memory; a top-level index is established for all external memory trees. The top-level index can be a simple array or hash table, which records the root node information of each tree and its location in the external memory; when searching in the memory AVL tree, if the data node is not in the memory, the corresponding external memory tree is located through the top-level index; further search operations are performed in the external memory tree to find the target data node. The search efficiency of the memory AVL tree is high, and it can quickly locate the data nodes in the memory. For data nodes that are not in the memory, the top-level index is used to quickly locate the corresponding small tree in the external memory, and then the search is performed in the small tree, which reduces the search range and improves the search efficiency. Balance and stability: The balance of the AVL tree enables both the data nodes in the memory and the small tree in the external memory to maintain high operating efficiency. Through block division and small tree construction, the management of external memory data nodes is more orderly, avoiding performance bottlenecks during large-scale data search. Resource optimization: data nodes in the memory are preferentially stored in the memory AVL tree, making full use of the high-speed access characteristics of the memory, and the external memory data nodes are stored in blocks, which reduces the amount of data in a single I / O operation and improves the efficiency of external memory access.

[0113] One embodiment of the present invention provides a multi-level data node storage index system, the system comprising:

[0114] Prediction module, used to predict the query frequency of data nodes in the future time period;

[0115] The partitioning and indexing module is used to calculate the score of the data node according to the predicted query frequency, the most recent access time and the total access frequency of the data node, divide the data node into memory data node and external storage data node according to the score, and establish a global index according to whether it is stored in memory or external storage.

[0116] The working principle and effect of the above technical solution are: use historical data and prediction models (such as time series analysis, machine learning models, etc.) to predict the query frequency of each data node in the future time period. The prediction model can consider multiple factors, such as historical access frequency, time trend, seasonal changes, etc.; calculate the score of each data node based on the predicted query frequency, the most recent access time and the total access frequency, and divide the data nodes into memory data nodes and external storage data nodes based on the calculated score. Data nodes with higher scores are stored in memory first to improve access speed; data nodes with lower scores are stored in external storage to save memory space; establish a global index based on the storage location of the data node (memory or external storage). Improve search efficiency. By predicting query frequency and calculating scores, high-frequency access data nodes are stored in memory first, reducing I / O operations and improving search efficiency. The access speed of memory data nodes is much higher than that of external storage data nodes, which can significantly improve the response speed of the system. Optimize resource utilization. Dynamically adjust the data distribution of memory and external storage according to the scores of data nodes to optimize the resource utilization of memory and external storage. Memory space is limited. By reasonably allocating memory and external storage data nodes, memory resources can be maximized while ensuring search efficiency. Flexible adaptation to data changes. The prediction model and score calculation can be dynamically adjusted according to data changes, and can flexibly adapt to data changes. Changes in access patterns; for example, if some data nodes are accessed more frequently within a specific time period, they can be temporarily stored in the memory by adjusting the score and then moved back to the external memory; the index structure can quickly locate the storage layer where the data node is located, reduce the search range, and improve search efficiency; the index is used to quickly locate the memory or external memory and reduce unnecessary I / O operations; through the above technical solutions, efficient data node management and search can be achieved, the resource utilization of memory and external memory can be optimized, and the overall performance of the system can be improved; this method is particularly suitable for large-scale data processing and storage scenarios, and can optimize resource utilization while ensuring search efficiency, thereby improving the overall performance of the system.

[0117] In one embodiment of the present invention, a multi-level data node storage index system is provided, wherein the prediction module comprises:

[0118] A data collection module, used to collect historical records about data nodes being queried, wherein the historical records include a timestamp of the query, a query type, and an identifier of the data node, wherein the query type includes reading, updating, and deleting the data node;

[0119] Add a feature module to add seasonal activities, special events and emergency event information to the historical records. These information are marked as binary features. If there is a special event on a certain day, the feature is 1, otherwise it is 0;

[0120] A feature extraction module, configured to extract date features from the query timestamp, the date features including month, day of the week and hour, and create new derived variables from the date features, the new derived variables including the interval from the last query time to the current query time;

[0121] A data set building module is used to build the historical records, seasonal activity information, special event information, emergency event information, date features and derived variables into a data set;

[0122] The cleaning and preprocessing module is used to handle missing values ​​and outliers in the data set, as well as to standardize the data;

[0123] The data set division module is used to divide the data set into a training set and a test set, with 80% of the data used as the training set and the remaining 20% ​​as the test set;

[0124] The training model module is used to train the random forest model on the training set and use cross-validation to optimize model parameters and avoid overfitting;

[0125] Generate future feature data set module, used to generate feature data set containing future time period, the feature data set includes date feature, seasonality and event feature and historical feature, the date feature is to generate the date of future time period and extract the date feature of future time period, the seasonality and time feature is to mark special events and seasonal activities in future time period according to expectation, the historical feature is to take the query times of data in past time period as feature;

[0126] The future time period prediction module is used to use the trained random forest model to predict the feature data set containing the future time period;

[0127] The output query frequency module is used to convert the query probability output by the random forest model into query frequency.

[0128] The working principle and effect of the above technical solution are as follows: collect historical records about data node queries, which include query timestamps, query types (read, update, delete), and identifiers of data nodes; integrate information about seasonal activities, special events, and emergency events into historical records in the form of binary features, where the feature is 1 if there is a special event, otherwise it is 0; extract date features such as month, day of the week, and hour from the query timestamp, and create derived variables, such as the time interval from the last query to the current query; handle missing values ​​and outliers in the data set, and standardize the data to meet the model input requirements; divide the sorted data set into 80% training sets and 20% test sets, use the random forest model to train on the training set, and optimize the model parameters through cross-validation to improve the generalization ability of the model and avoid overfitting; based on the trained random forest model, construct a feature data set containing future time periods, which includes date features, expected seasonality and event features, and features based on the number of historical queries; use the model to predict the query frequency of data nodes in the future time period; convert the query probability output by the model into a specific query frequency for further application in data management and optimization strategies. Improve prediction accuracy. By integrating historical query data, seasonality, and special event information, the random forest model is provided with richer features to learn, so as to more accurately predict future query frequencies. Dynamic data management. According to the predicted query frequency information, data nodes can be more intelligently allocated to memory or external storage to optimize the efficiency and cost of data access. Response performance improvement. Accurate prediction of expected query demand and corresponding data scheduling can improve the overall response performance of the system, especially during high load or peak data access periods. Resource optimization. The model can help determine the best use strategy for memory and external storage, reduce unnecessary resource waste, and ensure a balance between performance and cost. This technical solution uses data science methods and machine learning technology to make data storage and access more intelligent and efficient, especially for scenarios that need to process large-scale data and complex query patterns. By providing accurate data access predictions, the system performance and user experience can be significantly improved.

[0129] In one embodiment of the present invention, a multi-level data node storage index system is provided, wherein the partitioning and indexing module comprises:

[0130] The score calculation module is used to calculate the data node score based on the predicted query frequency, the most recent access time and the total access frequency of the data node through the heat model;

[0131] A module for determining the number of data nodes stored in memory is used to obtain the memory storage space and the size of each data node, and determine the number of data nodes that can be accommodated according to the memory storage space;

[0132] The sorting and storage module is used to sort all data nodes from high to low according to the scores, and select the corresponding number of data nodes in the front of the sorted data nodes according to the number of nodes that can be accommodated, and the remaining nodes are stored in the external memory;

[0133] The AVL tree construction module is used to construct an in-memory AVL tree for the data nodes in the memory, to construct an external memory AVL tree for the data nodes in the external memory, and to establish a top-level index for the external memory AVL tree;

[0134] The hash mapping indexing module is used to establish a global index according to whether it is stored in the memory or in the external memory. The global index uses a hash table as its data structure. The hash table uses a sub-hash function for mapping. Specifically, the sub-hash function is:

[0135]

[0136] Where NH is the calculated new hash value, H is the original hash value calculated based on the key value, i.e. the score, f is the frequency adjustment factor, F is the access frequency, Fc is the access frequency constant, Tc is the current time, T l is the time of the last access, d is the time decay factor, S is the size of the data node, S C is a space constant and m is the size of the hash table.

[0137] The working principle and effect of the above technical solution are as follows: use the heat model to calculate the score of each data node, taking into account factors including the predicted query frequency, the most recent access time and the total query frequency; determine the number of data nodes that can be stored in the memory based on the size of each data node and the total capacity of the memory; sort all data nodes according to the scores, select a part of the nodes with the highest scores to store in the memory, and store the remaining nodes in the external memory; construct an in-memory AVL tree of the nodes in the memory and an external memory AVL tree of the nodes in the external memory; establish a top-level index on the external memory AVL tree; the global index uses a hash table as the data structure, and uses a sub-hash function for mapping to determine the location of the node index; the hash mapping uses a sub-hash function. Resource optimization: By accurately calculating and sorting the heat scores of data nodes, memory can be used more efficiently, and the most frequently accessed data nodes can be kept in fast-access memory, while the less frequently accessed nodes are stored in external memory. Access efficiency is improved: through the hierarchical storage structure of memory and external memory, combined with an effective indexing system, data access speed and overall system performance can be significantly improved. Dynamic data management: data nodes are dynamically migrated according to the heat scores of real-time or periodic evaluations to ensure that the system adapts to changes in data access patterns and optimizes data access strategies. Query positioning optimization: using sub-hash functions to consider factors such as access frequency, time changes, and size, so that hash tables can be more adaptable to various load conditions, significantly reduce hash conflicts, and improve query efficiency. Through the above technical solutions, intelligent management of large-scale data storage and access can be achieved, storage efficiency and access speed can be improved, system operating costs can be reduced, and user experience can be optimized. It is particularly suitable for environments that need to handle data that is frequently accessed and has large dynamic changes, such as cloud services and big data analysis platforms.

[0138] The XOR operation in the sub-hash function is used to introduce another form of variation, so that even if two data items have the same original hash value H, the results after XOR processing may still be different, depending on the subsequent adjustment parameters, which helps to reduce hash collisions. The access frequency is introduced in the hash function for frequently accessed data items. Its hash value should be more randomized by combining items related to the access frequency, thereby reducing hash conflicts of hot data, making hot data scattered in different locations in the hash table, reducing the probability of multiple high-frequency access items falling in the same location, and increasing the access efficiency of the hash table;

[0139] Considering the time factor is used to incorporate the timeliness of data items into the calculation. A data item that has not been accessed for a long time should have a different hash value from a data item that has been frequently accessed recently. This helps to make old data and fresh data more evenly distributed in the hash table. It may also naturally "forget" data that is no longer frequently used due to the time factor, thereby providing faster access speed for frequently used data.

[0140] The introduction of data size may also affect the storage and access patterns of data. For example, large data items may require more frequent IO operations or more computing resources than small data items. This method optimizes hash value calculation by considering data size, which may make large objects relatively evenly distributed and avoid overcrowding of a hash bucket that affects performance. This design makes the hash function not only static or based only on the value of the data key, but dynamically adjusted to match the usage pattern and characteristics of the actual data. Such dynamic and adaptable nature improves the efficiency of data access, especially for frequently accessed data; reduces hash conflicts, enhances the overall performance of the hash table, and makes the performance of the hash table more adaptable to the needs of specific application scenarios, such as different strategies may be suitable for different markets or data processing needs.

[0141] In one embodiment of the present invention, a multi-level data node storage index system is provided, wherein the score calculation module includes a storage heat model module.

[0142] Specifically, the heat model is:

[0143]

[0144] Where f is the frequency at which the predicted data node is queried, and f c is the predicted frequency constant, L is the length of time since the last visit, and L C is the time constant, is the access frequency at time t, t0 is the current time, λ is the time decay factor used to adjust the impact of past accesses on the current time, and t represents all time points from the past to the current time point t0.

[0145] An embodiment of the present invention is a multi-level data node storage index system, wherein the AVL tree construction module includes:

[0146] Definition module, used to define the structure of data nodes. Each data node includes: key value, stored data, left child node, right child node and height;

[0147] The node insertion module is used to insert each memory data node into the AVL tree according to the AVL tree insertion algorithm, and keep the tree balanced, perform insertion operations on all memory data nodes, and build a memory AVL tree;

[0148] The external memory data node division module is used to divide the external memory data node into several small blocks, build an AVL tree for each small block of data nodes, and store these AVL trees in the external memory;

[0149] Establish a top-level index module to establish a top-level index for all external storage trees, search in the internal memory AVL tree, and locate the corresponding external storage AVL tree for search through the top-level index for data not in the internal memory (such as Figure 2 as shown).

[0150] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-level data node storage index method, characterized in that: The method comprises: Predict the query frequency of data nodes in the future time period; The score of the data node is calculated based on the predicted query frequency, the most recent access time, and the total access frequency of the data node. The data node is divided into memory data nodes and external storage data nodes based on the score, and a global index is established based on whether the data node is stored in memory or external storage. The predicted query frequency of the data node in the future time period includes: Collecting historical records about queries on the data node, the historical records including a timestamp of the query, a query type, and an identifier of the data node, the query type including reading, updating, and deleting the data node; Seasonal activities, special events and emergency events are added to the historical records. These information are marked as binary features. If there is a special event on a certain day, the feature is 1, otherwise it is 0; Extracting date features from the timestamp of the query, the date features including month, day of the week and hour, creating new derived variables from the date features, the new derived variables including the interval from the last query time to the current query time; constructing the historical records, seasonal activity information, special event information, emergency event information, date features and derived variables into a data set; Handle missing values ​​and outliers in the data set, and standardize the data; The data set is divided into a training set and a test set, with 80% of the data used as the training set and the remaining 20% ​​as the test set; Train a random forest model on the training set and use cross-validation to optimize model parameters and avoid overfitting; Generate a feature data set containing a future time period, the feature data set including date features, seasonality and event features, and historical features, the date feature is to generate the date of the future time period and extract the date features of the future time period, the seasonality and time features are to mark special events and seasonal activities in the future time period according to expectations, and the historical features are characterized by the number of queries for data in the past time period; Use the trained random forest model to make predictions on a feature dataset containing future time periods; Convert the query probabilities output by the random forest model into query frequencies.

2. A multi-level data node storage index method according to claim 1, characterized in that: The score of the data node is calculated based on the predicted query frequency, the most recent access time, and the total access frequency of the data node. The data node is divided into memory data nodes and external storage data nodes based on the score. A global index is established based on whether the data is stored in memory or external storage, including: The data node score is calculated based on the predicted query frequency, the most recent access time, and the total access frequency of the data node through the heat model; Obtain the memory storage space and the size of each data node, and determine the number of data nodes that can be accommodated based on the memory storage space; Sort all data nodes from high to low according to their scores, and select the corresponding number of data nodes in the front of the sorted data nodes according to the number of nodes that can be accommodated, and store the remaining nodes in external storage; Build an in-memory AVL tree for the data nodes in the memory, build an external memory AVL tree for the data nodes in the external memory, and establish a top-level index for the external memory AVL tree; A global index is established according to whether the data is stored in the memory or in the external memory. The global index uses a hash table as its data structure. The hash table uses a sub-hash function for mapping. Specifically, the sub-hash function is: Where NH is the calculated new hash value, H is the original hash value calculated based on the key value, i.e. the score, f is the frequency adjustment factor, F is the access frequency, Fc is the access frequency constant, Tc is the current time, T l is the time of the last access, d is the time decay factor, S is the size of the data node, S C is a space constant and m is the size of the hash table.

3. A multi-level data node storage index method according to claim 2, characterized in that: The heat model calculates the data node score based on the predicted query frequency, recent access time, and total access frequency of the data node, including: Specifically, the heat model is: Among them, α, β, and γ represent weight coefficients, f is the frequency of query of the predicted data node, and f c is the predicted frequency constant, L is the length of time since the last visit, and L C is the time constant, is the access frequency at time t, t0 is the current time, λ is the time decay factor, which is used to adjust the impact of past accesses on the current time, t represents all time points from the past to the current time point t0, A constant representing the number of visits.

4. A multi-level data node storage index method according to claim 2, characterized in that: Build an in-memory AVL tree for the data nodes in the memory, build an external memory AVL tree for the data nodes in the external memory, and create a top-level index for the external memory AVL tree, including: Define the structure of data nodes. Each data node includes: key value, stored data, left child node, right child node and height; According to the AVL tree insertion algorithm, insert each memory data node into the AVL tree and keep the tree balanced. Perform insertion operations on all memory data nodes to build a memory AVL tree. Divide the external memory data nodes into several small blocks, build an AVL tree for each small block of data nodes, and store these AVL trees in the external memory; A top-level index is established for all external storage trees, and searches are performed in the in-memory AVL tree. For data that is not in the memory, the top-level index is used to locate the corresponding external storage tree for search.

5. A multi-level data node storage index system, characterized in that: The system comprises: Prediction module, used to predict the query frequency of data nodes in the future time period; The partitioning and indexing module is used to calculate the score of the data node according to the predicted query frequency, the most recent access time and the total access frequency of the data node, divide the data node into memory data node and external storage data node according to the score, and establish a global index according to whether it is stored in memory or external storage; Wherein, the prediction module includes: A data collection module, used to collect historical records about data nodes being queried, wherein the historical records include a timestamp of the query, a query type, and an identifier of the data node, wherein the query type includes reading, updating, and deleting the data node; Add a feature module to add seasonal activities, special events and emergency event information to the historical records. These information are marked as binary features. If there is a special event on a certain day, the feature is 1, otherwise it is 0; A feature extraction module, configured to extract date features from the query timestamp, the date features including month, day of the week and hour, and create new derived variables from the date features, the new derived variables including the interval from the last query time to the current query time; A data set building module is used to build the historical records, seasonal activity information, special event information, emergency event information, date features and derived variables into a data set; The cleaning and preprocessing module is used to handle missing values ​​and outliers in the data set, as well as to standardize the data; The data set division module is used to divide the data set into a training set and a test set, with 80% of the data used as the training set and the remaining 20% ​​as the test set; The training model module is used to train the random forest model on the training set and use cross-validation to optimize model parameters and avoid overfitting; Generate future feature data set module, used to generate feature data set containing future time period, the feature data set includes date feature, seasonality and event feature and historical feature, the date feature is to generate the date of future time period and extract the date feature of future time period, the seasonality and time feature is to mark special events and seasonal activities in future time period according to expectation, the historical feature is to take the query times of data in past time period as feature; The future time period prediction module is used to use the trained random forest model to predict the feature data set containing the future time period; The output query frequency module is used to convert the query probability output by the random forest model into query frequency.

6. A multi-level data node storage index system according to claim 5, characterized in that: The partitioning and indexing module comprises: The score calculation module is used to calculate the data node score based on the predicted query frequency, the most recent access time and the total access frequency of the data node through the heat model; A module for determining the number of data nodes stored in memory is used to obtain the memory storage space and the size of each data node, and determine the number of data nodes that can be accommodated according to the memory storage space; The sorting and storage module is used to sort all data nodes from high to low according to the scores, and select the corresponding number of data nodes in the front of the sorted data nodes according to the number of nodes that can be accommodated, and the remaining nodes are stored in the external memory; The AVL tree construction module is used to construct an in-memory AVL tree for the data nodes in the memory, to construct an external memory AVL tree for the data nodes in the external memory, and to establish a top-level index for the external memory AVL tree; The hash mapping indexing module is used to establish a global index according to whether it is stored in the memory or in the external memory. The global index uses a hash table as its data structure. The hash table uses a sub-hash function for mapping. Specifically, the sub-hash function is: Where NH is the calculated new hash value, H is the original hash value calculated based on the key value, i.e. the score, f is the frequency adjustment factor, F is the access frequency, Fc is the access frequency constant, Tc is the current time, T l is the time of the last access, d is the time decay factor, S is the size of the data node, S C is a space constant and m is the size of the hash table.

7. A multi-level data node storage index system according to claim 6, characterized in that: The score calculation module includes a storage heat model module, Specifically, the heat model is: Among them, α, β, and γ represent weight coefficients, f is the frequency of query of the predicted data node, and f c is the predicted frequency constant, L is the length of time since the last visit, and L C is the time constant, is the access frequency at time t, t0 is the current time, λ is the time decay factor, which is used to adjust the impact of past accesses on the current time, t represents all time points from the past to the current time point t0, A constant representing the number of visits.

8. A multi-level data node storage index system according to claim 6, characterized in that: The AVL tree construction module includes: Definition module, used to define the structure of data nodes. Each data node includes: key value, stored data, left child node, right child node and height; The node insertion module is used to insert each memory data node into the AVL tree according to the AVL tree insertion algorithm, and keep the tree balanced, perform insertion operations on all memory data nodes, and build a memory AVL tree; The external memory data node division module is used to divide the external memory data node into several small blocks, build an AVL tree for each small block of data nodes, and store these AVL trees in the external memory; A top-level index module is established to establish a top-level index for all external storage treelets, and to search in the internal memory AVL tree. For data that is not in the internal memory, the top-level index is used to locate the corresponding external storage AVL treelet for search.