Automatic accurate memory expansion and contraction dynamic management method
By introducing estimated expansion threshold and data kinship, combined with time series analysis and exponential smoothing algorithm, the problems of slow response speed and low judgment accuracy of hot and cold data in the existing technology are solved, and efficient and accurate memory expansion is achieved Dynamic management reduces resource waste and repeated operations.
Patent Information
- Application Number
- CN202411063053.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-13
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art only adjusts resources after the data access mode changes, has a slow response speed, and relies on simple statistical methods to judge hot and cold data, which has low accuracy, resulting in frequent repeated operations and waste of resources.
The concept of estimated expansion threshold is introduced, the expansion value is calculated through data kinship and time series analysis algorithm, memory expansion operations are performed in advance, and the exponential smoothing algorithm and redis cache are combined to improve judgment accuracy and response speed.
It improves the response speed and accuracy of memory expansion, reduces repeated operations and resource waste, and has high system operation efficiency and strong applicability.
Abstract
Description
Technical Field
[0001] The present invention relates to the field of database technology, and in particular to an automatic and precise memory expansion and contraction dynamic management method. Background Art
[0002] With the rapid development of information technology, data has become one of the core assets of enterprises and organizations. Faced with the storage and management needs of massive data, traditional data storage methods have gradually revealed their limitations, especially in terms of data processing efficiency, storage cost, and data security. In order to more effectively cope with these challenges, hot and cold data storage methods have emerged and have gradually become an important research direction in the field of data storage. By storing these two types of data on storage media with different performance and costs, storage efficiency can be significantly improved and storage costs can be reduced. In hot and cold data storage systems, memory is the key link in data storage and access, and its performance directly affects the efficiency and stability of the entire system.
[0003] The existing technology sets a threshold for defining hot and cold data based on the experience of different businesses. The system monitors the access of all data in real time, records the number of accesses and the access time, and divides the data into hot data and cold data according to the preset access threshold. According to the hot and cold classification results of the data, the storage location of the data in the memory and the size of the allocated resources are dynamically adjusted. The defects of this method are:
[0004] Existing technologies often require significant changes in data access patterns before resources can be adjusted, and are unable to respond to changes in data hotness and coldness in real time, resulting in slow response speeds.
[0005] When judging the popularity of data, existing technologies often rely on simple statistical methods. However, since access frequency is not the only criterion, it is difficult to accurately reflect the actual usage of data, so the accuracy is not high.
[0006] The existing technology uses empirical values or 1.75 times expansion for memory expansion, which is a rough value and often leads to frequent operations, low efficiency, or waste of resources. Summary of the invention
[0007] In order to address the deficiencies in the prior art, the present invention provides an automated and precise dynamic management method for memory expansion and contraction. The automated and precise dynamic management method for memory expansion and contraction introduces the concept of estimated expansion and contraction thresholds to perform memory expansion and contraction operations in advance, thereby solving the technical problem of slow response speed. The method also solves the technical problem of low accuracy in judging hot and cold data through the integration of data lineage relationships. The method improves the accuracy of expansion and contraction predictions by calculating expansion and contraction values in advance rather than fixed expansion and contraction values, thereby solving the technical problems of frequent repeated operations and waste of resources.
[0008] To achieve the above object, the present invention adopts the following technical solution:
[0009] The present invention provides an automated and precise memory expansion and contraction dynamic management method, comprising the following steps:
[0010] Count the data access records of a single database table within a preset time range;
[0011] Integrate the data lineage relationship of all database tables, and record the access to the current data table and the previous data table at the same time;
[0012] Analyze the access records, calculate the estimated scaling threshold and the scaling threshold, wherein the scaling threshold is greater than the estimated scaling threshold;
[0013] When the access record data reaches the estimated scaling threshold, the scaling value is calculated;
[0014] When the access record data reaches the scaling threshold, automatic scaling is performed according to the scaling value.
[0015] The automated precise memory expansion and contraction dynamic management method of the present invention firstly integrates the data lineage of the database table, and expands and records the data of any operation upward, which is more beneficial to the judgment accuracy of hot and cold data, and is combined with the access records within a preset time range of statistics, which lays the foundation for accurately calculating the estimated expansion and contraction threshold and the expansion and contraction threshold. At the same time, the concept of estimated expansion and contraction threshold is introduced, which improves the efficiency and accuracy of expansion and contraction. Accurate automated expansion and contraction is performed by accurately calculating the expansion and contraction values based on actual conditions, which reduces repeated operations and avoids waste of resources.
[0016] The automated precise memory expansion and contraction dynamic management method of the present invention introduces the concept of estimated expansion and contraction thresholds to perform memory expansion and contraction operations in advance, thereby solving the technical problem of slow response speed. By integrating data lineage relationships, the technical problem of low accuracy in judging hot and cold data is solved. By calculating expansion and contraction values in advance rather than fixed expansion and contraction values, the accuracy of expansion and contraction predictions is improved, thereby solving the technical problems of frequent repeated operations and waste of resources.
[0017] In a further technical solution, the estimated scaling threshold and the scaling threshold are calculated by an ARIMA time series analysis algorithm.
[0018] The estimated scaling thresholds and scaling thresholds calculated by the ARIMA time series analysis algorithm are accurate in nodes, and the system operates efficiently and responds quickly.
[0019] In a further technical solution, the expansion and contraction value is calculated by an exponential smoothing algorithm.
[0020] The exponential smoothing algorithm is used to calculate the expansion and contraction values to further improve the accuracy of the expansion and contraction prediction.
[0021] In a further technical solution, when access records of the current data table and the previous data table are simultaneously performed on access data, the record times are cached using redis.
[0022] Using redis for caching can reduce database usage.
[0023] In a further technical solution, a specified instruction is received and the scaling threshold is adjusted.
[0024] By accepting instructions, the scaling thresholds can be adjusted manually, allowing manual adjustment and greater applicability.
[0025] The beneficial effects are:
[0026] 1. The automated precise memory expansion and contraction dynamic management method of the present invention introduces the concept of estimated expansion and contraction thresholds to perform memory expansion and contraction operations in advance, thereby solving the technical problem of slow response speed. By integrating data lineage relationships, the technical problem of low accuracy in judging hot and cold data is solved. By calculating expansion and contraction values in advance rather than fixed expansion and contraction values, the accuracy of expansion and contraction prediction is improved, thereby solving the technical problems of frequent repeated operations and waste of resources.
[0027] 2. The estimated scaling threshold and scaling threshold calculated by the ARIMA time series analysis algorithm have accurate nodes, high system operation efficiency and fast response speed.
[0028] 3. Use exponential smoothing algorithm to calculate the expansion and contraction values to further improve the accuracy of expansion and contraction prediction.
[0029] 4. Using redis for caching can reduce database usage.
[0030] 5. By accepting instructions, the scaling thresholds can be adjusted manually, allowing manual adjustment, which is more applicable. DETAILED DESCRIPTION
[0031] The present invention will be further described below:
[0032] Example:
[0033] An automated and precise memory expansion and contraction dynamic management method comprises the following steps:
[0034] Count the data access records of a single database table within a preset time range;
[0035] Integrate the data lineage relationship of all database tables, and record the access to the current data table and the previous data table at the same time;
[0036] Analyze the access records, calculate the estimated scaling threshold and the scaling threshold, wherein the scaling threshold is greater than the estimated scaling threshold;
[0037] When the access record data reaches the estimated scaling threshold, the scaling value is calculated;
[0038] When the access record data reaches the scaling threshold, automatic scaling is performed according to the scaling value.
[0039] Data lineage, also known as data pedigree, data origin, and data genealogy, refers to the natural relationship between data during its entire life cycle, from generation, processing, processing, fusion, circulation to final extinction. It records the link relationship generated by data, which is similar to human blood relationship, so it is called data lineage relationship.
[0040] Through data lineage, data that has a certain relationship with the accessed data can be counted in the access record. In reality, before and after the data is accessed, the data that has data lineage has a greater probability of being accessed. Therefore, from the perspective of considering hot and cold data, this recording method can obtain more accurate results when calculating the estimated scaling threshold, scaling threshold and scaling value.
[0041] The automated precise memory expansion and contraction dynamic management method of the present invention firstly integrates the data lineage of the database table, and expands and records the data of any operation upward, which is more beneficial to the judgment accuracy of hot and cold data, and is combined with the access records within a preset time range of statistics, which lays the foundation for accurately calculating the estimated expansion and contraction threshold and the expansion and contraction threshold. At the same time, the concept of estimated expansion and contraction threshold is introduced, which improves the efficiency and accuracy of expansion and contraction. Accurate automated expansion and contraction is performed by accurately calculating the expansion and contraction values based on actual conditions, which reduces repeated operations and avoids waste of resources.
[0042] The automated precise memory expansion and contraction dynamic management method of the present invention introduces the concept of estimated expansion and contraction thresholds to perform memory expansion and contraction operations in advance, thereby solving the technical problem of slow response speed. By integrating data lineage relationships, the technical problem of low accuracy in judging hot and cold data is solved. By calculating expansion and contraction values in advance rather than fixed expansion and contraction values, the accuracy of expansion and contraction predictions is improved, thereby solving the technical problems of frequent repeated operations and waste of resources.
[0043] In another embodiment, the estimated scaling threshold and the scaling threshold are calculated by an ARIMA time series analysis algorithm.
[0044] The estimated scaling thresholds and scaling thresholds calculated by the ARIMA time series analysis algorithm are accurate in nodes, and the system operates efficiently and responds quickly.
[0045] The core idea of ARIMA time series analysis is to transform a non-stationary time series into a stationary time series by differencing, and then use autoregression and moving average methods to build a model to predict future data points. The ARIMA model consists of three parts: autoregression, differencing, and moving average. Its general form is ARIMA (p, d, q), where p: the number of autoregressive terms, which indicates the number of lagged observations used in the model. d: the number of differencing operations, which indicates the number of differencing operations performed to make the series stationary. q: the number of moving average terms, which indicates the number of lagged forecast errors in the forecast error term.
[0046] In another embodiment, the scaling value is calculated by an exponential smoothing algorithm.
[0047] The exponential smoothing algorithm is used to calculate the expansion and contraction values to further improve the accuracy of the expansion and contraction prediction.
[0048] The exponential smoothing algorithm uses the smoothing constant a to adjust the degree of influence of observations at different time points on the predicted value. The larger the a value, the greater the influence of recent observations on the predicted value; the smaller the a value, the greater the influence of long-term observations on the predicted value.
[0049] In another embodiment, when access records of the current data table and the previous data table are simultaneously recorded for access data, the record times are cached using redis.
[0050] Using redis for caching can reduce database usage.
[0051] In another embodiment, a specified instruction is received to adjust the scaling threshold.
[0052] By accepting instructions, the scaling thresholds can be adjusted manually, allowing manual adjustment and greater applicability.
[0053] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.
Claims
1. An automated and precise memory expansion and contraction dynamic management method, It is characterized in that The following steps are involved: Count the data access records of a single database table within a preset time range; Integrate the data lineage relationship of all database tables, and record the access to the current data table and the previous data table at the same time; Analyze the access records, calculate the estimated scaling threshold and the scaling threshold, wherein the scaling threshold is greater than the estimated scaling threshold; When the access record data reaches the estimated scaling threshold, the scaling value is calculated; When the access record data reaches the scaling threshold, automatic scaling is performed according to the scaling value.
2. The automatic and precise memory expansion and contraction dynamic management method according to claim 1, It is characterized in that The estimated scaling threshold and scaling threshold are calculated by ARIMA time series analysis algorithm.
3. The automatic and precise memory expansion and contraction dynamic management method according to claim 1, It is characterized in that The scaling value is calculated by an exponential smoothing algorithm.
4. The automatic and precise memory expansion and contraction dynamic management method according to claim 1, It is characterized in that When access data is accessed to the current data table and the previous data table at the same time, the number of records is cached using redis.
5. The automatic and precise memory expansion and contraction dynamic management method according to claim 1, It is characterized in that Receive specified instructions and adjust scaling thresholds.