A method and system for dynamic management of multi-layer data cache

Abstract

The application relates to the technical field of data cache management, in particular to a multi-layer data cache dynamic management method and system, which comprises the following steps: based on a data access path, extracting a data access path level identifier, analyzing path level depth, recording access continuity, classifying paths as stable and unstable, and obtaining path level distribution information.In the application, through level analysis and stability classification of the data access path, the cache resource allocation efficiency is improved, stable paths and unstable paths are classified, access delay is reduced, access continuity is improved, cache waste caused by jumping is reduced, space utilization is improved, cache pollution is avoided, storage order is optimized, hit rate is improved, resource competition is reduced through dynamic elimination, storage medium I / O load is monitored, medium load is balanced, throughput capacity is improved, data flow is optimized, cache cooperation capacity is improved, the cache level can flexibly adapt to access mode changes, and data storage stability and load balancing are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data cache management, and particularly relates to a multi-layer data cache dynamic management method and system. BACKGROUND

[0002] The technical field of data cache management includes data storage, access optimization, cache replacement strategy and multi-layer cache structure design, and its core content relates to how to reasonably allocate cache resources in a computer system or a distributed environment, improve data reading speed, and reduce access delay. Current data cache management is widely used in fields such as databases, files, cloud computing platforms and embedded devices. Traditional data cache management methods mainly use static or dynamic allocation strategies, which set fixed cache sizes or adjust according to access frequencies. Some methods combine machine learning or adaptive algorithms to optimize cache hit rates. Multi-layer data cache technology is an important part of cache management, which uses different levels of cache storage media, combines cache consistency maintenance and replacement strategies, and realizes efficient data access.

[0003] Among them, the multi-layer data cache dynamic management method refers to dynamically adjusting the data distribution strategy between cache levels in a multi-layer cache system by real-time monitoring of data access patterns, cache hit rates and storage medium states. This method covers cache space allocation, data migration decision and cache replacement rule setting, and specific methods include dynamic adjustment of cache space based on access frequency and time effectiveness, migration of cache content based on data priority or access characteristics, and development of cache eviction strategies based on statistical analysis or rule matching. This method adjusts data storage locations by monitoring cache usage and optimizes cache resource utilization based on specific logic.

[0004] The traditional technology mainly relies on static or dynamic allocation strategy based on access frequency, which causes the cache resource to be difficult to finely adjust in the face of complex access mode, and the fixed cache size mode is easy to cause storage space waste or cache overflow in the case of large change of data access mode, which affects the cache utilization. The adjustment mode based on access frequency is easy to be affected by data access burst, and cannot quickly respond to the real-time change of data access demand, causing the high-frequency access data to be unable to be preferentially stored in the cache area, affecting the data reading performance. Although the traditional multi-layer cache management utilizes different levels of storage media for data storage, it lacks fine-grained analysis of access path stability and data cross-competition situation, causing the cache hit rate to be reduced, the data migration strategy to be difficult to adapt to complex access scenarios, and the data elimination strategy to rely on fixed rules, which cannot be flexibly adjusted, causing high-value data to be eliminated too early, affecting the data reading efficiency, and the storage medium load balancing regulation and control capability is limited, which cannot dynamically adjust the data storage proportion according to the I / O task load, causing a storage bottleneck, reducing the overall throughput capacity, and affecting the storage stability in a high-load environment. SUMMARY

[0005] The purpose of the present application is to solve the problems existing in the prior art and to provide a multi-layer data cache dynamic management method and system.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme: a multi-layer data cache dynamic management method, comprising the following steps:

[0007] S1: based on the data access path, extracting the data access path level identifier, analyzing the path level depth, recording the access continuity, classifying the path as stable and unstable, and obtaining the path level distribution information;

[0008] S2: based on the path level distribution information, analyzing the data cache level storage proportion, storing the stable path data into the cache area, storing the unstable path data into the low-speed cache area, adjusting the storage area, and obtaining the cache space allocation status;

[0009] S3: according to the cache space allocation status, extracting the data block access record, detecting the access cross frequency, screening the cross frequent data block, analyzing the storage state of the cache area, adjusting the storage order and data elimination, and obtaining the cache area data priority;

[0010] S4: calling the cache area data priority, monitoring the storage medium I / O task, analyzing the task execution situation, adjusting the cache data storage proportion according to the task load, adjusting the storage medium of part of the data, and obtaining the storage medium load change trend;

[0011] S5: Based on the storage medium load change trend, analyze data matching condition, call cache allocation rule, analyze data throughput condition, and output data storage state in cache level.

[0012] As a further scheme of the present application, the path level distribution information includes path level identification, path level depth, path access continuity, access stability parameter classification result, and path stability category; the cache space allocation condition includes data cache level storage proportion, cache area storage data, unstable path storage data, and storage area adjustment result; the cache area data priority includes data block access record, access cross frequent data block, cache area storage state, data storage order, and competitive elimination data; the storage medium load change trend includes storage medium I / O task, task execution state, task load, and storage medium adjustment result; and the data storage state in cache level includes data matching condition, storage area allocation, cache allocation rule, and data throughput condition.

[0013] As a further scheme of the present application, the path level distribution information acquisition step specifically includes:

[0014] S111: Based on data access path, extract path level identification, analyze level depth, record path access continuity, identify level access times, and obtain path level distribution characteristic value;

[0015] S112: Call the path level distribution characteristic value, classify paths according to access stability parameters, screen stable paths and unstable paths, calculate stable path proportion, and obtain path stability ratio;

[0016] S113: Call the path stability ratio, combine each level path access trend, analyze access change amplitude of different level paths, and use the formula:

[0017] ;

[0018] Calculate path level access fluctuation value, and obtain path level distribution information;

[0019] Wherein, represents path level access fluctuation value, represents access times of the i-th level, represents access times of the previous level, represents total level number of the i-th level, represents stability ratio of the i-th level path, represents total number of path levels.

[0020] ​​​As a further scheme of the present application, the step of obtaining the cache space allocation state is specifically:

[0021] S211: Based on the path level distribution information, the data storage amount of the path level is calculated, the storage proportion of the path level exceeding the threshold value is screened by comparing the storage capacity, and the path level storage proportion screening result is obtained;

[0022] S212: The path level storage proportion screening result is called, and the data is stored in the differential cache area according to the data stability classification, and the cache area data storage deviation value is obtained by comparing the storage proportion of the high-speed and low-speed cache areas;

[0023] S213: Based on the cache area data storage deviation value, the adjustment amplitude of the access jump degree to the storage area is analyzed, and the formula is:

[0024] ;

[0025] The storage area adjustment amplitude is calculated, and the cache space allocation state is obtained;

[0026] Among them, represents the storage area adjustment amplitude, represents the low-speed cache area storage amount of the first level, represents the high-speed cache area storage amount of the first level, represents the access frequency of the first level, represents the reference access frequency of the first level, represents the storage adjustment coefficient of the first level.

[0027] As a further scheme of the present application, the step of obtaining the cache area data priority is specifically:

[0028] S311: According to the cache space allocation state, the data block access record is extracted, the data block access times and time interval are counted, the access cross situation is analyzed, and the access cross measurement value is obtained;

[0029] S312: The access cross measurement value is called, the cross data block is screened according to the cross times threshold value, the cache occupation ratio is calculated, the distribution trend is analyzed, and the competitive data block set is obtained;

[0030] S313: Based on the competitive data block set, the cache storage state and the data block access priority are analyzed, the storage order and the elimination strategy are adjusted according to the priority, and the formula is:

[0031] ;

[0032] obtaining the cache data priority;

[0033] wherein, representing the cache data priority, representing the data block access cross times, representing the data block the last access time interval, representing the data block cache occupancy ratio, representing the data block storage location index in the cache, representing the average of the data block storage location index in the cache.

[0034] As a further scheme of the present application, the obtaining step of the storage medium load change trend is specifically:

[0035] S411: Call the cache data priority, monitor the storage medium I / O task, calculate the I / O response time ratio of the storage medium, filter the storage medium whose task proportion exceeds the task threshold, and obtain the I / O task proportion distribution data;

[0036] S412: Based on the I / O task proportion distribution data, analyze the load change rate of the storage medium, adjust the cache data storage proportion of the load storage medium, and obtain the cache allocation adjustment coefficient;

[0037] S413: Adjust the data storage medium according to the cache allocation adjustment coefficient, analyze the load change trend after the data adjustment, and use the formula:

[0038] ;

[0039] obtain the storage medium load change trend;

[0040] wherein, representing the storage medium load change trend value, representing the load level of the jth storage medium after adjustment, representing the load level of the jth storage medium before adjustment, representing the original task load stability of the jth storage medium, representing the data access fluctuation amplitude of the jth storage medium, representing the total number of adjusted storage media.

[0041] As a further scheme of the present application, the obtaining step of the data storage state in the cache level is specifically:

[0042] S511: Based on the storage medium load change trend, the data block access frequency, the migration times, the storage time length are extracted, the data matching degree is analyzed, the matching situation is screened, and the data matching state classification value is obtained;

[0043] S512: The data matching state classification value is called, the data throughput of the storage hierarchy is analyzed, and the formula is adopted:

[0044] ;

[0045] The data throughput state characteristic value is calculated, the storage area adjustment demand is judged, and the storage area adjustment demand amount is obtained;

[0046] Among them, represents the data throughput state characteristic value, represents the current storage hierarchy throughput, represents the original storage hierarchy throughput, represents the throughput fluctuation amplitude, represents the throughput mean value, represents the data migration rate, represents the data migration influence factor;

[0047] S513: Based on the storage area adjustment demand amount, according to the cache allocation rule, the data storage situation after adjustment is analyzed, and the storage state of data in the cache hierarchy is output.

[0048] The multi-layer data cache dynamic management system is used to execute the above multi-layer data cache dynamic management method, and the system comprises:

[0049] The path access analysis module extracts the level identifier based on the data access path, identifies the level depth, records the path access sequence, screens the stable path and counts the distribution proportion, and obtains the path level distribution data;

[0050] The cache layer allocation module calculates the storage proportion of the high-speed cache area and the low-speed cache area based on the path level distribution data, stores the stable path set and the unstable path set, adjusts the jump path storage area, and obtains the cache space allocation state;

[0051] The data competition detection module extracts the data block access record based on the cache space allocation state, screens the data blocks accessed in a short time, calculates the cross times and classifies them, analyzes the storage proportion of the cross access data blocks in the high-speed cache area, adjusts the storage order and screens the cross access data blocks, and obtains the data competition evaluation result;

[0052] The cache data priority adjustment module calls a cache area data priority based on the data competition evaluation result, identifies a load amount of a storage medium I / O task, adjusts a storage proportion, selects partial data for storage medium migration, and obtains a storage medium load state;

[0053] The storage medium load balancing module screens low-frequency access data adjustment storage areas based on the storage medium load state, calls a cache allocation rule, calculates cache data throughput, adjusts a data storage level, and obtains a data storage state in a cache level.

[0054] Compared with the prior art, the application has the advantages and positive effects that:

[0055] In the application, through hierarchical analysis and stability classification based on a data access path, the data cache level storage mode is optimized, the precision allocation efficiency of cache resources is improved, the path data is stably stored in the cache area, the data access delay is reduced, the continuity of data access is improved, and the waste of cache resources caused by access path jump is reduced. The unstable path data is stored in the low-speed cache area, the cache space utilization is improved, the reading efficiency of high-frequency data is reduced due to cache pollution, the storage order is optimized based on the access cross-frequency and data competition condition, the effective hit rate of the high-speed cache area data is improved, the resource competition is reduced through the dynamic elimination strategy, the cache stability is improved, the storage medium I / O task load condition is monitored, the storage proportion of the cache data is dynamically adjusted according to the storage pressure, the loads of different media are balanced, and the overall data throughput capacity is improved. The data storage position is adjusted in combination with the data matching condition and the cache allocation rule, the data flow efficiency of the cache level is optimized, the collaborative ability of the multi-layer data cache is improved, the overall optimization strategy improves the adaptability of the cache resources, the cache level structure can flexibly respond to the change of the data access mode, the data storage stability and the access efficiency are improved, the data access delay is reduced, the data collaborative management ability between the cache levels is enhanced, and the data throughput capacity and the system load balancing are improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 It is a workflow schematic diagram of the application;

[0057] Figure 2 It is a path level distribution information acquisition flowchart in the application;

[0058] Figure 3 It is a cache space allocation condition acquisition flowchart in the application;

[0059] Figure 4 It is a cache area data priority acquisition flowchart in the application;

[0060] Figure 5 It is a storage medium load change trend acquisition flowchart in the application;

[0061] Figure 6 Flow chart for obtaining the storage state of data in the cache level in the application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the application more clear, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0063] In the description of the application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application. In addition, in the description of the application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0064] Example one: please refer to Figure 1 The application provides a technical solution: a multi-layer data cache dynamic management method, comprising the following steps:

[0065] S1: based on the data access path, extracting the path level identifier, analyzing the path level depth, recording the path access continuity, classifying the path according to the access stability parameter, screening the stable path and the unstable path, and obtaining the path level distribution information;

[0066] S2: based on the path level distribution information, analyzing the data cache level storage proportion, storing the stable path data into the high-speed cache area, storing the unstable path data into the low-speed cache area, adjusting the storage area according to the access jump degree, and obtaining the cache space allocation status;

[0067] S3: according to the cache space allocation status, extracting the data block access record, detecting the access cross situation in a short time, judging the data competition situation according to the cross frequency, screening the data block with frequent access cross, analyzing the storage state of the high-speed cache area, adjusting the data storage order, adjusting the eliminated data according to the competition situation, and obtaining the cache area data priority;

[0068] S4: calling the cache area data priority, monitoring the storage medium I / O task, analyzing the task execution situation, adjusting the cache data storage proportion according to the task load, adjusting the storage medium of part of the data, and obtaining the storage medium load change trend;

[0069] S5: Based on the storage medium load change trend, analyze the data matching situation and adjust the storage area, call the cache allocation rule, analyze the data throughput situation, and output the data storage state in the cache level.

[0070] The path level distribution information includes path level identification, path level depth, path access continuity, access stability parameter classification result, path stability category, cache space allocation status includes data cache level storage proportion, cache area storage data, unstable path storage data, storage area adjustment result, cache area data priority includes data block access record, access cross frequent data block, cache area storage state, data storage order, competitive elimination data, storage medium load change trend includes storage medium I / O task, task execution state, task load, storage medium adjustment result, data storage state in cache level includes data matching situation, storage area allocation, cache allocation rule, data throughput situation.

[0071] Please refer to Figure 2 The acquisition steps of the path level distribution information are as follows:

[0072] S111: Based on the data access path, extract the path level identification, analyze the level depth, record the path access continuity, identify the level access times, and obtain the path level distribution characteristic value;

[0073] The access path of each file is extracted through the access record of the log file, and then the path is identified and classified, and the access frequency of each type of path in different time periods is counted. This process helps managers understand data access patterns and storage efficiency. For example, in the server of an e-commerce website, by analyzing the access data of the storage path of product pictures, the server resources can be adjusted and the data calling speed can be optimized. This method directly affects the structure optimization of the file system through specific path access records, and finally obtains the path level distribution characteristic value.

[0074] S112: Call the path level distribution characteristic value, classify the paths according to the access stability parameters, filter the stable paths and unstable paths, calculate the stable path proportion, and obtain the path stability ratio;

[0075] In cloud service providers, by monitoring and classifying different customers' data request paths, adjusting resource allocation according to the stability and volatility of the paths, including collecting access logs within a certain time, calculating the access stability of each path, and allocating more cache resources to paths that show high stability. This analysis helps improve the response speed and resource utilization efficiency of the service, and obtains the path stability ratio.

[0076] S113: Call the path stability ratio, combine the access trend of each level path, analyze the access change amplitude of the differentiated level path, and use the formula:

[0077] ;

[0078] Computing the path level access fluctuation value, obtaining path level distribution information;

[0079] Wherein, represents the path level access fluctuation value, represents the access frequency of the first level, represents the access frequency of the last level, represents the access frequency of the last level, represents the total number of levels of the first level, represents the stability ratio of the first level path, represents the total number of levels of the first level; In big data analysis, the calculation of path level access fluctuation value is crucial for predicting data traffic and resource demand. Assuming that in a cloud storage system, the file access situation of different levels needs to be analyzed to optimize server load, the access frequency of each level is extracted from log records, for example, in a certain time period:

[0080] The access frequency of the first level path ( ) is

[0081] ; The access frequency of the second level path ( ) is

[0082] ; The access frequency of the third level path ( ) is

[0083] ; The access frequency of the fourth level path ( ) is

[0084] ; The access data is derived from log statistics, and the access fluctuation between levels is calculated;

[0085] Wherein, the stability ratio of each level is

[0086] Calculated by the access frequency in the past 30 days, for example: 、 、 、 ; Substitute the data to calculate:

[0087]

[0088] ;

[0089] ; ​​

[0090] ;

[0091] ;

[0092] Calculate the result path level access fluctuation value This value indicates that the traffic fluctuation between access levels is large, which means that the access changes at different levels are significant. This data can be further used to predict future data access trends and, combined with historical data, determine whether it is necessary to optimize the cache strategy or adjust the server resource configuration, and ultimately obtain the path level distribution information.

[0093] See also Figure 3 ,The specific steps for obtaining the cache space allocation status are:

[0094] S211: Calculating the data storage capacity of the path level based on the path level distribution information, comparing the storage capacity, screening the path levels whose storage ratio exceeds a threshold, and obtaining a path level storage ratio screening result;

[0095] First, we monitor access logs from different data centers, collecting data access frequency and path information. Using this data, we calculate the data storage capacity for each path level. For example, in Data Center A, we monitor that path P1 is accessed 300 times in a week, while path P2 is accessed 50 times. Based on the number of accesses, we estimate the data storage requirements for P1 and P2. We then compare the storage capacity of each path level to determine which levels exceed a predetermined threshold. This threshold is set based on historical data and storage policies, such as 25%. If the storage share of level L1 reaches 30%, it is considered to have exceeded the threshold. Through detailed calculations and comparisons, we identify the path levels that require special attention, resulting in the path level storage share screening results.

[0096] S212: Calling the path-level storage ratio screening result, classifying and storing the data into differentiated cache areas according to data stability, comparing the storage ratios of the high-speed and low-speed cache areas, and obtaining a cache area data storage deviation value;

[0097] Data is categorized based on stability: stable data (e.g., frequently accessed data with minimal changes) is stored in the high-speed cache, while unstable data (e.g., infrequently accessed or frequently changing data) is stored in the low-speed cache. For example, if data at path level L2 changes monthly, while data at L3 changes weekly, L2 data is categorized as stable and L3 as unstable. Based on this categorization, the storage percentages of each type of data in the high-speed and low-speed caches are compared to determine if there is a deviation. A deviation threshold is set at 10%. If the deviation between the high-speed and low-speed cache percentages of L3 reaches 15%, adjustments are made, ultimately resulting in a cache data storage deviation value.

[0098] S213: Based on the cache data storage deviation value, analyze the adjustment range of the storage area according to the access jump degree, using the formula:

[0099] ;

[0100] Calculate the storage area adjustment range, and obtain the cache space allocation status;

[0101] Wherein, represents the storage area adjustment range, represents the low-speed cache area storage of the first level, represents the high-speed cache area storage of the first level, represents the access frequency of the first level, represents the reference access frequency of the first level, represents the storage adjustment coefficient of the first level;

[0102] Further calculate the adjustment range of the storage area according to the access jump degree for the cache data storage deviation value, , , , , respectively represent the low-speed cache area storage, high-speed cache area storage, actual access frequency, reference access frequency, and storage adjustment coefficient of the first level;

[0103] Set specific parameter values, assume that there are three path levels ( ), and the monitoring data is as follows:

[0104] Level 1: GB, GB, times / week, times / week, ;

[0105] Level 2: GB, GB, times / week, times / week, ;

[0106] Level 3: GB, GB, times / week, times / week, ;

[0107] Substitute the above values into the formula to calculate:

[0108] Calculate the sum of the storage difference:

[0109]

[0110] Calculate the square root of the sum of the square of the access frequency deviation:

[0111]

[0112] Calculate the sum of the storage adjustment coefficient:

[0113] Calculate the final adjustment amplitude:

[0114] The calculation result shows that the adjustment amplitude of the storage area is 0.807, which means that the storage ratio of high-speed and low-speed cache needs to be adjusted accordingly to reduce the mismatch between access frequency and storage deviation under the current path level distribution. Based on this adjustment amplitude, perform storage reallocation to migrate part of the low-speed cache data to the high-speed cache or vice versa, and finally obtain the cache space allocation status.

[0115] Please refer to Figure 4 , the data priority acquisition step is specifically:

[0116] S311: According to the cache space allocation status, extract the data block access record, count the data block access times and time interval, analyze the access cross situation, and obtain the access cross metric value;

[0117] Monitor the access of each data block in the cache. Assuming that in a cloud storage service, according to the log file, it is analyzed which data blocks are frequently accessed, the information is used for subsequent optimization operation, the access times and time interval of each data block are counted, and the statistics are based on real-time data stream analysis. For example, in a certain database, the access times of data block A within one hour are 100 times, and the average access interval is 5 minutes. Calculate the access cross situation, which refers to the situation that different data blocks are accessed by multiple users or programs within the same time period. By setting a cross access threshold, for example, if two data blocks are accessed more than 50 times within 10 minutes, it is considered that there is high cross access. Finally, the access cross metric value is obtained, which is obtained by comparison and analysis, for example, data blocks A and B are accessed 120 times within 10 minutes, so the access cross metric value is high.

[0118] S312: Call the access cross metric value, filter the cross data blocks according to the cross times threshold, calculate the cache occupancy ratio, analyze the distribution trend, and obtain the competitive data block set;

[0119] ​​​​Retrieving the cross metric value calculated in the previous step from the database, for example, extracting the access cross metric value of data blocks A and B from the cache management software, filtering cross data blocks according to the cross frequency threshold, filtering is based on the preset cross access threshold, for example, if the cross frequency exceeds 100 times, it is considered to be a high cross data block, calculating its cache occupancy ratio, that is, the ratio of the cache size occupied by the data block to the total cache size, for example, data blocks A and B occupy 30% of the cache, analyzing the distribution trend, determining whether the distribution of data blocks in the cache is concentrated or dispersed through statistical analysis, for example, through a chart showing that data blocks A and B are mainly concentrated in the first 50% area of the cache, obtaining a competitive data block set, which reflects the data blocks that need to be managed preferentially, for example, data blocks A and B are marked as high-competition data blocks due to frequent access and large cache occupation.

[0120] S313: Based on the competitive data block set, analyze the cache storage state and data block access priority, adjust the storage order and eviction strategy according to the priority, use the formula:

[0121] ;

[0122] Obtain the cache area data priority;

[0123] Wherein, represents the cache area data priority, represents the access cross frequency of data block , represents the latest access time interval of data block , represents the cache occupancy ratio of data block , represents the storage location index of data block in the cache, represents the mean value of the storage location index of data blocks in the cache area;

[0124] First, the data blocks marked as high competition are extracted from the log or cache management module, for example, in the cache of a certain cloud computing platform, data block A and data block B have been identified as high competition data blocks by the system due to high-frequency access by multiple users, their relevant parameters are read from the database or cache monitoring, including access frequency, access interval, cache occupancy ratio and storage location index, the cache storage state is analyzed, the analysis process involves calculating the utilization rate of the current cache storage, and checking the distribution of high competition data blocks in the cache, for example, the total cache capacity is 200GB, and data blocks A and B occupy 15GB and 10GB respectively, accounting for 12.5% of the cache space, at the same time, the distribution of all data blocks is counted, and whether there is a local storage hotspot is evaluated, the data block access priority is calculated, this process needs to be based on the cross access times of the data block, the latest access time interval, the cache occupancy ratio and its storage location index;

[0125] The specific calculation of data block A is as follows, wherein the cross access times of data block A , the latest access time interval minutes, the cache occupancy ratio (15GB / 200GB), the storage location index , the mean value of the cache block storage location index , and the formula is as follows:

[0126] ;

[0127] Similarly, if data block B is calculated, its parameters are set as , minutes, (10GB / 200GB), , and the same calculation is as follows:

[0128] ;

[0129] According to the cache priority adjustment storage order and elimination strategy, for example, in cache management, data block B with higher cache priority is moved to a higher level cache, while data block A with lower priority is moved to a slower storage area or considered for elimination, and the cache data priority is obtained, which shows that data block B will be stored first due to its high access priority, while data block A will be replaced or reduced in storage level due to its low priority.

[0130] Please refer to Figure 5 , the steps for obtaining the storage medium load change trend are as follows:

[0131] S411: Call the cache data priority, monitor the storage medium I / O task, calculate the I / O response time ratio of the storage medium, screen the storage medium whose task proportion exceeds the task threshold, and obtain the I / O task proportion distribution data;

[0132] First, set different data priorities, for example, in actual application scenarios, such as financial transactions, the priority of transaction data is the highest, followed by user verification data, calculate the proportion of each data type in the total task, and set the task threshold, assuming 30%, when the task proportion of any data type exceeds this threshold, automatically start the alarm mechanism, adjust the priority of high proportion task, then monitor the I / O response of the storage medium, if the I / O response time is found to be longer, the data is redistributed, the efficiency of the whole storage medium is optimized, and finally the I / O task proportion distribution data is obtained, for example, in a certain financial transaction, it is found that the I / O response time of transaction data is 0.01 seconds and the I / O response time of user verification data is 0.02 seconds.

[0133] S412: Based on the I / O task proportion distribution data, analyze the load change rate of the storage medium, adjust the cache data storage proportion of the load storage medium, and obtain the cache allocation adjustment coefficient;

[0134] The load change rate of each storage medium is analyzed, the data access frequency and data read-write times are monitored in real time, the load change of each storage medium is calculated, if the load change rate of a certain storage medium exceeds the safety threshold, the data storage strategy of the storage medium is automatically adjusted, for example, the data write to the storage medium is reduced, the data backup operation of the storage medium is increased, and the cache allocation adjustment coefficient is obtained, such as adjustment coefficient 1.5, which means the data write rate needs to be increased by 50% to reduce the pressure of the high load storage medium.

[0135] S413: Adjust the data storage medium according to the cache allocation adjustment coefficient, analyze the load change trend of the adjusted data, and use the formula:

[0136] ;

[0137] Obtain the load change trend of the storage medium;

[0138] Among them, represents the load change trend value of the storage medium, represents the load level of the jth storage medium after adjustment, represents the load level of the jth storage medium before adjustment, represents the original task load stability of the jth storage medium, represents the data access fluctuation amplitude of the jth storage medium, represents the total number of adjusted storage media;

[0139] Adjust the data storage medium distribution, and recalculate the load change trend of the storage medium. In practical applications, for example, a data storage contains 5 storage media, and the initial load levels of each storage medium are different. After adjusting the storage strategy, the load levels of each storage medium also change. Assuming that the initial load levels are as follows:

[0140] , , , , ;

[0141] After adjustment, the load levels of the storage media change to:

[0142] , , , , ;

[0143] The historical task load stability and data access fluctuation amplitude are set as follows:

[0144] , , , , ;

[0145] , , , , ;

[0146] Substitute the formula to calculate the load change trend value of each storage medium step by step:

[0147] ;

[0148] ;

[0149] ;

[0150] ;

[0151] ;

[0152] Add the load change trend values of each storage medium to obtain the total load change trend value of the storage medium:

[0153] ;

[0154] The load adjustment range of the storage medium as a whole is low, which indicates that the data distribution strategy still maintains a certain balance after adjustment and does not cause a dramatic change in the load of some storage media. The load optimization threshold value can be set according to the value. For example, if the total load change trend value exceeds 0.5, it is necessary to further adjust the data storage strategy and redistribute part of the data to the storage media with lower load to establish the storage medium load change trend.

[0155] Please refer to Figure 6 The obtaining step of the storage state of data in the cache hierarchy is specifically:

[0156] S511: Based on the storage medium load change trend, the data block access frequency, migration times, and storage duration are extracted, the data matching degree is analyzed, the matching condition is filtered, and a data matching state classification value is obtained.

[0157] First, the access frequency of each data block is monitored. For example, in a cloud storage environment, the access frequency of each data block per hour is obtained through a log analysis tool. The access data is then processed through a series of numerical integration, including averaging and comparing historical data, to determine the activity and frequency of use of the data block. The migration times of the data block between different storage hierarchies are calculated, which involves tracking the storage operation log, such as the number of records of the data block moving from the cache layer to the low-speed storage layer. The storage duration of the data block, i.e., the total time of the data block since its creation, needs to be calculated. The timestamp information of the data block needs to be extracted from the metadata management. According to the set storage adaptation threshold value, such as a data block with an access frequency of less than 10 times / day being marked as low-frequency access, the matching condition of the data block is filtered and classified, and finally the data matching state classification value is obtained. The value reflects the storage efficiency and access mode of the data, which is calculated according to the actual storage usage.

[0158] S512: The data matching state classification value is called to analyze the data throughput of the storage hierarchy, and the formula is:

[0159] ;

[0160] The data throughput state feature value is calculated, the storage area adjustment demand is judged, and the storage area adjustment demand quantity is obtained.

[0161] Among them, represents the data throughput state feature value, represents the current storage hierarchy throughput, represents the original storage hierarchy throughput, represents the throughput fluctuation range, represents the throughput mean value, represents the data migration rate, represents the data migration influence factor;

[0162] The data input and output rate of each storage level is monitored, and the read and write speed of the data is measured. For example, in a distributed storage, the storage levels include cache (such as SSD), standard storage (such as HDD), and archive storage (such as tape storage). First, the data throughput of each storage level is calculated. Assuming that the current throughput of the SSD level is MB / s, and the historical throughput of the last period is MB / s, the difference between the two can be used to evaluate the trend of the throughput. The mean and fluctuation range of the throughput are calculated. Assuming that the mean of the throughput data of this level in the past 24 hours is MB / s, and the fluctuation range is MB / s, it can be used to determine whether the current throughput is in a stable state. The data migration rate is calculated. This value can be obtained by counting the total number of data blocks migrated between different storage levels. Assuming that the current monitored data migration rate is , that is, 12% of the data blocks are migrated to different storage levels per hour. In calculating the data throughput state characteristic value;

[0163] where the data migration influence factor value depends on the type of storage medium migrated and the degree of influence of migration. Assuming that the current is set to

[0164] ;

[0165] ;

[0166] ;

[0167] The calculated data throughput state characteristic value is used to determine whether the storage area needs to be adjusted. If the set storage adjustment threshold is 60, the current calculation value has exceeded this threshold, indicating that the throughput of the storage area needs to be optimized. The storage area adjustment requirement calculation result is to adjust. The result can be used to further adjust the allocation strategy of data storage, such as migrating part of the data from the SSD level to the HDD level, or adding additional cache to the SSD level.

[0168] S513: Based on the storage area adjustment requirement, the data storage situation after adjustment is analyzed according to the cache allocation rule, and the storage state of the data in the cache level is output.

[0169] According to the data throughput state characteristic value, the position of data between different storage levels is adjusted, if the throughput state characteristic value of a level shows data congestion, part of the data is migrated to a lower or higher speed storage level to balance the load, the adjustment process involves the application of a storage resource allocation algorithm, such as based on a preset threshold and throughput model, dynamically adjusting the storage location of data blocks to improve the overall performance and response speed of the storage, outputting the adjusted data storage situation, and updating the storage metadata and user interface to ensure that users and administrators can clearly see the adjusted data distribution, and outputting the storage state of data in the cache level.

[0170] The multi-layer data cache dynamic management system is used to execute the above-mentioned multi-layer data cache dynamic management method, and the system comprises:

[0171] The path access analysis module extracts the level identifier based on the data access path, identifies the level depth, records the path access sequence, filters the stable path and counts the distribution proportion, and obtains the path level distribution data;

[0172] The cache layering allocation module calculates the storage proportion of the high-speed cache area and the low-speed cache area based on the path level distribution data, stores the stable path set and the unstable path set, adjusts the storage area of the jump path, and obtains the cache space allocation state;

[0173] The data competition detection module extracts the data block access record based on the cache space allocation state, filters the data blocks that are cross-accessed in a short time, calculates the cross frequency and classifies it, analyzes the storage proportion of the cross-access data blocks in the high-speed cache area, adjusts the storage order and filters the cross-access data blocks, and obtains the data competition evaluation result;

[0174] The cache data priority adjustment module calls the cache area data priority based on the data competition evaluation result, identifies the load of the storage medium I / O task, adjusts the storage proportion, selects part of the data for storage medium migration, and obtains the storage medium load state;

[0175] The storage medium load balancing module filters the low-frequency access data to adjust the storage area based on the storage medium load state, calls the cache allocation rule, calculates the cache data throughput, adjusts the data storage level, and obtains the storage state of data in the cache level.

[0176] The above is only a preferred embodiment of the present application, and does not limit the form of the present application, any skilled person in the art can use the above disclosed technical content to make changes or modifications as equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments within the technical solution content of the present application still belongs to the protection scope of the technical solution of the present application.

Claims

1. A multi-layer data cache dynamic management method, characterized in that: The method comprises the following steps: S1: based on the data access path, extracting the data access path level identification, analyzing the path level depth, recording the access continuity, classifying the path as stable and unstable, and obtaining the path level distribution information; S2: based on the path level distribution information, analyzing the data cache level storage proportion, storing the stable path data into the cache area, storing the unstable path data into the low-speed cache area, adjusting the storage area, and obtaining the cache space allocation status; S3: according to the cache space allocation status, extracting the data block access record, detecting the access cross frequency, screening the cross frequent data block, analyzing the cache area storage state, adjusting the storage order and data elimination, and obtaining the cache area data priority; The cache area data priority obtaining step specifically comprises: S311: according to the cache space allocation status, extracting the data block access record, counting the data block access times and time interval, analyzing the access cross situation, and obtaining the access cross metric value; S312: calling the access cross metric value, screening the high cross data block according to the cross times threshold, calculating the cache occupation proportion, analyzing the distribution trend, determining whether the high cross data block is concentrated or dispersed in the cache through statistical analysis, obtaining the competitive data block set, and the set reflects that the data block is frequently accessed and occupies the cache, and is marked as a high competition data block; S313: based on the competitive data block set, analyzing the cache storage state and the data block access priority, adjusting the storage order and elimination strategy according to the data block access priority, and finally obtaining the cache area data priority; The calculation formula of the data block access priority is: ; wherein, a representative data block access priority, a representative data block a number of access intersections, a representative data block a time interval of the last access, a representative data block a cache occupancy ratio, a representative data block a storage location index in the cache, a mean value of the data block storage location index in the cache. The cache storage state comprises calculating the utilization rate of the current cache storage, and checking the distribution of the high competition data block in the cache; S4: calling the cache area data priority, monitoring the storage medium I / O task, analyzing the task execution situation, adjusting the cache data storage proportion according to the task load, adjusting the storage medium of part of the data, and obtaining the storage medium load change trend; S5: based on the storage medium load change trend, analyzing the data matching situation, analyzing the data throughput situation, calling the cache allocation rule, and outputting the storage state of the adjusted data in the cache level.

2. The method of claim 1, wherein, The path level distribution information comprises path level identification, path level depth, path access continuity, access stability parameter classification result, and path stability category; the cache space allocation status comprises data cache level storage proportion, high-speed cache area storage data, unstable path storage data, and storage area adjustment result; the cache area data priority comprises data block access record, access cross frequent data block, high-speed cache area storage state, data storage order, and competitive elimination data; the storage medium load change trend comprises storage medium I / O task, task execution state, task load, and storage medium adjustment result; and the storage state of the data in the cache level comprises data matching situation, storage area allocation, cache allocation rule, and data throughput situation.

3. The method of claim 1, wherein, The path level distribution information obtaining step specifically comprises: S111: Based on the data access path, extract the path level identifier, analyze the level depth, record the path access continuity, identify the level access times, and obtain the path level distribution characteristic value; S112: Call the path level distribution characteristic value, classify the path according to the access stability parameter, screen the stable path and the unstable path, calculate the stable path proportion, and obtain the path stability ratio; S113: call the path stability ratio, combine the access trend of each level path, analyze the access change amplitude of the differentiated level path, use the formula: , calculate the path level access fluctuation value, and obtain the path level distribution information; wherein, represents a path level access fluctuation value, represents a number of accesses of a first level, represents a number of accesses of a previous level, represents a number of accesses of a first level, represents a total number of levels, represents a stability ratio of a first level path, represents a total number of path levels.

4. The method of claim 3, wherein, The acquisition step of the cache space allocation state is specifically: S211: Based on the path level distribution information, calculate the data storage amount of the path level, compare the storage capacity of each path level, screen the path level with storage proportion exceeding the threshold, and obtain the path level storage proportion screening result; S212: Call the path level storage proportion screening result, store to the differential cache area according to the data stability, compare the storage proportion of the high-speed and low-speed cache areas, and obtain the cache area data storage deviation value; S213: Based on the bias value of the cache area data storage, analyze the adjustment range of the storage area according to the access jump degree, and use the formula: , calculate the storage area adjustment range, and obtain the cache space allocation status; wherein, represents a storage area adjustment range, represents a first represents a low cache area storage amount of a hierarchy, represents a first represents a cache area storage amount of a hierarchy, represents a first represents an access frequency of a hierarchy, represents a first represents a reference access frequency of a hierarchy, represents a first represents a storage adjustment coefficient of a hierarchy.

5. The method of claim 1, wherein, The acquisition step of the storage medium load change trend is specifically: S411: Call the cache area data priority, monitor the storage medium I / O task, calculate the I / O response time ratio of the storage medium, screen the storage medium with task proportion exceeding the task threshold, and obtain the I / O task proportion distribution data; S412: Based on the I / O task proportion distribution data, analyze the load change rate of the storage medium, adjust the cache data storage proportion of the load storage medium, and obtain the cache allocation adjustment coefficient; S413: adjusting the data storage medium according to the cache allocation adjustment coefficient, analyzing the load change trend of the adjusted data, and using the formula: to obtain the load change trend of the storage medium; wherein, represents a storage medium load change trend value, represents an adjusted load level of the jth storage medium, represents a load level of the jth storage medium before adjustment, represents an original task load stability of the jth storage medium, represents a data access fluctuation amplitude of the jth storage medium, represents a total number of adjusted storage media.

6. The method of claim 5, wherein, The acquisition step of the data storage state in the cache level is specifically: S511: Based on the storage medium load change trend, extract the data block access frequency, migration times, and storage time length, analyze the data matching degree, screen the matching situation, and obtain the data matching state classification value; S512: call the data matching state classification value, analyze the data throughput of the storage level, and use the formula: , calculate the data throughput state characteristic value, judge the storage area adjustment demand, and obtain the storage area adjustment demand quantity; wherein, representing a data throughput state characteristic value, representing a current storage tier throughput, representing an original storage tier throughput, representing a throughput fluctuation amplitude, representing a throughput mean value, representing a data migration rate, representing a data migration impact factor; S513: Based on the storage area adjustment demand, according to the cache allocation rule, analyze the adjusted data storage situation, and output the storage state of the adjusted data in the cache level.

7. A multi-tier data cache dynamic management system, characterized by, The multi-layer data cache dynamic management method according to any one of claims 1-6, the system comprises: The path access analysis module extracts the level identifier based on the data access path, identifies the level depth, records the path access sequence, screens the stable path and calculates the distribution proportion, and obtains the path level distribution data; The cache layer allocation module calculates the storage proportion of the high-speed cache area and the low-speed cache area based on the path level distribution data, stores the stable path set and the unstable path set, adjusts the storage area of the jump path, and obtains the cache space allocation state; The data competition detection module extracts the data block access record based on the cache space allocation state, screens the data blocks accessed in a short time, calculates the cross times and classifies them, analyzes the storage proportion of the cross access data blocks in the high-speed cache area, adjusts the storage order and screens the cross access data blocks, and obtains the data competition evaluation result; The cache data priority adjustment module calls the cache area data priority based on the data competition evaluation result, identifies the load of the storage medium I / O task, adjusts the storage proportion, selects part of the data for storage medium migration, and obtains the storage medium load state; The storage medium load balancing module screens low-frequency access data based on the storage medium load state, calculates a cache data throughput, adjusts a data storage level, calls a cache allocation rule, and outputs a storage state of adjusted data in a cache level.

Images