An LSM engine data caching optimization method

Abstract

This invention relates to the field of database cache management technology, specifically to an LSM engine data cache optimization method, comprising the following steps: based on the cache area data of the LSM storage engine, recording the access frequency of implementation rules, statistically analyzing the distribution density of implementation rules, organizing the data into a priority sequence, and generating a set of implementation rule access distribution information. In this invention, by recording the access frequency and distribution location of implementation rules, the distribution density of implementation rules is accurately grasped, and these are grouped and organized into a priority sequence to improve cache data reading efficiency. By adjusting and optimizing the data storage layout, high-frequency data retrieval latency is reduced. Adjusting the cache area location, calculating capacity differences, and repartitioning solves the problem of uneven storage allocation and improves resource utilization. By comparing high-frequency implementation rules to optimize data distribution, dynamically adjusting the matching of new implementation rules, the hit rate and response speed are enhanced, system load is reduced, and storage performance and resource allocation efficiency are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of database cache management technology, and in particular to an LSM engine data cache optimization method. Background Technology

[0002] Database caching management technology focuses on improving the access speed and processing efficiency of database systems. This involves using caching mechanisms to store temporary data, thereby reducing the need for direct disk access. Database caching can be memory-based or disk-based, employing different caching strategies. By intelligently caching data such as query results and frequently used data pages, database caching management can significantly improve data retrieval speed, reduce database server load, and optimize overall system performance. In practical applications, caching strategy design considers data access frequency, storage costs, and cache invalidation strategies to achieve optimal performance improvement.

[0003] Among them, the LSM engine data caching optimization method involves using implementation details caching to enhance performance in a specific type of database storage engine, LSM (Log-Structured Merge-tree). The LSM storage engine optimizes operation performance by recording data change operations in memory and periodically merging the changes into the data structure on disk. Applying implementation details caching in this structure is mainly to optimize the efficiency of read operations. By caching commonly used data implementation details, query requests for data can be responded to quickly. This method is particularly suitable for scenarios with highly asynchronous read and write requests, such as large-scale web applications and real-time data processing, and can significantly improve the overall responsiveness of data access speed.

[0004] While existing technologies in database cache management can reduce disk access frequency through caching strategies, they are not very adaptable to dynamic changes in cached data and struggle to effectively address uneven data access distribution. In terms of differentiated management across storage levels, current technologies rely solely on static or preset strategies, failing to efficiently adjust detailed rules for frequently changing high-frequency data, resulting in low cache hit rates. Furthermore, existing technologies lack fine-grained adjustment mechanisms for cache capacity allocation, making it difficult to achieve reasonable resource allocation across different storage levels, leading to wasted storage resources in low-speed cache areas. They also have shortcomings in analyzing read / write efficiency and optimizing latency, especially in high-concurrency read / write scenarios. Existing solutions cannot adequately identify and adjust the priority distribution of cache resources, leading to decreased method responsiveness. For example, in real-time data processing or large-scale web application scenarios, the imbalance of read / write requests amplifies the problem, increases system load, and reduces overall performance. Summary of the Invention

[0005] To address the shortcomings of existing database cache management technologies, which, while reducing disk access frequency through caching strategies, are poorly adaptable to dynamic changes in cached data and struggle to effectively handle uneven data access distribution, current technologies rely solely on static or preset strategies for differentiated management across storage tiers. This fails to efficiently adjust detailed rules for frequently changing, high-frequency data, resulting in low cache hit rates. Furthermore, existing technologies lack fine-grained adjustment mechanisms for cache capacity allocation, hindering the rational allocation of resources across different storage tiers and leading to wasted storage resources in low-speed cache areas. They also fall short in analyzing read / write efficiency and optimizing latency, particularly in high-concurrency read / write scenarios where existing solutions cannot adequately identify and adjust the priority distribution of cache resources, leading to decreased responsiveness. For example, in real-time data processing or large-scale web applications, the imbalance of read / write requests amplifies the problem, increasing system load and reducing overall performance. This invention provides an LSM engine data caching optimization method. The technical solution is as follows:

[0006] On the one hand, an LSM engine data caching optimization method is provided, which includes:

[0007] S1: Based on the LSM storage engine, cache area data records the number of accesses to the implementation details, calculates the distribution density of the implementation details, organizes the data into a priority sequence, and generates a set of access distribution information for the implementation details.

[0008] S2: Based on the implementation details access distribution information set, sort the grouped data, analyze the hierarchical span, adjust the priority interval implementation details group order, optimize the implementation details arrangement order, and generate the implementation details cache priority distribution result;

[0009] S3: Based on the cache priority distribution results of the implementation details, adjust the data position of the cache area implementation details, repartition the remaining capacity implementation details, adjust the partition capacity ratio, and generate the cache partition adjustment configuration results;

[0010] S4: Based on the cache partition adjustment configuration results, record read and write operations within the partition, count the read and write frequency, analyze the read and write latency, integrate latency data and hit records, and generate a partition read and write efficiency data set.

[0011] S5: Based on the partition read / write efficiency data set, compare the high-frequency implementation details read / write frequency, rearrange the partition data order, adjust the implementation details arrangement order, iteratively allocate partition capacity, and generate an optimized cache partition structure.

[0012] S6: Based on the optimized cache partition structure, record the implementation details of new access requests, statistically analyze partition matching data, integrate the new implementation details information to the partition boundary, and generate a dynamic adjustment scheme for the implementation details cache.

[0013] As a further aspect of the present invention, the implementation details access distribution information set includes access frequency statistics of implementation details, storage level distribution location, and implementation details distribution density data; the implementation details cache priority distribution results include priority interval delineation, implementation details group arrangement order, and continuous sequence distribution optimization structure; the cache partition adjustment configuration results include implementation details cache area distribution, partition capacity allocation ratio, and low-speed cache area remaining capacity allocation; the partition read / write efficiency data set includes partition read / write frequency data, partition read / write latency statistics, and cache hit and migration frequency ratio; the optimized cache partition structure includes partition boundary data arrangement, implementation details partition capacity ratio delineation, and partition optimized data order; and the implementation details cache dynamic adjustment scheme includes new access implementation details matching partition data, new implementation details boundary information integration, and partition implementation details capacity adjustment records.

[0014] As a further aspect of the present invention, the steps of recording the access frequency of implementation details based on the cache area data of the LSM storage engine, calculating the distribution density of implementation details, organizing the data into a priority sequence, and generating a set of implementation detail access distribution information are as follows:

[0015] S101: Based on the LSM storage engine, cache area data captures data implementation details, corresponding access counts and storage level information from the LSM storage engine, traverses the storage records and extracts key information for each implementation detail, including implementation detail name, access count, and level identifier, to obtain the implementation detail access record table.

[0016] S102: Filter data from the implementation details access record table, compare the level and access frequency of each implementation details, count the access frequency of each level, and classify the data implementation details by analyzing the differences in access frequency between levels to obtain a level access difference table.

[0017] S103: Use the hierarchical access difference table to sort the data, and classify the implementation rules by setting a threshold for distribution density. Implementation rules with high distribution density are given priority, thus obtaining a set of implementation rule access distribution information.

[0018] As a further aspect of the present invention, the steps of sorting the grouped data, analyzing the hierarchical span, adjusting the priority interval implementation rule group order, optimizing the implementation rule arrangement order, and generating the implementation rule cache priority distribution result based on the implementation rule access distribution information set are as follows:

[0019] S201: Based on the access distribution information set of the implementation details, compare the group data item by item according to the number of accesses, extract the differences in access frequency between groups, sort out the increasing trend of the number of accesses for each group, and generate an access number ranking table by accumulating the number of accesses;

[0020] S202: Based on the access frequency sorting table, analyze the hierarchical span of each group item by item. By comparing the cumulative access frequency within the group with the hierarchical span, extract the hierarchical range of the interval and generate a priority interval division table according to the interval hierarchical priority.

[0021] S203: Based on the priority interval partitioning table, the implementation rules combination with high frequency within the interval and the implementation rules group with low level span are rearranged, the continuous sequence of the implementation rules group is adjusted, and the priority arrangement is optimized by the internal order of the interval to generate the implementation rules cache priority distribution result.

[0022] As a further aspect of the present invention, the steps of adjusting the data location of the cache region implementation details based on the cache priority distribution result of the implementation details, repartitioning the remaining capacity implementation details, adjusting the partition capacity ratio, and generating the cache partition adjustment configuration result are as follows:

[0023] S301: Based on the implementation details cache priority distribution results, extract the storage location and associated capacity data of high priority implementation details, compare the capacity of implementation details with the current distribution of cache areas in groups, count the storage requirements corresponding to each group of implementation details, divide the distribution range of priority data, and obtain priority implementation details grouping and capacity labeling data.

[0024] S302: Based on the priority implementation details grouping and capacity labeling data, calculate the difference between the capacity occupied by high priority implementation details data and the current available capacity of the buffer, compare the capacity of low priority implementation details corresponding to the difference distribution, extract the distribution data that does not meet the capacity requirements, and merge and classify the distribution data to obtain the buffer capacity difference and the remaining distribution data.

[0025] S303: Based on the cache capacity difference and remaining distribution data, identify the adaptation parameters between the remaining distributed capacity and the low-speed cache, re-divide the remaining capacity units, adjust the capacity ratio of the unit partitions, reorganize the storage of the allocated areas, and establish the cache partition adjustment configuration results.

[0026] As a further aspect of the present invention, the steps of adjusting the cache partition configuration result, recording read and write operations within the partition, statistically analyzing read and write frequencies, analyzing read and write latency, integrating latency data with hit records, and generating a partition read and write efficiency dataset are as follows:

[0027] S401: Based on the cache partition adjustment configuration result, read the read and write records in the partition, analyze the operation timestamp and data source, calculate the read and write operation time, perform read and write frequency classification statistics, compare the cache hit count, and obtain partition read and write and hit statistics.

[0028] S402: Based on the partition read / write and hit statistics, extract the operation time data, classify it into segments according to read / write frequency, calculate the proportion of latency time, compare the migration frequency relationship, and combine the cache hit records to obtain partition latency and hit integrated data.

[0029] S403: Based on the partition latency and hit rate integration data, extract the read and write time data within the partition, group them according to operation frequency, calculate the average time for each group, associate with the cache hit rate, integrate the partition data, and construct a partition read and write efficiency data set.

[0030] As a further aspect of the present invention, the formula for calculating the average time consumed in each group is as follows:

[0031]

[0032] in, w represents the weighted average time spent in group i. i T represents the weight value of the frequency of the i-th group of operations. i The time taken for a single read / write operation of the i-th group is represented by n, where n represents the number of data groups.

[0033] As a further aspect of the present invention, the steps of comparing the high-frequency implementation details read / write frequency with the partition read / write efficiency data set, rearranging the partition data order, adjusting the implementation details arrangement order, iteratively allocating partition capacity, and generating an optimized cache partition structure are as follows:

[0034] S501: Based on the partition read / write efficiency data set, collect read / write data of high-frequency implementation details within the partition, extract the read / write frequency and cache hit rate of each partition through the data access log, parse the data access entries in the log one by one and summarize the time series statistics to generate a partition performance table.

[0035] S502: Using the partition performance table, analyze the performance differences between partitions, analyze the read / write frequency and hit rate between partitions to set performance thresholds, gradually adjust the boundary positions and analyze the access efficiency within the partitions, perform cross-validation on the boundary configuration, and generate a new partition boundary map.

[0036] S503: Based on the new partition boundary map, allocate cache partition capacity, determine the capacity size using partition read / write ratio and load distribution, optimize capacity by dynamically adjusting capacity and analyzing access response time, and generate an optimized cache partition structure.

[0037] As a further aspect of the present invention, based on the optimized cache partition structure, the steps of recording new access request implementation details information, statistically analyzing partition matching data, integrating new implementation details information to partition boundaries, and generating a dynamic adjustment scheme for the implementation details cache are as follows:

[0038] S601: Based on the optimized cache partition structure, extract the feature values ​​of the implementation details and combine them with the distribution data of the implementation details set within the partition. Compare the implementation details information with the matching rules of the implementation details within the partition, calculate the matching hit index data, and statistically analyze the matching distribution to obtain the partition matching hit data.

[0039] S602: Based on the partition matching hit data, calculate the hit ratio of each partition and analyze the deviation between the hit and the capacity, adjust the partition boundaries and redistribute the partition capacity, update the capacity and redefine the boundaries of the implementation details set of each partition, and obtain partition capacity distribution optimization data.

[0040] S603: Based on the partition capacity distribution optimization data, merge the implementation details information of new access requests into the corresponding partitions according to the adjusted partition boundaries, perform addition and update operations on the implementation details set within the partitions, record the current state of the partition boundaries, and generate a dynamic adjustment scheme for the implementation details cache.

[0041] As a further aspect of the present invention, the formula for calculating the matching hit index data is as follows:

[0042]

[0043] Among them, M hit P represents the hit metric data for the match. d This represents the feature value of the implementation details in the cache. The feature value that matches the implementation rules within the partition is represented by F, where F represents the total number of implementation rules in the set. This represents the absolute error between the implementation rule feature value in the cache and the matching feature value of the implementation rule within the partition, where α is the error adjustment coefficient and w f β is the weighting coefficient for the feature values ​​of the implementation rules, and w is the adjustment coefficient for the feature weights of the implementation rules. d γ is the weighting coefficient for the matching degree of the implementation details within the partition, and γ is the matching degree adjustment coefficient.

[0044] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0045] By recording the access frequency and distribution location of implementation details, and statistically analyzing the differences between access frequency and hierarchical location, the distribution density of implementation details can be accurately grasped. After grouping and organizing these details into a priority sequence, access distribution information is generated, thereby improving the reading efficiency of cached data. By sorting access frequencies, analyzing hierarchical spans, and adjusting priority intervals, the layout structure of data storage is optimized, significantly reducing the retrieval latency of frequently accessed data. Through adjusting the data location in the cache area, calculating capacity differences, and repartitioning the remaining capacity, the problem of uneven storage allocation in the cache area is solved, improving the utilization rate of cache resources. By recording read and write operations within partitions and statistically analyzing hit records, the differences in read and write efficiency are clearly identified, providing a precise basis for subsequent partition optimization. By comparing high-frequency implementation details with hit records, the boundary migration ratio is identified and the data order is rearranged to ensure the priority allocation of high-frequency access data, thereby improving the overall caching response capability. The dynamic adjustment of the new implementation details to partitions further enables real-time optimization of the cache, ensuring dynamic changes in data access needs, effectively enhancing the cache hit rate and response speed, reducing the database system load, and achieving significant gains compared to existing technologies in terms of storage performance, access efficiency, and resource allocation optimization. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the workflow of the present invention;

[0047] Figure 2 This is a detailed flowchart of S1 of the present invention;

[0048] Figure 3 This is a detailed flowchart of the S2 process of the present invention;

[0049] Figure 4 This is a detailed flowchart of the S3 process of the present invention;

[0050] Figure 5 This is a detailed flowchart of the S4 process of the present invention;

[0051] Figure 6 This is a detailed flowchart of S5 of the present invention;

[0052] Figure 7 This is a detailed flowchart of S6 of the present invention. Detailed Implementation

[0053] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0054] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0055] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent.

[0056] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0057] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0058] Please see Figure 1 This invention provides an LSM engine data caching optimization method, the processing flow of which may include the following steps:

[0059] S1: Based on the LSM storage engine, the cache area data records the number of accesses to the implementation details and their distribution position in the differentiated storage tier. By comparing the differences between the number of accesses and the tier position item by item, the distribution density of the implementation details is statistically analyzed. The data is grouped and organized into a priority sequence to generate a set of access distribution information for the implementation details.

[0060] S2: Based on the set of access distribution information of implementation details, sort the grouped data according to the number of accesses, analyze the hierarchical span and the number of accesses, determine the priority interval, adjust the order of implementation details groups in the priority interval, identify the continuous sequence distribution, optimize the arrangement order of implementation details, and generate the implementation details cache priority distribution result.

[0061] S3: Based on the implementation details cache priority distribution results, adjust the cache area implementation details data location, allocate priority implementation details to the cache area, calculate the allocation capacity difference, determine the remaining capacity of the low-speed cache area, repartition the remaining capacity implementation details, adjust the partition capacity ratio, and generate the cache partition adjustment configuration results.

[0062] S4: Based on the configuration adjustment results of the cache partition, record the read and write operations within the partition, count the read and write frequency and cache hit count, analyze the ratio of read and write latency to migration frequency, integrate latency data and hit records, and generate a partition read and write efficiency data set based on the partition read and write average.

[0063] S5: Based on the partition read / write efficiency dataset, compare the high-frequency implementation details read / write frequency with the hit records, identify the partition boundary migration ratio, rearrange the partition data order, adjust the implementation details arrangement order, redefine the partition capacity ratio, iteratively allocate partition capacity, and generate an optimized cache partition structure.

[0064] S6: Based on the optimized cache partition structure, record the implementation details of new access requests, match the new access implementation details to the corresponding cache partition, count the partition matching degree data, determine whether it is necessary to adjust the capacity of the implementation details in the partition, integrate the new implementation details information to the partition boundary and record it, and generate a dynamic adjustment scheme for the implementation details cache.

[0065] The implementation details access distribution information set includes statistics on the number of accesses to the implementation details, the location of the storage hierarchy, and the distribution density data of the implementation details. The implementation details cache priority distribution results include priority interval definition, implementation details group arrangement order, and continuous sequence distribution optimization structure. The cache partition adjustment configuration results include the implementation details cache area distribution, partition capacity allocation ratio, and low-speed cache area remaining capacity allocation. The partition read / write efficiency data set includes partition read / write frequency data, partition read / write latency statistics, and cache hit and migration frequency ratio. The optimized cache partition structure includes partition boundary data arrangement, implementation details partition capacity ratio definition, and partition optimized data order. The implementation details cache dynamic adjustment scheme includes new access implementation details matching partition data, new implementation details boundary information integration, and partition implementation details capacity adjustment records.

[0066] Specifically, such as Figure 2 As shown, the specific steps for generating a set of implementation rule access distribution information based on LSM storage engine cache area data, including recording the access frequency of implementation rules, calculating the distribution density of implementation rules, organizing the data into a priority sequence, and generating the implementation rule access distribution information set, are as follows:

[0067] S101: Based on the LSM storage engine, cache area data captures data implementation details, corresponding access counts and storage level information from the LSM storage engine, traverses the storage records and extracts key information for each implementation detail, including implementation detail name, access count, and level identifier, to obtain the implementation detail access record table.

[0068] By traversing the storage records, key information for each implementation rule is extracted, including the implementation rule name, access count, and hierarchy identifier. To ensure data accuracy, the mapping relationship between the implementation rule information in the storage records and the access count and hierarchy identifier in each record must first be clarified. This ensures that the data fully reflects the access situation of each implementation rule at different levels. Then, a matching algorithm is used to filter and compare the implementation rules and storage hierarchy information. The implementation rule name can be obtained from the index structure of the LSM storage engine. The access count is obtained by accumulating the number of accesses of each implementation rule in the cache area. The hierarchy identifier is identified by the metadata provided by the storage engine's internal data structure. The final output table will contain three fields: implementation rule name, access count, and hierarchy identifier. The extraction of the implementation rule name requires retrieval and matching of each record. The access count is obtained by accumulating frequently accessed records. The hierarchy identifier is extracted from the storage engine's internal hierarchical data structure. The obtained implementation rule access record table will provide a data foundation for subsequent analysis.

[0069] S102: Filter data from the implementation rules access record table, compare the level and access frequency of each implementation rule, count the access frequency of each level, and classify the data implementation rules by analyzing the differences in access frequency between levels to obtain the level access difference table.

[0070] First, by statistically analyzing access records, the access frequency of implementation rules within each level is calculated. Based on the access frequency, a differential comparison algorithm is used to evaluate the differences in access counts between different levels. The access frequency of each level can be obtained by summing the access counts of implementation rule records within each level and calculating the average access count of all implementation rules within that level. Based on this, a differential analysis is performed between levels. Based on the differences in access counts, the implementation rules are classified. Commonly used methods include bucketing or hierarchical clustering techniques. The implementation rules are divided into different categories according to the differences in access frequency, and a hierarchical access difference table is generated using a classification algorithm. This table shows the level of each implementation rule and its corresponding access count difference for subsequent processing and analysis.

[0071] S103: Use the hierarchical access difference table to sort the data. By setting a threshold for distribution density, prioritize the implementation rules. Assign priority to the implementation rules with high distribution density to obtain a set of implementation rule access distribution information.

[0072] Data is sorted using a hierarchical access difference table, according to the formula:

[0073]

[0074] Calculate the distribution density of each implementation rule and classify them by priority, assigning priority to implementation rules with higher distribution densities;

[0075] In the formula, P represents the priority of each implementing rule, and f j This indicates the frequency of access to the implementation details in level j, d j m represents the total number of visits to this level, where m is the number of levels.

[0076] Based on the access frequency of each implementation detail extracted from the hierarchical access difference table and the total number of accesses for each level, the distribution density of each level is first calculated. For each implementation detail, the distribution density is calculated as the ratio of its access frequency in a certain level to the total number of accesses in that level. The distribution densities of each implementation detail are summed to obtain the final priority value. In a storage system with 5 levels, the total number of accesses from level 1 to level 5 are d1=100, d2=150, d3=200, d4=120, and d5=80, respectively. If the access frequencies of a certain implementation detail in each level are f1=40, f2=60, f3=80, f4=30, and f5=50, then its priority is calculated as follows:

[0077]

[0078] The result indicates that the priority of this implementing rule is 2.075, meaning that this implementing rule has a high priority among all implementing rules.

[0079] Specifically, such as Figure 3 As shown, the specific steps for generating the implementation detail cache priority distribution result are as follows: Based on the implementation detail access distribution information set, sort the grouped data, analyze the hierarchical span, adjust the priority interval implementation detail group order, optimize the implementation detail arrangement order, and generate the implementation detail cache priority distribution result:

[0080] S201: Based on the access distribution information set of the implementation details, compare the group data item by item according to the number of accesses, extract the differences in access frequency between groups, sort out the increasing trend of the number of accesses for each group, and generate an access number ranking table by accumulating the number of accesses;

[0081] By analyzing the access records of each implementation rule in the database, all implementation rules are arranged according to the number of accesses. Precisely measuring the access frequency of each implementation rule ensures an accurate reflection of the true distribution of access volume. The data for each group is compared item by item to clarify the differences in access frequency between different groups. For example, for two different data groups A and B, the number of accesses for each implementation rule in group A is recorded and compared with the number of accesses for the same implementation rule in group B. In this way, we can record in detail the access differences of each implementation rule in different groups and analyze which implementation rules are particularly frequent or infrequent in a certain group. It is also necessary to pay attention to the increasing trend of access volume within each group, i.e., observe the growth curve of access volume in each group. This helps to understand the changing patterns of user behavior. Through detailed data analysis, an access volume ranking table is finally generated, listing the implementation rules in all groups according to access volume, providing basic data for subsequent analysis.

[0082] S202: Based on the access frequency sorting table, analyze the hierarchical span of each group item by item. By comparing the cumulative access frequency within the group with the hierarchical span, extract the hierarchical range of the interval and generate a priority interval division table according to the interval hierarchical priority.

[0083] By summing the access counts for each group, the total access volume for each group can be obtained. This total access volume is then compared with its hierarchical span to identify which hierarchical groups have higher access density, indicating a higher user demand for data at that level. Based on this, the hierarchical range is extracted. By precisely measuring the access density of each level, the relative importance of each level is determined. Using this as a basis, the levels are further sorted according to their priority, generating a priority interval partitioning table. This table not only reflects the access priority of each level but also provides a basis for data management and optimization, helping managers to allocate and optimize data resources more effectively.

[0084] S203: Based on the priority interval partitioning table, the combination of implementation details with high frequency within the interval and the group of implementation details with low level span are rearranged, the continuous sequence of implementation details groups is adjusted, and the priority arrangement is optimized by the internal order of the interval to generate the implementation details cache priority distribution result.

[0085] The most frequent implementation rule combinations within each interval are extracted and compared one by one with the implementation rule groups of lower-level spans. When rearranging the implementation rule combinations, the access frequency priority rule within the implementation rule group is adopted, placing the most frequent implementation rules at the beginning of the sequence. Simultaneously, adjustments are made based on the grouping characteristics of lower-level spans to ensure that the implementation rules in the lower-span groups can form a continuous access sequence. By adjusting the order of implementation rule groups within the interval, the priority arrangement of the entire interval conforms to the distribution optimization goal. During the optimization process, the adjustment results within each interval are recorded, marking the access frequency distribution of the adjusted implementation rules and the priority number of its interval. Finally, the implementation rule cache priority distribution result is generated. The entire optimization result, while ensuring the orderly distribution, also provides clear guidance for the actual implementation of the caching strategy, thereby improving the access efficiency and hierarchical optimization effect of the distributed structure.

[0086] Specifically, such as Figure 4 As shown, the specific steps for adjusting the cache partition adjustment configuration result based on the implementation details cache priority distribution result, adjusting the cache area implementation details data location, repartitioning the remaining capacity implementation details, adjusting the partition capacity ratio, and generating the cache partition adjustment configuration result are as follows:

[0087] S301: Based on the implementation details cache priority distribution results, extract the storage location and associated capacity data of high-priority implementation details, compare the capacity of implementation details with the current distribution of cache areas in groups, count the storage requirements corresponding to each group of implementation details, divide the distribution range of priority data, and obtain priority implementation details grouping and capacity labeling data.

[0088] By scanning cache logs to determine a set of high-priority implementation rules, the hit count of each implementation rule is counted, and the priority of each implementation rule is calculated based on the hit count and data distribution. Simultaneously, the implementation rule data is grouped based on the storage requirements of each rule. During grouping, grouping conditions are set based on the correlation between capacity and priority. After grouping, the capacity requirements of each group of implementation rules are merged and calculated, and the priority information of each group of implementation rules is labeled to the corresponding group. After priority grouping, the distribution intervals are divided. The interval division process is based on the gradient distribution of storage capacity, using the difference method to calculate gradient change points and determine partition boundaries. Regions with small gradient changes are merged to reduce the fragmentation of storage partitions. Finally, priority implementation rule grouping and capacity labeling data are output, providing detailed statistical data on grouping results and capacity requirements for subsequent operations.

[0089] S302: Based on priority implementation details grouping and capacity labeling data, calculate the difference between the capacity occupied by high-priority implementation details data and the current available capacity of the buffer, compare the capacity of low-priority implementation details corresponding to the difference distribution, extract the distribution data that does not meet the capacity requirements, and merge and classify the distribution data to obtain the buffer capacity difference and the remaining distribution data.

[0090] Calculate the difference between the capacity occupied by high-priority implementation details data and the current available capacity of the cache, according to the formula:

[0091]

[0092] Calculate the capacity difference, where:

[0093] P k =α k ·S k

[0094] In the formula, ΔC represents the capacity difference, and P k C represents the total capacity requirement of the k-th high-priority implementation rules. av Indicates the available capacity of the cache, α k The implementation details hit the weighting factor, S k This is the storage requirement for the kth set of implementation rules;

[0095] S k The total storage requirement of the kth group of implementation rules is calculated using direct data statistics methods.

[0096] α k The ratio of the number of times the implementation details were hit to the total number of hits was obtained by analyzing the logs.

[0097] C av Obtain directly from the allocation information in the cache;

[0098] Assuming the total cache capacity is 1000, and the currently used capacity is 700, then:

[0099] Available capacity: C av =1000-700=300;

[0100] Suppose there are two sets of high-priority implementation rules: the first set S1 = 200, with a hit rate weight α1 = 0.8; the second set S2 = 150, with a hit rate weight α2 = 0.7.

[0101] Calculate the implementation details requirements for each group: P1 = 0.8 * 200 = 160, P2 = 0.7 * 150 = 105;

[0102] Calculate the total capacity requirement and the difference:

[0103] ΔC = (160 + 105) - 300 = -35

[0104] The result shows a capacity difference of -35, indicating that the current available cache capacity can meet the storage requirements of high-priority implementation rules, but further optimization still requires adjustments to low-priority implementation rules. The calculation of the capacity difference provides a parameter basis for subsequent merging, classification, and storage adjustments.

[0105] S303: Based on the difference in cache capacity and the remaining distribution data, identify the adaptation parameters between the remaining distributed capacity and the low-speed cache, re-divide the remaining capacity units, adjust the capacity ratio of the unit partitions, reorganize the storage of the allocated areas, and establish the cache partition adjustment configuration results.

[0106] The system retrieves the storage capacity from the distributed data and the maximum fit value of the low-speed cache units within the partition. It optimizes capacity by gradually adjusting the partition capacity ratio. Simultaneously, a dynamic adjustment algorithm analyzes the distribution structure of the remaining capacity. When re-partitioning the remaining capacity units, a storage mapping table is constructed based on the similarity of storage requirements between units. This mapping table is used to prioritize storage capacity allocation, and the low-speed cache's fit parameters are used for matching adjustments. The storage reorganization strategy optimizes the balance of partition capacity distribution while ensuring full utilization of resources within the storage area. During storage reorganization, the storage units are iteratively adjusted multiple times, using the remaining capacity gradient change value as the adjustment boundary. A boundary dynamic optimization algorithm adjusts the capacity ratio configuration, ultimately generating the cache partition adjustment configuration result.

[0107] Specifically, such as Figure 5 As shown, the specific steps for generating a partition read / write efficiency dataset based on the cache partition configuration adjustment results, recording read / write operations within the partition, statistically analyzing read / write frequency, analyzing read / write latency, integrating latency data with hit records, and then performing the following steps:

[0108] S401: Based on the configuration adjustment results of the cache partition, read the read and write records in the partition, analyze the operation timestamp and data source, calculate the read and write operation time, perform read and write frequency classification statistics, compare the number of cache hits, and obtain partition read and write and hit statistics.

[0109] By analyzing the operation timestamps of each record, the occurrence time of read and write operations is extracted, and the records are organized in chronological order to ensure a clear data flow. When analyzing timestamps and data sources, the cached data block corresponding to each operation is matched with the external data source to determine whether the data is retrieved from the cache or read from external storage, and the operation time is recorded for each. Based on the recorded read and write time data, they are categorized and statistically analyzed according to time intervals to clarify the frequency distribution of read and write operations within different time periods. When calculating read and write frequency, read and write operations are divided into three categories: high-frequency, medium-frequency, and low-frequency. The total number of operations and average time for each frequency category are calculated, and the specific number of cache hits is recorded. Through comparative analysis of cache hit counts, the impact of caching on the efficiency of data access within a partition is further clarified, ultimately obtaining partition read / write and hit statistics, providing a foundation for subsequent in-depth analysis of latency and hit rate.

[0110] S402: Based on partition read / write and hit statistics, extract operation time data, classify by read / write frequency, calculate the latency ratio, compare migration frequency relationship, and combine with cache hit records to obtain partition latency and hit integrated data;

[0111] High-frequency operations are statistically analyzed separately, and their total and average latency are calculated. Meanwhile, mid-frequency and low-frequency operations are grouped separately to form latency distribution tables for different frequency categories. When calculating the proportion of latency, the total latency is compared with the overall latency of all operations within the partition to clarify the proportion of each type of operation in the overall latency. When analyzing the migration frequency relationship, the migration and cache hit times of high-frequency operations are compared to deduce the impact of migration operations on partition latency. Combined with cache hit records, the distribution pattern of hit rate under different frequency categories is further analyzed, thus obtaining integrated partition latency and hit data, providing detailed evidence for optimizing partition latency and hit performance.

[0112] S403: Based on partition latency and hit rate integration data, extract read and write time data within the partition, aggregate according to operation frequency, calculate the average time of each group, correlate with cache hit rate, integrate partition data, and construct a partition read and write efficiency data set.

[0113] The formula for calculating the average time for each group is as follows:

[0114]

[0115] in, w represents the weighted average time spent in group i. i T represents the weight value of the frequency of the i-th group of operations. i The time taken for a single read / write operation of the i-th group is represented by n, where n represents the number of data groups.

[0116] Detailed explanation of the formula and its calculation derivation:

[0117] This formula is used to calculate the weighted average time consumption. The weighted average time consumption is calculated by weighting the operation frequency of each group with the corresponding time consumption. The result can reflect the average time consumption of each group of operations and provide a reference for subsequent performance optimization.

[0118] w i The weight value of each operation frequency represents the proportion of that operation in the total number of operations. The weight value is set based on the actual monitoring data of each operation frequency. Specifically, it is the ratio of the number of times that operation is executed during system operation to the total number of operations. It is obtained through data monitoring during actual operation. For example, if an operation is executed 200 times, accounting for 20% of the total 1000 operations, its weight is 0.2.

[0119] T i The single read / write time for each group of operations represents the time required to execute the group of operations once, in milliseconds. This value is obtained through actual performance testing or data collection. The average time for a certain group of operations is set to 15 milliseconds, and is obtained through measurement or calculation.

[0120] n: Number of data groups, indicating how many groups all operations are divided into. For example, if the operations are divided according to their type, frequency, etc., then there are 5 different operation groups.

[0121] Assume there are 5 operation groups, with operation frequency and time consumption as follows:

[0122] Group 1: w1 = 0.2, T1 = 12 milliseconds;

[0123] Group 2: w2 = 0.3, T2 = 18 milliseconds;

[0124] Group 3: w3 = 0.1, T3 = 8 milliseconds;

[0125] Group 4: w4 = 0.25, T4 = 22 milliseconds;

[0126] Group 5: w5 = 0.15, T5 = 16 milliseconds;

[0127] The calculation process of the formula is as follows:

[0128]

[0129] The result of 16.5 milliseconds indicates the weighted average time of all operation groups. Combining the execution frequency of each operation group and the time of each operation, in this example, the result reflects the weighted calculation of the time of each of the five operation groups, representing the overall average read / write time. This value can serve as an important performance reference and further help analyze which operations need to be optimized.

[0130] Specifically, such as Figure 6 As shown, the specific steps for generating an optimized cache partition structure, based on a partition read / write efficiency dataset, comparing the high-frequency implementation details' read / write frequencies, rearranging the partition data order, adjusting the implementation details' arrangement order, iteratively allocating partition capacity, and generating the optimized cache partition structure, are as follows:

[0131] S501: Based on the partition read and write efficiency dataset, collect read and write data of high-frequency implementation details within the partition, extract the read and write frequency and cache hit rate of each partition through the data access log, parse the data access entries in the log one by one and summarize the time series statistics to generate a partition performance table.

[0132] First, high-frequency implementation details read / write data within the partitions are collected. The partition's data access logs are then analyzed line by line to extract the read / write frequency data for each partition. Simultaneously, cache hit rates are extracted, and a time-series statistical model is established. The time series data called in the statistical model needs to be arranged in chronological order of access time. Statistical analysis methods, such as data distribution histograms, are used to summarize and calculate the frequencies. Key read / write operations in the time series are marked as high-frequency access points. The criteria for marking high-frequency points is that the access frequency exceeds twice the average. Finally, the statistical results for each partition are output to the partition performance table. The partition performance table is generated by linking the partition read / write data in the data table with the access time interval distribution of the high-frequency points extracted from the time series.

[0133] S502: Uses a partition performance table to analyze the performance differences between partitions, analyzes the read / write frequency and hit rate between partitions to set performance thresholds, gradually adjusts the boundary positions and analyzes the access efficiency within the partitions, performs cross-validation on the boundary configuration, and generates a new partition boundary map.

[0134] Analyze the read / write frequency and hit rate between partitions to set performance thresholds, according to the formula:

[0135]

[0136] Calculate the threshold for performance differences between intervals;

[0137] In the formula, E represents the performance difference threshold, and R i R represents the read / write frequency of the i-th partition. avg Q represents the average read / write frequency across all partitions, where Q is the total number of partitions.

[0138] R i The frequency of read and write operations for each partition is calculated by analyzing the partition log data.

[0139] R avg The baseline level of overall performance is obtained by dividing the sum of the read and write frequencies of all partitions by the number of partitions.

[0140] Q represents the total number of partitions, which can be obtained directly through system settings.

[0141] Given four partitions with read / write frequencies of 200, 250, 300, and 350 respectively, calculate the average read / write frequency:

[0142]

[0143] Calculate the performance difference threshold:

[0144]

[0145] E = 3125

[0146] The results indicate that the standard value for partition performance differences is 3125. This value can be used to determine the performance optimization space between different partitions and to perform boundary adjustments.

[0147] S503: Based on the new partition boundary map, allocate cache partition capacity, determine the capacity size by using the partition read / write ratio and load distribution, optimize the capacity by dynamically adjusting the capacity and analyzing the access response time, and generate an optimized cache partition structure.

[0148] During the allocation of cache partition capacity, it is necessary to call the read / write ratio data of the partition and calculate the initial capacity of each partition based on the load distribution. The load distribution data comes from the access frequency and read / write offset time points statistically recorded in the logs. When the capacity is dynamically adjusted, it is based on the access response time. The difference is calculated by performing a difference operation on the access response time series of the partition, and the change in the difference is used as a reference for capacity optimization. If the response time deviation is greater than the set optimization threshold, the capacity size is increased by a step value and optimized again to finally obtain the optimized cache partition structure.

[0149] Specifically, such as Figure 7 As shown, based on the optimized cache partition structure, the specific steps for recording implementation details of new access requests, statistically analyzing partition matching data, integrating new implementation details information to partition boundaries, and generating a dynamic adjustment scheme for the implementation details cache are as follows:

[0150] S601: Based on the optimized cache partition structure, extract the feature values ​​of the implementation rules and combine them with the distribution data of the implementation rule set within the partition. Compare the implementation rule information with the matching rules of the implementation rules within the partition, calculate the matching hit index data, and statistically analyze the matching distribution to obtain the partition matching hit data.

[0151] The formula for calculating the match hit index data is as follows:

[0152]

[0153] Among them, Mhit P represents the hit metric data for the match. d

[0154] This represents the feature value of the implementation details in the cache. The feature value that matches the implementation rules within the partition is represented by F, where F represents the total number of implementation rules in the set. α represents the absolute error between the implementation rule feature value in the cache and the implementation rule matching feature value in the partition, where w is the error. f β is the weighting coefficient for the feature values ​​of the implementation rules, and w is the adjustment coefficient for the feature weights of the implementation rules. d

[0155] γ is the weighting coefficient for the matching degree of the implementation details within the partition, and γ is the matching degree adjustment coefficient;

[0156] This formula is used to calculate the data metrics of partition matching hits. The final result is used to evaluate the efficiency and accuracy of implementation rule matching within the cache partition. The parameters in the formula are obtained through the data collection and calculation process in the cache management system to quantify the accuracy of implementation rule matching, the impact of implementation rule features, and the degree of partition matching.

[0157] P d This represents the feature value of the implementation details in the cache. The set value is the weighted average of the implementation details length and partition matching degree. The frequency and distribution of the implementation details are calculated by the cache monitoring system. It is usually set to 128 bytes, which reflects the size of the implementation details feature.

[0158] This represents the feature value that matches the implementation details within the partition. The value is set to the weighted value of the matching implementation details within the cache partition. The calculation method is based on the average value of all implementation detail feature values ​​within the cache partition, and is set to 120 bytes based on the actual measured data.

[0159] α is the error adjustment coefficient, which represents the impact of implementation rule feature errors on the matching hit rate. This coefficient is set to 2.5, mainly calculated based on the error tolerance between implementation rule features and the system's tolerance to errors, reflecting the sensitivity to the impact of implementation rule errors.

[0160] w f The weighting coefficient for the feature value of the implementation details reflects the importance of the implementation details in the caching system. It is obtained by monitoring the frequency of each implementation detail in the caching system. This value is positively correlated with the frequency of the implementation details. When the frequency is high, the weighting coefficient is set to 1.2 to reflect the weighted influence of high-frequency implementation details.

[0161] β is the feature weight adjustment coefficient of the implementation rules, which is used to adjust the contribution of frequently occurring implementation rules to the matching hit rate. The calculation basis is the frequency of occurrence of the implementation rules, and the value is set to 1.4, which means that the implementation rules with higher frequency have a greater contribution to the matching hit rate.

[0162] w d The weighting coefficient for the matching degree of implementation rules within a partition reflects the impact of the matching degree of implementation rules on the hit rate. This weighting coefficient is calculated based on the matching of implementation rules within a partition and is set to 1.5, which means that the matching degree of a partition has a strong impact on the hit rate.

[0163] γ is the matching degree adjustment coefficient, set to 2.0. It mainly adjusts the hit index based on the matching degree between the implementation details in the partition and the implementation details in the cache. When the value is large, the implementation details with high matching degree have a greater impact on the final hit data.

[0164] F represents the total number of implementation rules, which is obtained through the cache monitoring system. The total number of implementation rules is set to 10,000.

[0165] Substitute the parameters into the formula to calculate:

[0166]

[0167]

[0168] The hit rate was 0.0583. This value represents the hit rate of implementation rules matching within a given cache partition. It further illustrates the matching accuracy of implementation rules within the cache partition and the sensitivity of the caching system to implementation rule features. A low hit rate (such as 0.0583) may indicate poor matching of implementation rules within that cache partition, requiring further optimization to improve matching efficiency and reduce the frequency of false matches.

[0169] S602: Based on the partition matching hit data, calculate the hit ratio of each partition and analyze the deviation between the hit and the capacity, adjust the partition boundaries and redistribute the partition capacity, update the capacity and redefine the boundaries of the implementation details set for each partition, and obtain partition capacity distribution optimization data.

[0170] By comparing the total number of recorded access requests with the number of hits, the hit rate of each partition is determined. For partitions with low hit rates, their hit data is compared in detail with the current capacity to analyze whether the hit deviation stems from insufficient capacity or uneven capacity distribution. When adjusting partition boundaries, the partition capacity distribution range is redefined based on the relationship between partition hit rate and access frequency. Priority is given to increasing the capacity of partitions with high hit rates and high access volumes, while reducing the capacity of partitions with low hit rates. By updating the set of implementation rules for each partition, the newly added capacity is allocated to the corresponding partitions, and implementation rules that exceed the new boundary range are removed or transferred. After the boundaries are redefined, the capacity and boundary data of each partition are recorded in the optimization log to ensure the completeness and traceability of partition capacity distribution optimization data. Obtaining partition capacity distribution optimization data lays the foundation for improving overall hit efficiency.

[0171] S603: Based on the partition capacity distribution optimization data, the implementation details information of new access requests are merged into the corresponding partitions according to the adjusted partition boundaries, the implementation details set within the partition is added and updated, the current state of the partition boundaries is recorded, and a dynamic adjustment scheme for the implementation details cache is generated.

[0172] The implementation details of new access requests are matched one by one with the adjusted partition boundaries. According to the partition boundary range, the implementation details are merged into the corresponding partitions. For the merged partitions, it is checked whether the same implementation details already exist in the implementation details set. For implementation details that do not exist, an addition operation is performed and they are assigned to the appropriate position in the partition. For existing implementation details, their access count, last access time and other dynamic information are updated to ensure that the implementation details set can reflect the current access status. After completing the addition and update operations, the current status of the partition boundary is recorded, including partition capacity, number of implementation details and boundary range. The information is organized into a partition dynamic adjustment log to provide data support for subsequent optimization, generate a dynamic adjustment scheme for implementation details caching, optimize the utilization efficiency of cache resources within the partition, and ensure that the cache can maintain high efficiency in a dynamic environment.

[0173] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data caching optimization method for an LSM engine, characterized in that, Includes the following steps: Based on the LSM storage engine, the cache area data records the number of accesses to the implementation details, calculates the distribution density of the implementation details, organizes the data into a priority sequence, and generates a set of access distribution information for the implementation details. Based on the implementation details access distribution information set, the grouped data is sorted, the hierarchical span is analyzed, the priority interval implementation details group order is adjusted, the implementation details arrangement order is optimized, and the implementation details cache priority distribution result is generated. Based on the cache priority distribution results of the implementation details, adjust the data location of the cache area implementation details, repartition the remaining capacity implementation details, adjust the partition capacity ratio, and generate cache partition adjustment configuration results; Based on the cache partition adjustment configuration results, record the read and write operations within the partition, count the read and write frequency, analyze the read and write latency, integrate the latency data and hit records, and generate a partition read and write efficiency data set. Based on the partition read / write efficiency data set, the high-frequency implementation details read / write frequency is compared, the partition data order is rearranged, the implementation details arrangement order is adjusted, the partition capacity allocation is iteratively performed, and an optimized cache partition structure is generated. Based on the optimized cache partition structure, record the implementation details of new access requests, statistically analyze partition matching data, integrate the new implementation details information to the partition boundary, and generate a dynamic adjustment scheme for the implementation details cache.

2. The LSM engine data caching optimization method according to claim 1, characterized in that, The implementation details access distribution information set includes statistics on the number of accesses to the implementation details, the storage level distribution location, and the distribution density data of the implementation details. The implementation details cache priority distribution results include priority interval definition, implementation details group arrangement order, and continuous sequence distribution optimization structure. The cache partition adjustment configuration results include implementation details cache area distribution, partition capacity allocation ratio, and low-speed cache area remaining capacity allocation. The partition read / write efficiency data set includes partition read / write frequency data, partition read / write latency statistics, and cache hit and migration frequency ratio. The optimized cache partition structure includes partition boundary data arrangement, implementation details partition capacity ratio definition, and partition optimized data order. The implementation details cache dynamic adjustment scheme includes new access implementation details matching partition data, new implementation details boundary information integration, and partition implementation details capacity adjustment records.

3. The LSM engine data caching optimization method according to claim 1, characterized in that, The specific steps for generating a set of implementation rule access distribution information based on LSM storage engine cache area data, recording the access frequency of implementation rules, calculating the distribution density of implementation rules, organizing the data into a priority sequence, and generating the implementation rule access distribution information set are as follows: Based on the cache area data of the LSM storage engine, the data implementation details, corresponding access counts and storage level information are captured from the LSM storage engine. The storage records are traversed and the key information of each implementation detail is extracted, including the implementation detail name, access count and level identifier, to obtain the implementation detail access record table. Data is filtered from the access record table of the implementation rules, and the hierarchy and access frequency of each implementation rule are compared. The access frequency of each level is counted. By analyzing the differences in access frequency between levels, the data implementation rules are classified and processed to obtain the hierarchy access difference table. The hierarchical access difference table is used to sort the data. By setting a threshold for distribution density, the implementation rules are prioritized and classified. The implementation rules with higher distribution density are given priority, thus obtaining a set of implementation rule access distribution information.

4. The LSM engine data caching optimization method according to claim 1, characterized in that, Based on the aforementioned implementation rules access distribution information set, the steps of sorting the grouped data, analyzing the hierarchical span, adjusting the priority interval implementation rules group order, optimizing the implementation rules arrangement order, and generating the implementation rules cache priority distribution result are as follows: Based on the access distribution information set of the implementation details, the group data is compared item by item according to the number of accesses, the difference in access frequency between groups is extracted, the increasing trend of the number of accesses for each group is sorted, and the access number ranking table is generated by accumulating the number of accesses. Based on the access frequency sorting table, the hierarchical span of each group is analyzed item by item. By comparing the cumulative access frequency within the group with the hierarchical span, the hierarchical range of the interval is extracted, and a priority interval division table is generated according to the interval hierarchical priority. Based on the priority interval partitioning table, the implementation rules combination with high frequency within the interval and the implementation rules group with low level span are rearranged, the continuous sequence of implementation rules groups is adjusted, and the priority arrangement is optimized by the internal order of the interval to generate the implementation rules cache priority distribution result.

5. The LSM engine data caching optimization method according to claim 1, characterized in that, Based on the cache priority distribution results of the implementation details, the specific steps for adjusting the data location of the cache area implementation details, repartitioning the remaining capacity implementation details, adjusting the partition capacity ratio, and generating the cache partition adjustment configuration results are as follows: Based on the implementation details cache priority distribution results, the storage location and associated capacity data of high-priority implementation details are extracted, the capacity of implementation details is compared with the current distribution of cache areas in groups, the storage requirements corresponding to each group of implementation details are statistically analyzed, and the distribution range of priority data is divided to obtain priority implementation details grouping and capacity labeling data. Based on the priority implementation details grouping and capacity labeling data, calculate the difference between the capacity occupied by high priority implementation details data and the current available capacity of the cache area, compare the capacity of low priority implementation details corresponding to the difference distribution, extract the distribution data that does not meet the capacity requirements, and merge and classify the distribution data to obtain the cache area capacity difference and the remaining distribution data. Based on the cache capacity difference and remaining distribution data, the matching parameters between the remaining distributed capacity and the low-speed cache are identified, the remaining capacity units are re-divided, the capacity ratio of the unit partitions is adjusted, the storage of the allocated areas is reorganized, and the cache partition adjustment configuration results are established.

6. The LSM engine data caching optimization method according to claim 1, characterized in that, The specific steps for generating a partition read / write efficiency dataset based on the cache partition adjustment configuration results, recording read / write operations within the partition, statistically analyzing read / write frequency, analyzing read / write latency, integrating latency data with hit records, and ultimately generating the partition read / write efficiency dataset are as follows: Based on the cache partition adjustment configuration results, read the read and write records in the partition, analyze the operation timestamps and data sources, calculate the read and write operation time, perform read and write frequency classification statistics, compare the number of cache hits, and obtain partition read and write and hit statistics. Based on the partition read / write and hit statistics, operation time data is extracted, segmented and classified according to read / write frequency, the proportion of latency time is calculated, the migration frequency relationship is compared, and combined with cache hit records, partition latency and hit integrated data are obtained. Based on the partition latency and hit rate integration data, the read and write time data within the partition is extracted, grouped by operation frequency, the average time of each group is calculated, and the cache hit rate is correlated to integrate the partition data and construct a partition read and write efficiency data set.

7. The LSM engine data caching optimization method according to claim 6, characterized in that, The formula for calculating the average time for each group is as follows: ； in, This represents the weighted average time consumed in the i-th group. The weight value represents the frequency of the i-th group of operations. This represents the time taken for a single read / write operation in the i-th group. This represents the number of data sets.

8. The LSM engine data caching optimization method according to claim 1, characterized in that, Based on the partition read / write efficiency data set, the steps of comparing the high-frequency implementation details read / write frequencies, rearranging the partition data order, adjusting the implementation details arrangement order, iteratively allocating partition capacity, and generating an optimized cache partition structure are as follows: Based on the partition read / write efficiency data set, high-frequency implementation details read / write data within the partition are collected. The read / write frequency and cache hit rate of each partition are extracted through the data access log. The data access entries in the log are parsed one by one and time series statistics are summarized to generate a partition performance table. Using the partition performance table, the performance differences between partitions are analyzed, the read / write frequency and hit rate between partitions are analyzed to set performance thresholds, the boundary positions are gradually adjusted and the access efficiency within the partitions is analyzed, the boundary configuration is cross-validated, and a new partition boundary map is generated. Based on the new partition boundary map, the cache partition capacity is allocated, and the capacity size is determined by the partition read / write ratio and load distribution. The capacity is optimized by dynamically adjusting the capacity and analyzing the access response time, and an optimized cache partition structure is generated.

9. The LSM engine data caching optimization method according to claim 1, characterized in that, Based on the optimized cache partitioning structure, the specific steps for recording new access request implementation details, statistically analyzing partition matching data, integrating new implementation details information to partition boundaries, and generating a dynamic adjustment scheme for the implementation details cache are as follows: Based on the optimized cache partition structure, feature values ​​of implementation rules are extracted and combined with the distribution data of the implementation rule set within the partition. The implementation rule information is compared with the matching rules of the implementation rules within the partition. The matching hit index data is calculated and the matching distribution is statistically analyzed to obtain the partition matching hit data. Based on the partition matching hit data, the hit ratio of each partition is calculated and the deviation between the hit and the capacity is analyzed. The partition boundaries are adjusted and the partition capacity is redistributed. The capacity of each partition's implementation details set is updated and the boundaries are redefined to obtain partition capacity distribution optimization data. Based on the partition capacity distribution optimization data, the implementation details information of new access requests are merged into the corresponding partitions according to the adjusted partition boundaries. The implementation details set within the partitions is added and updated. The current state of the partition boundaries is recorded, and a dynamic adjustment scheme for the implementation details cache is generated.

10. The LSM engine data caching optimization method according to claim 9, characterized in that, The formula for calculating the hit index data of the match is as follows: ； in, This represents the hit metric data for the match. This represents the feature value of the implementation details in the cache. This represents the feature value that matches the implementation details within the partition. This represents the total number of implementation rules. This represents the absolute error between the implementation rule feature value in the cache and the implementation rule matching feature value within the partition. This is the error adjustment factor. The weighting coefficients for the characteristic values ​​of the implementation rules, The feature weight adjustment coefficient is used to implement the detailed rules. The weighting coefficients for the matching degree of implementation details within the partition. Adjust the coefficient for matching degree.

Images