LSM engine data cache optimization method

Abstract

The invention relates to the technical field of database cache management, in particular to an LSM engine data cache optimization method, which comprises the following steps of: based on cache region data of an LSM storage engine, recording access times of implementation detailed rules, counting distribution density of implementation detailed rules, arranging the data into a priority sequence, and generating an implementation detailed rule access distribution information set. According to the method, accurate mastering of the implementation detailed rule distribution density is achieved by recording the implementation detailed rule access times and distribution positions, the implementation detailed rule distribution density is grouped and sorted into the priority sequence, the cache data reading efficiency is improved, and the cache data reading efficiency is improved by adjusting and optimizing the data storage layout and reducing the high-frequency data retrieval delay. The method has the advantages that the problem of uneven storage allocation is solved, the resource utilization rate is increased, data distribution is optimized through high-frequency implementation rule comparison, new implementation rule matching is dynamically adjusted, the hit rate and the response speed are increased, the system load is reduced, and the storage performance and the resource allocation efficiency are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of database cache management, and in particular to a method for optimizing LSM engine data cache. Background Art

[0002] The field of database cache management technology focuses on improving the access speed and processing efficiency of database systems. It involves using a cache mechanism in the database system to store temporary data, thereby reducing the need for direct access to the disk. The database cache can be memory-based or based on different levels of hard disk cache strategies. By intelligently caching data, such as query results, commonly used data pages, etc., database cache management can significantly improve the speed of data retrieval, reduce the load on the database server, and optimize the overall system performance. In actual applications, the design of the cache strategy takes into account the data access frequency, storage cost, and cache invalidation strategy to achieve the best performance improvement effect.

[0003] Among them, the LSM engine data cache optimization method involves a technology that uses implementation details caching to enhance performance in a specific type of database storage engine LSM (Log-Structured Merge-tree). The LSM storage engine optimizes the performance of operations by recording data change operations in memory and periodically merging the changes into the data structure on the disk. The implementation details caching method is applied in this structure mainly to optimize the efficiency of read operations. By caching commonly used data implementation details, query requests for data can be responded to quickly. This method is particularly suitable for scenarios with highly asynchronous read and write requests, such as large-scale Web applications and real-time data processing, and can significantly improve the overall responsiveness of data access speed.

[0004] In database cache management, although the existing technology can reduce the frequency of disk access through cache strategies, it has weak adaptability to dynamic changes in cached data and is difficult to effectively deal with the imbalance in data access distribution. In the differentiated management of storage levels, the existing technology only relies on static or preset strategies and cannot efficiently adjust the implementation details of frequently changing high-frequency data, resulting in a low cache data hit rate. The existing technology lacks a refined adjustment mechanism in cache capacity allocation, making it difficult to achieve reasonable allocation of resources between different storage levels, resulting in a waste of storage resources in low-speed cache areas. There are also deficiencies in the analysis of read and write efficiency and delay optimization. Especially in high-concurrency read and write scenarios, the existing solutions cannot fully identify and adjust the priority distribution of cache resources, resulting in a decrease in the method's responsiveness. For example, in real-time data processing or large-scale Web application scenarios, the imbalance of read and write requests will amplify the problem, increase system load, and reduce overall performance. Summary of the invention

[0005] In order to solve the problem that the existing technology in database cache management can reduce the frequency of disk access through cache strategies, but has weak adaptability to dynamic changes in cache data and is difficult to effectively deal with the imbalance of data access distribution, in the differentiated management of storage levels, the existing technology only relies on static or preset strategies, and cannot efficiently adjust the implementation details of frequently changing high-frequency data, resulting in a low cache data hit rate. The existing technology lacks a refined adjustment mechanism in cache capacity allocation, and it is difficult to achieve reasonable allocation of resources between different storage levels, resulting in a waste of storage resources in low-speed cache areas. There are also deficiencies in the analysis of read and write efficiency and delay optimization. Especially in high-concurrency read and write scenarios, the existing scheme cannot fully identify and adjust the priority distribution of cache resources, resulting in a decrease in the method's responsiveness. For example, in real-time data processing or large-scale Web application scenarios, the imbalance of read and write requests will amplify the problem, increase system load, and reduce the overall performance. The technical problem, the embodiment of the present invention provides a LSM engine data cache optimization method. The technical solution is as follows:

[0006] On the one hand, a method for optimizing LSM engine data cache is provided, the method comprising:

[0007] S1: Based on the cache area data of the LSM storage engine, record the number of accesses to the implementation details, count the distribution density of the implementation details, organize the data into a priority sequence, and generate a set of implementation details access distribution information;

[0008] S2: Based on the implementation rule access distribution information set, sort the grouped data, analyze the level span, adjust the priority interval implementation rule group order, optimize the implementation rule arrangement order, and generate the implementation rule cache priority distribution result;

[0009] S3: Based on the cache priority distribution result of the implementation rules, adjust the data location of the cache area implementation rules, repartition the remaining capacity implementation rules, adjust the partition capacity ratio, and generate a cache partition adjustment configuration result;

[0010] S4: Based on the cache partition adjustment configuration result, record the read and write operations in the partition, count the read and write frequencies, analyze the read and write delay time, integrate the delay data and hit records, and generate a partition read and write efficiency data set;

[0011] S5: Based on the partition read-write efficiency data set, compare the read-write frequency of the high-frequency implementation details, rearrange the partition data order, adjust the implementation details arrangement order, iteratively perform partition capacity allocation, and generate an optimized cache partition structure;

[0012] S6: Based on the optimized cache partition structure, record the implementation details information of the new access request, count the partition matching data, integrate the new implementation details information to the partition boundary, and generate a dynamic adjustment plan for the implementation details cache.

[0013] As a further solution of the present invention, the implementation details access distribution information set includes the implementation details access count statistics, storage hierarchy distribution position, and implementation details distribution density data; the implementation details cache priority distribution results include priority interval demarcation, implementation details group arrangement order, and continuous sequence distribution optimization structure; the cache partition adjustment configuration results include implementation details cache area distribution, partition capacity allocation ratio, and low-speed cache area remaining capacity division; the partition read and write efficiency data set includes partition read and write frequency data, partition read and write delay time statistics, and cache hit and migration frequency ratio; the optimized cache partition structure includes partition boundary data arrangement, implementation details partition capacity ratio demarcation, and partition optimized data order; the implementation details cache dynamic adjustment scheme includes new access implementation details matching partition data, new implementation details boundary information integration, and partition implementation details capacity adjustment records.

[0014] As a further solution of the present invention, based on the cache area data of the LSM storage engine, the number of accesses to the implementation details is recorded, the distribution density of the implementation details is counted, and the data is sorted into a priority sequence. The steps of generating an implementation details access distribution information set are specifically as follows:

[0015] S101: Based on the cache area data of the LSM storage engine, capture data implementation rules, corresponding access times and storage level information from the LSM storage engine, traverse the storage records and extract key information of each implementation rule, including the implementation rule name, access count, and level identifier, to obtain an implementation rule access record table;

[0016] S102: Filtering data from the implementation rule access record table, comparing the level and access times of each implementation rule, counting the access frequency of each level, and classifying the data implementation rules by analyzing the difference in access times between levels to obtain a level access difference table;

[0017] S103: sorting data using the hierarchical access difference table, and classifying implementation details by priority by setting a threshold of distribution density, giving priority to implementation details with large distribution density, and obtaining an implementation details access distribution information set.

[0018] As a further solution of the present invention, based on the implementation details access distribution information set, the grouped data is sorted, the hierarchical span is analyzed, the priority interval implementation details group order is adjusted, the implementation details arrangement order is optimized, and the steps of generating the implementation details cache priority distribution result are specifically as follows:

[0019] S201: Based on the access distribution information set of the implementation details, the group data is compared item by item according to the access times, the access frequency differences between the groups are extracted, the access frequency increase trend of each group is sorted, and the access times ranking table is generated by accumulating the access times;

[0020] S202: Based on the access frequency ranking table, analyze the hierarchical span of each group item by item, extract the hierarchical range of the interval by comparing the access frequency accumulation in the group with the hierarchical span, and generate a priority interval division table according to the interval hierarchical priority;

[0021] S203: Based on the priority interval division table, the implementation rule combinations with high frequencies in the interval are rearranged with the implementation rule groups with low-level spans, the continuous sequence of the implementation rule groups is adjusted, the priority arrangement is optimized through the internal sequence of the interval, and the implementation rule cache priority distribution result is generated.

[0022] As a further solution of the present invention, based on the cache priority distribution result of the implementation details, the steps of adjusting the data location of the cache area implementation details, repartitioning the remaining capacity implementation details, adjusting the partition capacity ratio, and generating the cache partition adjustment configuration result are specifically:

[0023] S301: Based on the implementation rules cache priority distribution result, the storage location and associated capacity data of high-priority implementation rules are extracted, the implementation rules capacity and the current distribution of the cache area are compared in groups, the storage requirements corresponding to each group of implementation rules are counted, and the distribution interval of the priority data is divided to obtain the priority implementation rules grouping and capacity annotation data;

[0024] S302: Based on the priority implementation rule grouping and capacity annotation data, calculate the difference between the capacity occupied by the high-priority implementation rule data and the available capacity of the current cache area, compare the capacity of the low-priority implementation rule corresponding to the difference distribution, extract the distribution data that does not meet the capacity requirements, merge and classify the distribution data, and obtain the cache area capacity difference and remaining distribution data;

[0025] S303: Based on the cache capacity difference and the remaining distribution data, identify the adaptation parameters of the remaining distribution capacity and the low-speed cache area, re-divide the remaining capacity units, adjust the capacity ratio of the unit partitions, reorganize the storage of the allocated area, and establish the cache partition adjustment configuration result.

[0026] As a further solution of the present invention, based on the cache partition adjustment configuration result, the steps of recording the read and write operations in the partition, counting the read and write frequency, analyzing the read and write delay time, integrating the delay data and the hit record, and generating the partition read and write efficiency data set are specifically as follows:

[0027] S401: Based on the cache partition adjustment configuration result, read the read and write records in the partition, analyze the operation timestamp and data source, calculate the time consumption of the read and write operations, perform classification statistics of the read and write frequencies, compare the cache hit counts, and obtain partition read and write and hit statistics;

[0028] S402: Based on the partition read / write and hit statistics, extract operation time-consuming data, classify by segment according to read / write frequency, calculate the delay time ratio, compare the migration frequency relationship, and obtain partition delay and hit integration data in combination with cache hit records;

[0029] S403: Based on the partition delay and hit integration data, extract the read and write time consumption data in the partition, group them by operation frequency, calculate the average time consumption of each group, associate the cache hit rate, integrate the partition data, and build a partition read and write efficiency data set.

[0030] As a further solution of the present invention, the formula for calculating the average time consumption of each group is as follows:

[0031]

[0032] in, represents the weighted average time consumed by group i, w i represents the weight value of the i-th group of operating frequencies, T i represents the time taken for a single read or write operation of the i-th group, and n represents the number of data groups.

[0033] As a further solution of the present invention, based on the partition read-write efficiency data set, the read-write frequency of the high-frequency implementation details is compared, the partition data order is rearranged, the implementation details arrangement order is adjusted, and the partition capacity allocation is iteratively performed to generate the optimized cache partition structure. Specifically, the steps are as follows:

[0034] S501: Based on the partition read-write efficiency data set, collect the read-write data of the high-frequency implementation details in the partition, extract the read-write frequency and cache hit rate of each partition through the data access log, parse the data access entries in the log one by one and summarize the time series statistics to generate a partition performance table;

[0035] S502: using the partition performance table, analyzing the performance differences between partitions, analyzing the read and write frequencies and hit rates between partitions to set performance thresholds, gradually adjusting the boundary positions and analyzing the access efficiency within the partitions, cross-validating the boundary configuration, and generating a new partition boundary map;

[0036] S503: Allocate cache partition capacity according to the new partition boundary map, determine the capacity size using the partition read-write ratio and load distribution, optimize the capacity by dynamically adjusting the capacity and analyzing the access response time, and generate an optimized cache partition structure.

[0037] As a further solution of the present invention, based on the optimized cache partition structure, the steps of recording the implementation details of new access requests, counting the partition matching data, integrating the new implementation details information to the partition boundary, and generating the implementation details cache dynamic adjustment plan are as follows:

[0038] S601: Based on the optimized cache partition structure, extract the characteristic value of the implementation details and combine it with the distribution data of the implementation details set in the partition, compare the implementation details information with the matching rules of the implementation details in the partition, calculate the matching hit index data and count the matching distribution, and obtain the partition matching hit data;

[0039] S602: Based on the partition matching hit data, calculate the hit ratio of each partition and analyze the deviation between the hit and the capacity, adjust the partition boundary to reallocate the partition capacity, update the capacity and redefine the boundary of the implementation rule set of each partition, and obtain partition capacity distribution optimization data;

[0040] S603: Based on the partition capacity distribution optimization data, the implementation details information of the new access request is merged into the corresponding partition according to the adjusted partition boundary, the implementation details set in the partition is added and updated, the current status of the partition boundary is recorded, and a dynamic adjustment plan for the implementation details cache is generated.

[0041] As a further solution of the present invention, the formula for calculating the matching hit index data is as follows:

[0042]

[0043] Among them, M hit Indicates the matching hit indicator data, P d Characteristic value indicating the implementation details of the cache, represents the feature value that matches the implementation rules in the partition, F represents the total number of implementation rule sets, represents the absolute error between the implementation rule feature value in the cache and the matching feature value of the implementation rule in the partition, α is the error adjustment coefficient, and w f is the weight coefficient of the characteristic value of the implementation rules, β is the weight adjustment coefficient of the characteristic value of the implementation rules, and w d is the weight coefficient of the matching degree of the implementation rules within the partition, and γ is the matching degree adjustment coefficient.

[0044] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0045] By recording the number of accesses and distribution positions of the implementation details and counting the differences between the number of accesses and the hierarchical positions, the distribution density of the implementation details can be accurately grasped. After grouping them into priority sequences, access distribution information is generated, thereby improving the reading efficiency of cached data. By sorting the number of accesses, analyzing the hierarchical span and adjusting the priority interval, the layout structure of data storage is optimized, and the retrieval delay of high-frequency access data is significantly reduced. By adjusting the data position in the cache area, calculating the capacity difference and repartitioning the remaining capacity, the problem of uneven storage distribution in the cache area is solved, and the utilization rate of cache resources is improved. By recording the read and write operations within the partition and counting the hit records, the difference in read and write efficiency is clarified, providing an accurate basis for subsequent partition optimization. By comparing the high-frequency implementation details with the hit records, the boundary migration ratio is identified and the data order is rearranged to ensure the priority allocation of high-frequency access data and improve the overall response capability of the cache. The new implementation details are matched to the dynamic adjustment of the partition to further realize the real-time optimization of the cache, ensure the dynamic changes in data access requirements, effectively enhance the hit rate and response speed of the cache data, reduce the database system load, and achieve significant gains in storage performance, access efficiency, and resource allocation optimization compared to existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0047] Figure 2 This is a detailed flow chart of S1 of the present invention;

[0048] Figure 3 This is a detailed flow chart of S2 of the present invention;

[0049] Figure 4 This is a detailed flow chart of S3 of the present invention;

[0050] Figure 5 This is a detailed flow chart of S4 of the present invention;

[0051] Figure 6 This is a detailed flow chart of S5 of the present invention;

[0052] Figure 7 This is a detailed flow chart of S6 of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0054] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0055] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0056] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0057] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0058] See also Figure 1 The embodiment of the present invention provides a method for optimizing LSM engine data cache. The processing flow of the method may include the following steps:

[0059] S1: Based on the cache area data of the LSM storage engine, the access times of the implementation details and their distribution positions in the differentiated storage hierarchies are recorded. By comparing the differences between the access times and the hierarchical positions item by item, the distribution density of the implementation details is counted, and the data is grouped into priority sequences to generate a set of implementation details access distribution information.

[0060] S2: Based on the set of implementation rule access distribution information, sort the grouped data according to the number of accesses, analyze the level span and the number of accesses, determine the priority interval, adjust the priority interval implementation rule group order, identify the continuous sequence distribution, optimize the implementation rule arrangement order, and generate the implementation rule cache priority distribution result;

[0061] S3: Based on the cache priority distribution result of the implementation rules, adjust the data location of the cache area implementation rules, assign the priority implementation rules to the cache area, calculate the allocation capacity difference, determine the remaining capacity of the low-speed cache area, re-partition the remaining capacity implementation rules, adjust the partition capacity ratio, and generate the cache partition adjustment configuration result;

[0062] S4: Based on the cache partition adjustment configuration results, record the read and write operations in the partition, count the read and write frequency and cache hit times, analyze the ratio of read and write delay time to migration frequency, integrate delay data and hit records, and generate a partition read and write efficiency data set based on the partition read and write mean;

[0063] S5: Based on the partition read and write efficiency data set, compare the read and write frequency of high-frequency implementation details with the hit records, identify the partition boundary migration ratio, rearrange the partition data order, adjust the implementation details arrangement order, re-define the partition capacity ratio, iteratively allocate partition capacity, and generate the optimized cache partition structure;

[0064] S6: Based on the optimized cache partition structure, record the implementation details of new access requests, match the new access implementation details to the corresponding cache partition, count the partition matching data, determine whether it is necessary to adjust the implementation details capacity within the partition, integrate the new implementation details information to the partition boundary and record it, and generate a dynamic adjustment plan for the implementation details cache.

[0065] The access distribution information set of the implementation details includes the access count statistics of the implementation details, the storage hierarchy distribution position, and the implementation details distribution density data. The implementation details cache priority distribution results include the priority interval demarcation, the implementation details group arrangement order, and the continuous sequence distribution optimization structure. The cache partition adjustment configuration results include the implementation details cache area distribution, the partition capacity allocation ratio, and the low-speed cache area remaining capacity division. The partition read and write efficiency data set includes the partition read and write frequency data, the partition read and write delay time statistics, and the cache hit and migration frequency ratio. The optimized cache partition structure includes the partition boundary data arrangement, the implementation details partition capacity ratio demarcation, and the partition optimized data order. The implementation details cache dynamic adjustment plan includes the new access implementation details matching partition data, the new implementation details boundary information integration, and the partition implementation details capacity adjustment record.

[0066] Specifically, if Figure 2 As shown, based on the cache area data of the LSM storage engine, the steps of recording the number of accesses to the implementation details, counting the distribution density of the implementation details, arranging the data into a priority sequence, and generating a set of implementation details access distribution information are as follows:

[0067] S101: Based on the cache area data of the LSM storage engine, capture data implementation rules, corresponding access times and storage level information from the LSM storage engine, traverse the storage records and extract key information of each implementation rule, including the implementation rule name, access count, and level identifier, to obtain an implementation rule access record table;

[0068] By traversing the storage records, the key information of each implementation rule is extracted, including the implementation rule name, access count, and level identifier. To ensure the accuracy of the data, it is first necessary to clarify the mapping relationship between the implementation rule information in the storage record and the access count and level identifier in each record, so as to ensure that the data fully reflects the access status of each implementation rule at different levels. Then, the implementation rules and storage level information are screened and compared through the matching algorithm. The implementation rule name can be obtained from the index structure of the LSM storage engine, the access count is obtained by accumulating the number of accesses of each implementation rule in the cache area, and the level identifier is identified by the metadata provided by the internal data structure of the storage engine. The final output format of the table will include three fields: implementation rule name, access count, and level identifier. The extraction of the implementation rule name requires retrieval and matching of each record, the access count is obtained by accumulating frequently accessed records, and the level identifier is extracted from the hierarchical data structure inside the storage engine. The obtained implementation rule access record table will provide a data basis for subsequent analysis.

[0069] S102: Filter data from the implementation rule access record table, compare the level and access times of each implementation rule, count the access frequency of each level, classify the data implementation rules by analyzing the difference in access times between levels, and obtain a level access difference table;

[0070] First, by counting the access records, the access frequency of the implementation details in each level is calculated. According to the access frequency, a differentiated comparison algorithm is used to evaluate the difference in access times between different levels. The access frequency of each level can be obtained by summarizing the access times of the implementation details records in each level and calculating the average access times of all the implementation details in the level. Based on this, the difference analysis between levels is carried out. Based on the difference in access times, the implementation details are classified. Commonly used methods include bucketing or hierarchical clustering technology. The implementation details are divided into different categories according to the difference in access frequency, and a hierarchical access difference table is generated through a classification algorithm. The table shows the level of each implementation detail and its corresponding access time difference for subsequent processing and analysis.

[0071] S103: sorting data using the hierarchical access difference table, and classifying the implementation details by priority by setting a threshold of distribution density, giving priority to implementation details with large distribution density, and obtaining a set of implementation details access distribution information;

[0072] The data is sorted by using the hierarchical access difference table, according to the formula:

[0073]

[0074] Calculate the distribution density of each implementation detail and classify their priorities. Give priority to the implementation details with large distribution density;

[0075] Where P represents the priority of each implementation rule, f j represents the access frequency of the implementation details in level j, d j is the total number of visits to the level, and m is the number of levels;

[0076] According to the access frequency of each implementation rule extracted from the hierarchical access difference table and the total number of accesses at each level, the distribution density of each level is first calculated. For each implementation rule, its distribution density is calculated by the ratio of its access frequency at a certain level to the total number of accesses at that level. The distribution density of each implementation rule is accumulated to obtain the final priority value. In a certain storage system, there are 5 levels in total, and the total number of accesses from level 1 to level 5 are d1=100, d2=150, d3=200, d4=120, and d5=80 respectively. If the access frequencies of a certain implementation rule at each level are f1=40, f2=60, f3=80, f4=30, and f5=50 respectively, then its priority is calculated as follows:

[0077]

[0078] The result shows that the priority of this implementation rule is 2.075, which means that this implementation rule has a higher priority among all implementation rules.

[0079] Specifically, if Figure 3 As shown, based on the implementation details access distribution information set, the grouped data is sorted, the hierarchical span is analyzed, the priority interval implementation details group order is adjusted, the implementation details arrangement order is optimized, and the steps of generating the implementation details cache priority distribution result are as follows:

[0080] S201: Based on the access distribution information set of the implementation details, the group data is compared item by item according to the access times, the access frequency differences between the groups are extracted, the access frequency increase trend of each group is sorted out, and the access times ranking table is generated by accumulating the access times;

[0081] By counting the access records of each implementation rule in the database, all implementation rules are arranged according to the number of accesses. By accurately measuring the access frequency of each implementation rule, it is ensured that the actual distribution of access volume can be accurately reflected. The data of each group is compared item by item to clarify the difference in access frequency between different groups. For example, for two different data groups A and B, the number of accesses of each implementation rule in group A will be recorded and compared with the number of accesses of the same implementation rule in group B. In this way, we can record in detail the difference in accesses of each implementation rule in different groups and analyze which implementation rules are particularly frequent or uncommon in a certain group. It is also necessary to pay attention to the increasing trend of the number of accesses within each group, that is, to observe the growth curve of the number of accesses in each group. This helps to understand the changing pattern of user behavior. Through detailed data analysis, a ranking table of accesses will eventually be obtained. The table lists the ranking of the implementation rules in all groups by the number of accesses, providing basic data for subsequent analysis.

[0082] S202: Based on the access frequency ranking table, the hierarchical span of each group is analyzed item by item, and the hierarchical range of the interval is extracted by comparing the accumulated access frequency in the group with the hierarchical span, and a priority interval division table is generated according to the interval hierarchical priority;

[0083] By accumulating the number of visits to each group, the total number of visits to each group can be obtained, and this total number of visits can be compared with its hierarchical span. The purpose of the comparison is to find out which hierarchical groups have a higher access density, which means that users have a higher demand for hierarchical data. On this basis, the hierarchical range will be extracted, and the relative importance of each level will be determined by accurately measuring the access density of each level. Based on this, the interval levels are further sorted according to their priorities to generate a priority interval division table. This table not only reflects the access priority of each level, but also provides a basis for data management and optimization, helping managers to allocate and optimize data resources more effectively.

[0084] S203: Based on the priority interval division table, rearrange the implementation rule combinations with high frequency in the interval and the implementation rule groups with low-level span, adjust the continuous sequence of the implementation rule groups, optimize the priority arrangement through the internal sequence of the interval, and generate the implementation rule cache priority distribution result;

[0085] The high-frequency implementation rule combinations in each interval are extracted and compared with the implementation rule groups of the low-level spans one by one. When rearranging the implementation rule combinations, the access frequency priority rule within the implementation rule group is adopted to arrange the high-frequency implementation rules at the front of the sequence. At the same time, adjustments are made according to the grouping characteristics of the low-level span to ensure that the implementation rules in the low-span group can form a continuous access sequence. By adjusting the order of the implementation rule groups within the interval, the priority arrangement of the entire interval meets the goal of distribution optimization. During the optimization process, the adjustment results within each interval are recorded, and the access frequency distribution of the implementation rules after adjustment and the priority number of the interval in which they are located are marked. Finally, the implementation rule cache priority distribution results are generated. While ensuring the orderliness of the distribution, the entire optimization result also provides a clear guiding basis for the actual implementation of the cache strategy, thereby improving the access efficiency of the distribution structure and the hierarchical optimization effect.

[0086] Specifically, if Figure 4 As shown, based on the cache priority distribution result of the implementation details, the data location of the cache area implementation details is adjusted, the remaining capacity implementation details are repartitioned, the partition capacity ratio is adjusted, and the steps of generating the cache partition adjustment configuration result are as follows:

[0087] S301: Based on the implementation rules cache priority distribution result, the storage location and associated capacity data of the high-priority implementation rules are extracted, the implementation rules capacity and the current distribution of the cache area are compared in groups, the storage requirements corresponding to each group of implementation rules are counted, and the distribution interval of the priority data is divided to obtain the priority implementation rules grouping and capacity annotation data;

[0088] By scanning the cache log, the set of high-priority implementation details is determined, the number of hits for each implementation detail is counted, the priority of the implementation details is calculated based on the number of hits and data distribution, and the implementation details data is grouped in combination with the storage requirements of the implementation details. During the grouping process, the grouping conditions are set based on the correlation between capacity and priority. After grouping, the capacity requirements of each group of implementation details are merged and calculated, and the priority information of each group of implementation details is marked to the corresponding group. After the priority grouping is completed, the distribution interval is divided. The interval division process is based on the gradient distribution of storage capacity. The difference method is used to calculate the gradient change points and determine the partition boundaries. The areas with smaller gradient changes are merged to reduce the fragmentation of the storage partitions. Finally, the priority implementation details grouping and capacity annotation data are output, and detailed statistical data on the grouping results and capacity requirements are provided for subsequent operations.

[0089] S302: Based on the priority implementation rule grouping and capacity annotation data, calculate the difference between the capacity occupied by the high-priority implementation rule data and the available capacity of the current cache area, compare the capacity of the low-priority implementation rule corresponding to the difference distribution, extract the distribution data that does not meet the capacity requirements, merge and classify the distribution data, and obtain the cache area capacity difference and remaining distribution data;

[0090] Calculate the difference between the capacity occupied by the high-priority implementation details data and the available capacity of the current cache area according to the formula:

[0091]

[0092] Calculate the capacity difference, where:

[0093] P k =α k ·S k

[0094] In the formula, ΔC represents the capacity difference, P k represents the total capacity requirement of the kth group of high priority implementation rules, C av represents the available capacity of the cache, α k is the implementation details hit weight factor, S k is the storage requirement of the kth group of implementation details;

[0095] S k By counting the total storage requirements of the implementation details of the kth group, the calculation is performed using direct data statistics method;

[0096] α k The result is obtained by analyzing the ratio of the number of hits of the implementation rules to the total number of hits through log analysis;

[0097] C av Get it directly from the allocation information of the cache;

[0098] Assume that the total capacity of the cache is 1000 and the current used capacity is 700, then:

[0099] Available Capacity: C av =1000-700=300;

[0100] Assume that there are two groups of high-priority implementation rules: Group 1 S1 = 200, hit rate weight α1 = 0.8; Group 2 S2 = 150, hit rate weight α2 = 0.7;

[0101] Calculate the requirements for each set of implementation details: P1 = 0.8 200 = 160, P2 = 0.7 150 = 105;

[0102] Calculate the total capacity requirement and the difference:

[0103] ΔC=(160+105)-300=-35

[0104] The result shows that the capacity difference is -35, which means that the available capacity of the current cache area can meet the storage requirements of the high-priority implementation rules, but further optimization still requires adjustments to the low-priority implementation rules. The calculation of the capacity difference provides a parameter basis for subsequent merging classification and storage adjustment.

[0105] S303: Based on the cache capacity difference and the remaining distribution data, identify the adaptation parameters of the remaining distribution capacity and the low-speed cache area, re-divide the remaining capacity units, adjust the capacity ratio of the unit partitions, reorganize the storage of the allocated area, and establish the cache partition adjustment configuration result;

[0106] The storage capacity in the distribution data and the maximum adaptation value of the low-speed cache unit in the partition are called, and the capacity optimization is achieved by gradually adjusting the partition capacity ratio. At the same time, a dynamic adjustment algorithm is used to analyze the distribution structure of the remaining capacity. When the remaining capacity units are redivided, a storage mapping table is constructed according to the similarity of storage requirements between units. The mapping table is used to implement the allocation priority sorting of storage capacity, and matching adjustments are made in combination with the adaptation parameters of the low-speed cache. The balance of the partition capacity distribution is optimized through the storage reorganization strategy, while ensuring that the resources in the storage area are fully utilized. During the storage reorganization process, the storage unit is iteratively adjusted multiple times, and the gradient change value of the remaining capacity is used as the adjustment boundary. The capacity ratio configuration is adjusted through the boundary dynamic optimization algorithm, and finally the cache partition adjustment configuration result is generated.

[0107] Specifically, if Figure 5 As shown in the figure, based on the cache partition adjustment configuration result, the steps of recording the read and write operations in the partition, counting the read and write frequency, analyzing the read and write delay time, integrating the delay data and hit records, and generating the partition read and write efficiency data set are as follows:

[0108] S401: Based on the cache partition adjustment configuration result, read the read and write records in the partition, analyze the operation timestamp and data source, calculate the time consumption of the read and write operations, perform classification statistics of the read and write frequencies, compare the cache hit counts, and obtain the partition read and write and hit statistics;

[0109] By analyzing the operation timestamp of each record, the occurrence time of the read and write operations is extracted, and the records are sorted in chronological order to ensure that the data flow process is clear. When analyzing the timestamp and data source, the cache data block corresponding to each operation is matched with the external data source to determine whether the data is obtained from the cache or read from the external storage, and the operation time is recorded separately. Through the recorded read and write time data, it is classified and counted according to the time interval, so as to clarify the frequency distribution of read and write operations in different time periods. When counting the read and write frequency, the read and write operations are divided into three categories: high frequency, medium frequency and low frequency. The total number of operations and average time consumption under each frequency category are counted separately, and the specific number of cache hits is recorded. Through the comparative analysis of the number of cache hits, the impact of the cache on the data access efficiency in the partition is further clarified, and finally the partition read and write and hit statistics are obtained, which provides a basis for the subsequent in-depth analysis of the delay time and hit rate.

[0110] S402: Based on the partition read / write and hit statistics, extract the operation time-consuming data, classify by segment according to the read / write frequency, calculate the delay time ratio, compare the migration frequency relationship, and combine the cache hit record to obtain the partition delay and hit integration data;

[0111] High-frequency operations are counted separately, and their corresponding total and average time consumption are calculated. At the same time, medium-frequency and low-frequency operations are grouped separately to form a time consumption distribution table under different frequency categories. When calculating the proportion of delay time, the total time consumption is compared with the overall time consumption of all operations in the partition to clarify the proportion of each type of operation in the overall delay time. When analyzing the relationship between migration frequencies, the number of migrations and hits of high-frequency operations is combined to compare the difference between the migration time consumption and the cache hit time consumption, from which the degree of influence of the migration operation on the partition delay is derived. Combined with the cache hit records, the distribution law of the hit rate under different frequency categories is further analyzed, so as to obtain the integrated data of partition delay and hit, and provide a detailed basis for optimizing the partition delay and hit performance.

[0112] S403: Based on the partition latency and hit integration data, extract the read and write time consumption data in the partition, group them by operation frequency, calculate the average time consumption of each group, associate the cache hit rate, integrate the partition data, and build a partition read and write efficiency data set;

[0113] The formula for calculating the average time spent in each group is as follows:

[0114]

[0115] in, represents the weighted average time consumed by group i, w i represents the weight value of the i-th group of operating frequencies, T i represents the time taken for a single read or write operation of the i-th group, and n represents the number of data groups;

[0116] Detailed explanation of the formula and the process of formula calculation and derivation:

[0117] This formula is used to calculate the weighted average time consumption, which is calculated by weighting the frequency of each group of operations and the corresponding time consumption. The result can reflect the average time consumption of each group of operations and provide a reference for subsequent performance optimization;

[0118] w i : The weight value of each group of operation frequencies indicates the proportion of this group of operations in the total operations. The weight value is set based on the actual monitoring data of each operation frequency. Specifically, it is the ratio of the number of executions of this group of operations during system operation to the total number of operations. It is obtained through data monitoring during actual operation. For example, if a group of operations is executed 200 times, accounting for 20% of the total 1000 operations, its weight is 0.2.

[0119] T i : The single read and write time of each group of operations, which indicates the time required to execute the group of operations once, in milliseconds. This value is obtained through actual performance testing or data collection. The average time of a group of operations is set to 15 milliseconds, which is obtained through measurement or calculation;

[0120] n: number of data groups, indicating how many groups all operations are divided into. Assuming that they are divided according to operation category, frequency, etc., 5 different operation groups are obtained;

[0121] Assume there are 5 operation groups, the operation frequency and time consumption are:

[0122] Group 1: w1 = 0.2, T1 = 12 ms;

[0123] Group 2: w2 = 0.3, T2 = 18 ms;

[0124] Group 3: w3 = 0.1, T3 = 8 ms;

[0125] Group 4: w4 = 0.25, T4 = 22 ms;

[0126] Group 5: w5 = 0.15, T5 = 16 ms;

[0127] The calculation process of the formula is:

[0128]

[0129] The result of 16.5 milliseconds indicates the weighted average duration of all operation groups, which combines the execution frequency of each operation group and the duration of each operation. In this example, the result reflects the weighted calculation result of the duration of each group in the five operation groups, indicating the overall average read and write duration. This value can be used as an important reference for performance and further help analyze which operations need to be optimized.

[0130] Specifically, if Figure 6 As shown, based on the partition read and write efficiency data set, the read and write frequencies of high-frequency implementation details are compared, the partition data order is rearranged, the implementation details arrangement order is adjusted, and the partition capacity is iteratively allocated to generate the optimized cache partition structure. The specific steps are:

[0131] S501: Based on the partition read-write efficiency data set, collect the read-write data of the high-frequency implementation details in the partition, extract the read-write frequency and cache hit rate of each partition through the data access log, parse the data access entries in the log one by one and summarize the time series statistics to generate a partition performance table;

[0132] First, collect the high-frequency implementation details read and write data within the partition, parse it one by one by calling the partition data access log, extract the read and write frequency data of each partition, and extract the cache hit rate and establish a time series statistical model. The time series called in the statistical model needs to arrange the data points in chronological order of access time, and use statistical analysis methods such as data distribution histograms to summarize and calculate the frequency, mark the key read and write operations in the time series as high-frequency access points, and mark the high-frequency points based on the access frequency that is more than twice the average value. Finally, output the statistical results of each partition to the partition performance table, and generate the partition performance table through the partition read and write data associated with the data table and the access time interval distribution of the high-frequency points extracted from the time series.

[0133] S502: using the partition performance table, analyzing the performance differences between partitions, analyzing the read and write frequencies and hit rates between partitions to set performance thresholds, gradually adjusting the boundary positions and analyzing the access efficiency within the partitions, cross-validating the boundary configuration, and generating a new partition boundary map;

[0134] Analyze the read and write frequency and hit rate between partitions to set performance thresholds according to the formula:

[0135]

[0136] Calculate the performance difference threshold between partitions;

[0137] Where E represents the performance difference threshold, R i represents the read and write frequency of the i-th partition, R avg is the average read and write frequency of all partitions, Q is the total number of partitions;

[0138] R i The reading and writing frequency of each partition is calculated by counting the partition log data;

[0139] R avg The result is obtained by dividing the sum of read and write frequencies of all partitions by the number of partitions, ensuring that the baseline level of overall performance can be quantified;

[0140] Q is the total number of partitions, which can be directly obtained through the system setting value;

[0141] Assume that there are 4 partitions with read and write frequencies of 200, 250, 300, and 350 respectively. Then calculate the average read and write frequency:

[0142]

[0143] Calculate the performance difference threshold:

[0144]

[0145] E=3125

[0146] The result shows that the standard value of partition performance difference is 3125, through which the performance optimization space between different partitions can be determined and boundary adjustment can be performed.

[0147] S503: Allocate cache partition capacity according to the new partition boundary map, determine the capacity size using the partition read-write ratio and load distribution, optimize the capacity by dynamically adjusting the capacity and analyzing the access response time, and generate an optimized cache partition structure;

[0148] In the process of allocating cache partition capacity, it is necessary to call the partition's read-write ratio data and calculate the initial capacity of each partition based on the load distribution. The load distribution data comes from the access frequency and read-write offset time points counted in the log. The capacity is dynamically adjusted based on the access response time. By performing a difference operation on the partition's access response time series, the change range of the difference is calculated as a reference for capacity optimization. If the response time deviation is greater than the set optimization threshold, the capacity size is increased by the step value and optimized again to finally obtain the optimized cache partition structure.

[0149] Specifically, if Figure 7 As shown, based on the optimized cache partition structure, the steps of recording the implementation details of new access requests, counting the partition matching data, integrating the new implementation details information to the partition boundary, and generating the dynamic adjustment plan of the implementation details cache are as follows:

[0150] S601: Based on the optimized cache partition structure, extract the feature value of the implementation details and combine it with the distribution data of the implementation details set in the partition, compare the implementation details information with the matching rules of the implementation details in the partition, calculate the matching hit index data and count the matching distribution, and obtain the partition matching hit data;

[0151] The formula for calculating the matching hit index data is as follows:

[0152]

[0153] Among them, Mhit Indicates the matching hit indicator data, P d

[0154] Characteristic value indicating the implementation details of the cache, represents the feature value that matches the implementation rules in the partition, F represents the total number of implementation rule sets, represents the absolute error between the implementation rule feature value in the cache and the matching feature value of the implementation rule in the partition, and α is the error w f is the weight coefficient of the characteristic value of the implementation rules, β is the weight adjustment coefficient of the characteristic value of the implementation rules, and w d

[0155] is the weight coefficient of the matching degree of the implementation rules within the zone, and γ is the matching degree adjustment coefficient;

[0156] This formula is used to calculate the data index of partition matching hits. The final result is used to evaluate the efficiency and accuracy of implementation rule matching within the cache partition. The various parameters in the formula are obtained through the data collection and calculation process in the cache management system to quantify the accuracy of implementation rule matching, the impact of implementation rule characteristics, and the degree of partition matching;

[0157] P d Indicates the characteristic value of the implementation details in the cache. The setting value is the weighted average of the length of the implementation details and the partition matching degree. The frequency and distribution of the implementation details are calculated by the cache monitoring system. It is usually set to 128 bytes, reflecting the size of the implementation details feature;

[0158] Indicates the feature value that matches the implementation details in the partition. The set value is the weighted value of the matching implementation details in the cache partition. The calculation method is based on the average value of the feature values ​​of all implementation details in the cache partition. According to the actual measured data, it is set to 120 bytes;

[0159] α is the error adjustment coefficient, which indicates the impact of the implementation details feature error on the matching hit rate. The coefficient is set to 2.5. It is mainly calculated based on the error tolerance between the implementation details features and the system's tolerance for errors, reflecting the sensitivity to the impact of implementation details errors.

[0160] w f is the weight coefficient of the characteristic value of the implementation details, which reflects the importance of the implementation details in the cache system. It is obtained by monitoring the frequency of each implementation detail through the cache system. This value is positively correlated with the frequency of the implementation details. In the case of high frequency, the weight coefficient is set to 1.2 to reflect the weighted influence of high-frequency implementation details.

[0161] β is the implementation details feature weight adjustment coefficient, which is used to adjust the contribution of frequently appearing implementation details to the matching hit rate. The calculation basis is the frequency of the implementation details. The setting value is 1.4, which means that the implementation details with higher frequency contribute more to the matching hit rate.

[0162] w d is the weight coefficient of the matching degree of implementation rules within the partition, reflecting the impact of the matching degree of implementation rules on the hit rate. The weight coefficient is calculated based on the matching situation of implementation rules within the partition and is set to 1.5, which means that the partition matching degree has a strong impact on the hit index;

[0163] γ is the matching adjustment coefficient, which is set to 2.0. It mainly adjusts the hit index according to the matching degree between the implementation details in the partition and the implementation details in the cache. When the value is large, the implementation details with high matching degree have a greater impact on the final hit data;

[0164] F represents the total number of implementation rules, which is collected through the cache monitoring system, and the total number of implementation rules is set to 10,000;

[0165] Substitute the parameters into the formula for calculation:

[0166]

[0167]

[0168] The matching hit data is 0.0583, which indicates the hit rate of the implementation rule matching in a given cache partition, further illustrating the matching accuracy of the implementation rules in the cache area and the sensitivity of the cache system to the characteristics of the implementation rules. A lower matching hit rate (such as 0.0583) may indicate that the matching of the implementation rules in the cache partition is poor, and further optimization is needed to improve the matching efficiency and reduce the frequency of false matches.

[0169] S602: Based on the partition matching hit data, the hit ratio of each partition is calculated and the deviation between the hit and the capacity is analyzed, the partition boundary is adjusted to reallocate the partition capacity, the capacity of each partition implementation rule set is updated and the boundary is redefined, and the partition capacity distribution optimization data is obtained;

[0170] By comparing the total number of recorded access requests with the number of hits, the hit situation of each partition is obtained. For partitions with low hit ratios, the hit data is compared with the current capacity in detail to analyze whether the hit deviation is caused by insufficient capacity or uneven capacity distribution. When adjusting the partition boundary, the partition capacity distribution range is redefined based on the relationship between the partition hit ratio and the access popularity, giving priority to increasing the capacity of partitions with high hit rates and high access volume, and reducing the capacity of partitions with low hit rates. By updating the implementation rules set for each partition, the newly added capacity is allocated to the corresponding partition, and the implementation rules in the implementation rules set that exceed the new boundary range are removed or transferred. After the boundary is redefined, the capacity and boundary data of each partition are recorded in the optimization log to ensure that the partition capacity distribution optimization data is complete and traceable. The partition capacity distribution optimization data is obtained to lay the foundation for improving the overall hit efficiency.

[0171] S603: Based on the partition capacity distribution optimization data, the implementation details information of the new access request is merged into the corresponding partition according to the adjusted partition boundary, the implementation details set in the partition is added and updated, the current state of the partition boundary is recorded, and the implementation details cache dynamic adjustment plan is generated;

[0172] Match the implementation details of new access requests with the adjusted partition boundaries one by one, merge the implementation details into the corresponding partitions according to the partition boundary range, check whether the same implementation details already exist in the merged partitions, perform new operations on the implementation details that do not appear, and assign them to the appropriate position of the partition, and update the dynamic information such as the number of accesses and the most recent access time for the existing implementation details to ensure that the implementation details collection can reflect the current access status. After completing the addition and update operations, record the current status of the partition boundary, including partition capacity, number of implementation details, boundary range and other information, organize the information into partition dynamic adjustment logs, provide data support for subsequent optimization, generate a dynamic adjustment plan for the implementation details cache, optimize the efficiency of cache resource utilization within the partition, and ensure that the cache can maintain efficient operation in a dynamic environment.

[0173] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for optimizing LSM engine data cache, characterized in that: The following steps are involved: Based on the cache area data of the LSM storage engine, record the number of accesses to the implementation details, count the distribution density of the implementation details, organize the data into priority sequences, and generate a set of implementation details access distribution information; Based on the implementation rule access distribution information set, the grouped data is sorted, the hierarchical span is analyzed, the priority interval implementation rule group order is adjusted, the implementation rule arrangement order is optimized, and the implementation rule cache priority distribution result is generated; Based on the cache priority distribution result of the implementation details, adjust the data location of the cache area implementation details, repartition the remaining capacity implementation details, adjust the partition capacity ratio, and generate a cache partition adjustment configuration result; Based on the cache partition adjustment configuration result, record the read and write operations in the partition, count the read and write frequencies, analyze the read and write delay time, integrate the delay data and hit records, and generate a partition read and write efficiency data set; Based on the partition read-write efficiency data set, the read-write frequency of the high-frequency implementation details is compared, the partition data order is rearranged, the implementation details arrangement order is adjusted, the partition capacity is iteratively allocated, and an optimized cache partition structure is generated; Based on the optimized cache partition structure, the implementation details information of new access requests is recorded, the partition matching data is counted, the new implementation details information is integrated into the partition boundary, and the implementation details cache dynamic adjustment plan is generated.

2. The LSM engine data cache optimization method according to claim 1, characterized in that: The implementation details access distribution information set includes the implementation details access count statistics, storage hierarchy distribution position, and implementation details distribution density data; the implementation details cache priority distribution results include priority interval demarcation, implementation details group arrangement order, and continuous sequence distribution optimization structure; the cache partition adjustment configuration results include implementation details cache area distribution, partition capacity allocation ratio, and low-speed cache area remaining capacity division; the partition read and write efficiency data set includes partition read and write frequency data, partition read and write delay time statistics, and cache hit and migration frequency ratio; the optimized cache partition structure includes partition boundary data arrangement, implementation details partition capacity ratio demarcation, and partition optimized data order; the implementation details cache dynamic adjustment plan includes new access implementation details matching partition data, new implementation details boundary information integration, and partition implementation details capacity adjustment records.

3. The LSM engine data cache optimization method according to claim 1, characterized in that: Based on the cache area data of the LSM storage engine, the steps of recording the number of accesses to the implementation details, counting the distribution density of the implementation details, arranging the data into a priority sequence, and generating a set of implementation details access distribution information are as follows: Based on the cache area data of the LSM storage engine, the data implementation rules, corresponding access times and storage level information are captured from the LSM storage engine, the storage records are traversed and the key information of each implementation rule is extracted, including the implementation rule name, access count and level identifier, to obtain the implementation rule access record table; Filtering data from the implementation rules access record table, comparing the level and access times of each implementation rule, counting the access frequency of each level, and classifying the data implementation rules by analyzing the differences in the number of accesses between levels to obtain a level access difference table; The hierarchical access difference table is used to sort data, and the implementation details are prioritized by setting a threshold of distribution density, and implementation details with large distribution density are given priority to obtain an implementation details access distribution information set.

4. The LSM engine data cache optimization method according to claim 1, characterized in that: Based on the implementation rule access distribution information set, the grouped data is sorted, the hierarchical span is analyzed, the priority interval implementation rule group order is adjusted, the implementation rule arrangement order is optimized, and the steps of generating the implementation rule cache priority distribution result are specifically as follows: Based on the access distribution information set of the implementation details, the group data is compared item by item according to the access times, the access frequency differences between the groups are extracted, the access frequency increase trend of each group is sorted out, and the access times ranking table is generated by accumulating the access times; Based on the access frequency ranking table, the hierarchical span of each group is analyzed item by item, and the hierarchical range of the interval is extracted by comparing the accumulated access frequency in the group with the hierarchical span, and a priority interval division table is generated according to the interval hierarchical priority; Based on the priority interval division table, the implementation rule combinations with high frequency in the interval are rearranged with the implementation rule groups with low-level spans, the continuous sequence of the implementation rule groups is adjusted, the priority arrangement is optimized through the internal order of the interval, and the implementation rule cache priority distribution result is generated.

5. The LSM engine data cache optimization method according to claim 1, characterized in that: Based on the cache priority distribution result of the implementation details, the steps of adjusting the data location of the cache area implementation details, repartitioning the remaining capacity implementation details, adjusting the partition capacity ratio, and generating the cache partition adjustment configuration result are specifically as follows: Based on the implementation rules cache priority distribution result, the storage location and associated capacity data of high-priority implementation rules are extracted, the implementation rules capacity and the current distribution of the cache area are compared in groups, the storage requirements corresponding to each group of implementation rules are counted, and the distribution interval of the priority data is divided to obtain the priority implementation rules grouping and capacity annotation data; Based on the priority implementation rule grouping and capacity annotation data, calculate the difference between the capacity occupied by the high-priority implementation rule data and the available capacity of the current cache area, compare the capacity of the low-priority implementation rule corresponding to the difference distribution, extract the distribution data that does not meet the capacity requirements, merge and classify the distribution data, and obtain the cache area capacity difference and remaining distribution data; Based on the cache capacity difference and the remaining distribution data, the adaptation parameters of the remaining distribution capacity and the low-speed cache area are identified, the remaining capacity units are re-divided, and the capacity ratio of the unit partitions is adjusted, the storage of the allocated area is reorganized, and the cache partition adjustment configuration result is established.

6. The LSM engine data cache optimization method according to claim 1, characterized in that: Based on the cache partition adjustment configuration result, the steps of recording the read and write operations in the partition, counting the read and write frequency, analyzing the read and write delay time, integrating the delay data and the hit record, and generating the partition read and write efficiency data set are specifically as follows: Based on the cache partition adjustment configuration result, read the read and write records in the partition, analyze the operation timestamp and data source, calculate the time consumption of the read and write operations, perform classification statistics of the read and write frequencies, compare the number of cache hits, and obtain the partition read and write and hit statistics; Based on the partition read / write and hit statistics, extract the operation time-consuming data, classify by segment according to the read / write frequency, calculate the delay time ratio, compare the migration frequency relationship, and combine the cache hit records to obtain the partition delay and hit integration data; Based on the partition delay and hit integration data, the read and write time consumption data within the partition is extracted, grouped according to the operation frequency, the average time consumption of each group is calculated, the cache hit rate is associated, the partition data is integrated, and a partition read and write efficiency data set is constructed.

7. The LSM engine data cache optimization method according to claim 6, characterized in that: The formula for calculating the average time consumption of each group is as follows: in, represents the weighted average time consumed by group i, w i represents the weight value of the i-th group of operating frequencies, T i represents the time taken for a single read or write operation of the i-th group, and n represents the number of data groups.

8. The LSM engine data cache optimization method according to claim 1, characterized in that: Based on the partition read-write efficiency data set, the read-write frequency of high-frequency implementation details is compared, the partition data order is rearranged, the implementation details arrangement order is adjusted, and the partition capacity is iteratively allocated to generate the optimized cache partition structure. Specifically, the steps are as follows: Based on the partition read-write efficiency data set, collect the read-write data of the high-frequency implementation details in the partition, extract the read-write frequency and cache hit rate of each partition through the data access log, parse the data access entries in the log one by one and summarize the time series statistics to generate a partition performance table; The partition performance table is used to analyze the performance differences between partitions, analyze the read and write frequencies and hit rates between partitions to set performance thresholds, gradually adjust the boundary positions and analyze the access efficiency within the partitions, cross-validate the boundary configuration, and generate a new partition boundary map; According to the new partition boundary map, cache partition capacity is allocated, the capacity size is determined by using the partition read-write ratio and load distribution, and the capacity is optimized by dynamically adjusting the capacity and analyzing the access response time to generate an optimized cache partition structure.

9. The LSM engine data cache optimization system according to claim 1, characterized in that: Based on the optimized cache partition structure, the steps of recording the implementation details of new access requests, counting the partition matching data, integrating the new implementation details information to the partition boundary, and generating the dynamic adjustment plan of the implementation details cache are as follows: Based on the optimized cache partition structure, extract the characteristic value of the implementation details and combine it with the distribution data of the implementation details set in the partition, compare the implementation details information with the matching rules of the implementation details in the partition, calculate the matching hit index data and count the matching distribution, and obtain the partition matching hit data; Based on the partition matching hit data, the hit ratio of each partition is calculated and the deviation between the hit and the capacity is analyzed, the partition boundary is adjusted to reallocate the partition capacity, the capacity of each partition implementation rule set is updated and the boundary is redefined, and the partition capacity distribution optimization data is obtained; Based on the partition capacity distribution optimization data, the implementation details information of the new access request is merged into the corresponding partition according to the adjusted partition boundary, the implementation details set in the partition is added and updated, the current status of the partition boundary is recorded, and a dynamic adjustment plan for the implementation details cache is generated.

10. The LSM engine data cache optimization system according to claim 9, characterized in that: The formula for calculating the matching hit index data is as follows: Among them, M hit Indicates the matching hit indicator data, P d Characteristic value indicating the implementation details of the cache, represents the feature value that matches the implementation rules in the partition, F represents the total number of implementation rule sets, represents the absolute error between the implementation rule feature value in the cache and the matching feature value of the implementation rule in the partition, α is the error adjustment coefficient, and w f is the weight coefficient of the characteristic value of the implementation rules, β is the weight adjustment coefficient of the characteristic value of the implementation rules, and w d is the weight coefficient of the matching degree of the implementation rules within the partition, and γ is the matching degree adjustment coefficient.

Images