Data lake cache optimization method based on data correlation analysis

By mining data access, semantic and metadata correlation in the data lake, and using machine learning to make cache decisions, the problem of traditional cache algorithms ignoring data correlation in multi-source heterogeneous data environments is solved, and cache access efficiency is improved. Hit rate reduces storage costs.

CN119046328APending Publication Date: 2024-11-29ZHEJIANG UNIV OF TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411188891.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Traditional cache eviction algorithms ignore the correlation between data in multi-source heterogeneous data environments in data lakes, resulting in low cache access efficiency and inability to make full use of the rich information in the data lakes.

Method used

Through the data processing module, the access, semantic and metadata correlation between data is mined, and the cache management module is used to make cache decisions based on machine learning. The frequent access object, semantic related object and metadata related object mining technology is used to generate related object groups. And group eviction and cache policy optimization are carried out in cache management.

Benefits of technology

Improve cache hit rate, make full use of data lake information, improve data access performance, and reduce storage costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046328A_ABST
    Figure CN119046328A_ABST
Patent Text Reader

Abstract

A data lake cache optimization method based on data correlation analysis comprises a data monitoring and processing module, a cache management module and a cache information module. And the data monitoring and processing module is used for recording an access sequence of the data, mining access correlation and semantic correlation among the data and metadata correlation of a data lake, and periodically sending the mined data to the cache information module. The cache information module is used for storing current data correlation information. And the cache management module is used for acquiring the correlation information from the cache information module according to the request content, grouping the data by using the correlation information, and making a cache decision on a group level based on machine learning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a data lake cache optimization method based on data correlation analysis. Background Art

[0002] With the development of big data technology, data lakes have been widely used as a new type of data storage and management solution. Data lakes can store massive amounts of data from different sources and provide efficient data analysis and processing capabilities. However, due to the openness of data lakes and the rapid growth of data volume, how to optimize data access efficiency has become an important issue. In order to improve the access performance of data lakes, caching technology is widely used. Traditional cache eviction algorithms often ignore the correlation between data, especially in a complex environment such as a data lake, where data from different sources often have potential correlations. Although the existing time-grouping-based caching technology performs well in cache management, it still has room for improvement when processing multi-source heterogeneous data in data lakes. Therefore, this invention proposes a cache strategy improvement solution based on the correlation of multi-source heterogeneous data in a data lake. By analyzing and mining the access, semantic and metadata correlations between data, the invention can more accurately predict which data groups should be retained in the cache. This method can not only improve the cache hit rate, but also make full use of the rich information in the data lake in data access. In addition, due to the large amount of data in the data lake, the use of high-storage-density SMR (Shingled Magnetic Recording) hard disk technology can effectively reduce storage costs. At the same time, the cache design based on high-performance SSD (solid-state drive) can further accelerate data access and improve the overall performance of the system. Summary of the invention

[0003] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a data lake cache optimization method based on data correlation analysis, which includes a data processing module, a cache management module and a cache information module.

[0004] The data processing module is used to record the access sequence of data, mine the access correlation, semantic correlation and metadata correlation of data lake, and regularly send the mined data to the cache information module. The cache information module is used to store the current data correlation information. The cache management module is used to obtain correlation information from the cache information module according to the request content, group the data using the correlation information, and make cache decisions at the group level based on machine learning.

[0005] In order to solve the above technical problems, the solution provided by the present invention is as follows:

[0006] A data lake cache optimization method based on data correlation analysis includes the following steps:

[0007] (1) Mining the correlation data between data and storing them in the cache information module.

[0008] (1-1) Data collection and preparation: Collect cache access sequences to mine access correlations between data.

[0009] (1-2) Frequently accessed object mining: Analyze the access sequence and use FCM to mine objects that are frequently accessed together from the access sequence.

[0010] FCM sets multiple constraints to evaluate the strength of the relationship between data. The first is the search radius. FCM will only search for related objects of a given object within the search radius. The second is the search depth. If you are searching for related objects of object m, if the search depth is 10, and object m appears N (N>10) times in the access sequence, then FCM will only check the 10 most recent occurrences of the object in the access sequence. The third is support and confidence. Using the association rule x→y to indicate that object x is accessed, then y will also be accessed within the search radius of object x. If object x appears N times in the access sequence, and object y appears M times within the search radius of object m, then the confidence of the association rule x→y is M / N and the support is M. The higher the confidence, the higher the correlation between the two objects, and the higher the support, the higher the repetitiveness of the two objects. Assume that a related group C has minimum support and minimum confidence restrictions, it must satisfy the following rules: For all C i ∈C and C j ∈C, association rule C i →C j and C j →C i The confidence and support of are not less than the minimum confidence and minimum support.

[0011] The search process of FCM is as follows: 1. FCM scans the access sequence to count the frequency of each object, determines the appropriate search depth and deletes objects with low access frequency. 2. FCM scans the access sequence to generate candidate association rules. Whenever object j appears in the viewing circle of object i, FCM increases the support of association rule i→j by 1. 3. FCM generates correlation groups according to the group definition. First, association rules that do not meet the minimum support and minimum confidence are deleted, and then each object is scanned to find potential groups according to the association rules. For example, for object a, it has association rules a→b and a→c. FCM checks the association rules in descending order of confidence value. For association rule a→b, if b→a also exists, the two objects are divided into a group {ab}. Then, FCM continues to check a→c. Object c can be added to group {ab} if and only if association rules c→a, c→b and b→c also exist. This is based on the definition of the group. Any two objects in the group should be related to each other.

[0012] (1-3) Mining semantically related objects: Obtain the semantic information of objects and calculate the correlation between objects based on the semantic information. For video, audio, and image file types, first convert these files into text data, use the Whisper model to convert video and audio into text, use the convolutional neural network model to understand the image content, and obtain the text data of the image. Then build a Doc2Vec model to characterize the correlation between texts. Specifically, it includes the following steps:

[0013] First, data preprocessing is done to convert objects such as video, audio, and images into text.

[0014] The Doc2Vec model is then used to infer the vector representation of each text.

[0015] Next, we calculate the relevance of the text. After obtaining the vector representation of each text, we use cosine similarity to calculate the relevance score between samples. The calculation formula is as follows:

[0016]

[0017] (1-4) Metadata-related object mining: Obtain the metadata information of the object and calculate the correlation between the objects based on the metadata information. The invention uses the path information in the metadata information and uses a path segmentation algorithm to calculate the path correlation. First, calculate the maximum value of the subdirectories of path A and path B as the denominator, and then calculate the intersection of the directories of A and B as the numerator. If A = / home / user1 / paper / a, B = / home / user1 / paper / b, then max(A,B) = 4, A∩B = 3, sim(A,B) = 3 / 4. The calculation formula is as follows:

[0018]

[0019] (1-5) Generation of correlation information: Generate the final group of related objects based on the information obtained in steps (1-2) (1-3) (1-4), and store the data in the cache information module. First, based on the frequently accessed object group, the semantic correlation and metadata correlation of the objects in the group are calculated to obtain the total correlation between the objects. The invention represents the weight of the correlation as ω. Assuming that object B is an element in the frequently accessed object group of object A, the semantic correlation and metadata correlation of object A and object B are mined respectively. If object A and object B are both semantically correlated and metadata correlated, it means that the correlation weight of object A and object B is ω=α. If only semantic correlation or metadata correlation exists, the correlation weight is ω=β, otherwise ω=γ.

[0020] (2) The cache management module uses the cache information module to manage cache data. Specifically, it includes the following steps:

[0021] (2-1) Cache eviction algorithm: Use group eviction to remove data from the cache to make room for new data. Specifically, it includes the following steps:

[0022] 1. Group utility: Calculate group utility to measure the subsequent impact of evicting or retaining an object group. Object groups with lower utility are more likely to be evicted. The invention defines the utility of an object as the ratio of cost to benefit, where the cost is a miss and the benefit is the freed space multiplied by the time until the next request.

[0023]

[0024] Where T " (t) represents the time interval between the last request and the next request, s " Indicates the size of the object.

[0025] Therefore, the utility of the group of objects is the sum of the utilities of the individual objects.

[0026]

[0027] 2. Feature and model selection: The invention selects the following features for learning: the total number of requests for each object in the group, the number of frequently accessed objects in the group, the write rate when inserting into the cache, the miss rate of the cache when inserting, the request rate of the cache when inserting, the average object size in the group, and the age of the group. If the group input to the model belongs to the Correlation Cache part, a feature will be added: the correlation of the group. The invention uses the gradient boosting tree to design a regression algorithm as the objective function.

[0028] 3. Training: The algorithm uses the training data collected online to train the model and generates new training data by sampling the cached object groups. Specifically, it includes the following steps:

[0029] Sampling: Assuming there are M groups in the cache, and N groups need to be sampled each time, the step size is M / N, and one group is sampled every other step size. The utility of the sampled group is initialized to 0 at the beginning. When an object o in the sampled group is requested, the algorithm calculates the utility of the object using formula (3) and adds the utility to the group utility.

[0030] Data processing: Convert the features of each group in the sampling into a feature matrix as the X value of the model input, and use the utility of the sampling group as the Y value of the model input. Assume a group G i , there are n features in the group, denoted as x ij , for group G i , whose corresponding utility value is y i , then the characteristic matrix and utility value of group G can be expressed as:

[0031]

[0032] If the sampling contains k groups, each group G i Corresponding to a feature matrix X i and utility value Y i The final model input is:

[0033]

[0034] Initialize model: Initialize the model's initial prediction value, which is set to the average of the sample group's utility values. The calculation formula is:

[0035]

[0036] in is the initial prediction value, N is the number of samples, y i is the utility value.

[0037] Define the loss function: For regression problems, use MSE as the loss function to measure the gap between the predicted value and the actual value. The calculation formula is:

[0038]

[0039] Calculate gradients and second-order derivatives: For each iteration, calculate the gradient g of the current model prediction value relative to the loss function i and the second-order derivative h i The calculation formula is:

[0040]

[0041] where g i and h i are the gradient and second-order derivative of the i-th sample, respectively, and t represents the current iteration round.

[0042] Building a decision tree: Using the gradient g i and the second-order derivative h i To build a new decision tree. The weight ω0 of each leaf node of the new tree can be obtained by optimizing the objective function:

[0043]

[0044] Among them, λ is the regularization parameter used to prevent the model from overfitting, and i∈j means that sample i belongs to leaf node j.

[0045] Update model: Add the prediction results of the new decision tree to the prediction values ​​of the existing model. The calculation formula is:

[0046]

[0047] Among them, f . (x) is the output of the newly constructed decision tree.

[0048] The gradient and second-order derivative calculations, decision tree construction, and model update are repeated until a predetermined number of iterations is reached or the model converges.

[0049] 4. Inference: When the cache needs to be evicted, it predicts the group utility of all objects, sorts these groups, and evicts the N groups of objects with the lowest utility after merging them.

[0050] 5. Eviction: This invention uses merging-based eviction. Each time an eviction occurs, the least useful N groups of objects are selected for merging. During the merging process, M objects are retained and all other objects are expelled. The retained objects form a new group. The object selection during the merging process uses a simple metric based on the age and size of the object to calculate the utility of a single object, i.e. Where age is the time since the last access. If the Correlation Cache is merged, first based on Calculate the object with the greatest utility, and then obtain the weight of the relevance between other objects in the group and this object through the cache information module, and then Sorting is performed to determine the objects that are ultimately retained. The characteristics of the group generated after merging are averaged over the N groups.

[0051] (2-2) The cache management module receives the content request sent by the user and determines what kind of data request it is.

[0052] Specifically, it includes the following:

[0053] 1. If it is a write request, determine whether the requested content is in the cache information module. The data in the cache information module is generated regularly by step (1). Choose whether to put the data into the Time Cache part or the Correlation Cache part based on the judgment result. If the data in the request exists in the cache information module, obtain the relevant group of the data from the cache information module, and determine whether each data in the relevant group exists in the Correlation Cache part. If so, insert the data into the same group and add the correlation of the two objects to the group correlation. If not, create a new data group and insert it. If the requested data does not exist in the cache information module, write the data into the Time Cache part in sequence.

[0054] 2. If it is a read request, first determine whether the requested content is in the cache information module. If the requested data exists in the cache information module, first obtain the data from the Correlation Cache. If the data is successfully obtained, the response is successful. If the data cannot be obtained from the Correlation Cache, obtain the data from the Time Cache. If the data is successfully obtained, the response is successful. Otherwise, query the data from the data lake and respond to the data. At the same time, start a background thread to query whether the data exists in the cache information module. If so, all the data in the cache information module will be pre-fetched into the cache. If not, the data will only be inserted into the cache.

[0055] A second aspect of the present invention relates to a data lake cache optimization device based on data correlation analysis, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a data lake cache optimization method based on data correlation analysis of the present invention.

[0056] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a data lake cache optimization method based on data correlation analysis of the present invention.

[0057] The innovation of the present invention is that a cache strategy based on data access patterns, semantic associations and metadata features is proposed, data is grouped using this correlation information, and cache replacement decisions are made at the group level based on machine learning.

[0058] Traditional caching algorithms usually cannot fully consider the potential associations between multi-source data in the data lake. The present invention builds a more accurate caching decision model by deeply analyzing and mining the access patterns, semantic associations and metadata characteristics of these data. The model can predict and select which data groups should be retained in the cache.

[0059] The advantages of the present invention are: by utilizing the correlation between the data in the data lake, the present invention can more accurately select cache data and improve the cache hit rate; at the same time, the model built based on machine learning can cope with various workloads. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of the present invention.

[0061] Figure 2 It is a structural diagram of the present invention.

[0062] Figure 3 It is a schematic diagram of the calculation of relevance weight of the present invention. Specific implementation plan

[0063] The present invention will be further described below in conjunction with the accompanying drawings;

[0064] Example 1

[0065] Reference Figure 1 to Figure 3 , applying the Apache Hudi data lake cache optimization method based on data correlation analysis proposed by the present invention, comprising the following steps:

[0066] (1) Mining correlation data between data.

[0067] (1-1) Data collection and preparation: Collect cache access sequences to mine access correlations between data.

[0068] (1-2) Frequently accessed object mining: Analyze the access sequence and use FCM to mine objects that are frequently accessed together from the access sequence.

[0069] FCM sets multiple constraints to evaluate the strength of the relationship between data. The first is the search radius. FCM will only search for related objects of a given object within the search radius. The second is the search depth. If you are searching for related objects of object m, if the search depth is 10, and object m appears N (N>10) times in the access sequence, then FCM will only check the 10 most recent occurrences of the object in the access sequence. The third is support and confidence. Using the association rule x→y to indicate that object x is accessed, then y will also be accessed within the search radius of object x. If object x appears N times in the access sequence, and object y appears M times within the search radius of object m, then the confidence of the association rule x→y is M / N and the support is M. The higher the confidence, the higher the correlation between the two objects, and the higher the support, the higher the repetitiveness of the two objects. Assume that a related group C has minimum support and minimum confidence restrictions, it must satisfy the following rules: For all C i ∈C and C j ∈C, association rule C i →C j and C j →C i The confidence and support of are not less than the minimum confidence and minimum support.

[0070] The search process of FCM is as follows: 1. FCM scans the access sequence to count the frequency of each object, determines the appropriate search depth and deletes objects with low access frequency. 2. FCM scans the access sequence to generate candidate association rules. Whenever object j appears in the viewing circle of object i, FCM increases the support of association rule i→j by 1. 3. FCM generates correlation groups according to the group definition. First, association rules that do not meet the minimum support and minimum confidence are deleted, and then each object is scanned to find potential groups according to the association rules. For example, for object a, it has association rules a→b and a→c. FCM checks the association rules in descending order of confidence value. For association rule a→b, if b→a also exists, the two objects are divided into a group {ab}. Then, FCM continues to check a→c. Object c can be added to group {ab} if and only if association rules c→a, c→b and b→c also exist. This is based on the definition of the group. Any two objects in the group should be related to each other.

[0071] (1-3) Mining semantically related objects: Obtain the semantic information of objects and calculate the correlation between objects based on the semantic information. For video, audio, and image file types, first convert these files into text data, use the Whisper model to convert video and audio into text, use the convolutional neural network model to understand the image content, and obtain the text data of the image. Then build a Doc2Vec model to characterize the correlation between texts. Specifically, it includes the following steps:

[0072] First, data preprocessing is done to convert objects such as video, audio, and images into text.

[0073] The Doc2Vec model is then used to infer the vector representation of each text.

[0074] Next, we calculate the relevance of the text. After obtaining the vector representation of each text, we use cosine similarity to calculate the relevance score between samples. The calculation formula is as follows:

[0075]

[0076] (1-4) Metadata-related object mining: Obtain the metadata information of the object and calculate the correlation between the objects based on the metadata information. The invention uses the path information in the metadata information and uses a path segmentation algorithm to calculate the path correlation. First, calculate the maximum value of the subdirectories of path A and path B as the denominator, and then calculate the intersection of the directories of A and B as the numerator. If A = / home / user1 / paper / a, B = / home / user1 / paper / b, then max(A,B) = 4, A∩B = 3, sim(A,B) = 3 / 4. The calculation formula is as follows:

[0077]

[0078] (1-5) Generation of correlation information: Generate the final group of related objects based on the information obtained in steps (1-2) (1-3) (1-4), and store the data in the cache information module. First, based on the frequently accessed object group, the semantic correlation and metadata correlation of the objects in the group are calculated to obtain the total correlation between the objects. The invention represents the weight of the correlation as ω. Assuming that object B is an element in the frequently accessed object group of object A, the semantic correlation and metadata correlation of object A and object B are mined respectively. If object A and object B are both semantically correlated and metadata correlated, it means that the correlation weight of object A and object B is ω=α. If only semantic correlation or metadata correlation exists, the correlation weight is ω=β, otherwise ω=γ.

[0079] (2) The cache management module uses the cache information module to manage cache data. Specifically, it includes the following steps:

[0080] (2-1) Cache eviction algorithm: Use group eviction to remove data from the cache to make room for new data. Specifically, it includes the following steps:

[0081] 1. Group utility: Calculate group utility to measure the subsequent impact of evicting or retaining an object group. Object groups with lower utility are more likely to be evicted. The invention defines the utility of an object as the ratio of cost to benefit, where the cost is a miss and the benefit is the freed space multiplied by the time until the next request.

[0082]

[0083] Where T " (t) represents the time interval between the last request and the next request, s " Indicates the size of the object.

[0084] Therefore, the utility of the group of objects is the sum of the utilities of the individual objects.

[0085]

[0086] 2. Feature and model selection: The invention selects the following features for learning: the total number of requests for each object in the group, the number of frequently accessed objects in the group, the write rate when inserting into the cache, the miss rate of the cache when inserting, the request rate of the cache when inserting, the average object size in the group, and the age of the group. If the group input to the model belongs to the Correlation Cache part, a feature will be added: the correlation of the group. The invention uses the gradient boosting tree to design a regression algorithm as the objective function.

[0087] 3. Training: The algorithm uses the training data collected online to train the model and generates new training data by sampling the cached object groups. Specifically, it includes the following steps:

[0088] Sampling: Assuming there are M groups in the cache, and N groups need to be sampled each time, the step size is M / N, and one group is sampled every other step size. The utility of the sampled group is initialized to 0 at the beginning. When an object o in the sampled group is requested, the algorithm calculates the utility of the object using formula (3) and adds the utility to the group utility.

[0089] Data processing: Convert the features of each group in the sampling into a feature matrix as the X value of the model input, and use the utility of the sampling group as the Y value of the model input. Assume a group G i , there are n features in the group, denoted as x ij , for group G i , whose corresponding utility value is y i , then the characteristic matrix and utility value of group G can be expressed as:

[0090]

[0091] If the sampling contains k groups, each group G iCorresponding to a feature matrix X i and utility value Y i . The final model input is:

[0092]

[0093] Initialize model: Initialize the model's initial prediction value, which is set to the average of the sample group's utility values. The calculation formula is:

[0094]

[0095] in is the initial prediction value, N is the number of samples, y i is the utility value.

[0096] Define the loss function: For regression problems, use MSE as the loss function to measure the gap between the predicted value and the actual value. The calculation formula is:

[0097]

[0098] Calculate gradients and second-order derivatives: For each iteration, calculate the gradient g of the current model prediction value relative to the loss function i and the second-order derivative h i The calculation formula is:

[0099]

[0100] where g i and h i are the gradient and second-order derivative of the i-th sample, respectively, and t represents the current iteration round.

[0101] Building a decision tree: Using the gradient g i and the second-order derivative h i To build a new decision tree. The weight ω0 of each leaf node of the new tree can be obtained by optimizing the objective function:

[0102]

[0103] Among them, λ is the regularization parameter used to prevent the model from overfitting, and i∈j means that sample i belongs to leaf node j.

[0104] Update model: Add the prediction results of the new decision tree to the prediction values ​​of the existing model. The calculation formula is:

[0105]

[0106] Among them, f t (x) is the output of the newly constructed decision tree.

[0107] The gradient and second-order derivative calculations, decision tree construction, and model update are repeated until a predetermined number of iterations is reached or the model converges.

[0108] 4. Inference: When the cache needs to be evicted, it predicts the group utility of all objects, sorts these groups, and evicts the N groups of objects with the lowest utility after merging them.

[0109] 5. Eviction: This invention uses merging-based eviction. Each time an eviction occurs, the least useful N groups of objects are selected for merging. During the merging process, M objects are retained and all other objects are expelled. The retained objects form a new group. The object selection during the merging process uses a simple metric based on the age and size of the object to calculate the utility of a single object, i.e. Where age is the time since the last access. If the Correlation Cache is merged, first based on Calculate the object with the greatest utility, and then obtain the weight of the relevance between other objects in the group and this object through the cache information module, and then Sorting is performed to determine the objects that are ultimately retained. The characteristics of the group generated after merging are averaged over the N groups.

[0110] (2-2) The cache management module receives the content request sent by the user and determines what kind of data request it is.

[0111] Specifically, it includes the following:

[0112] 1. If it is a write request, determine whether the requested content is in the cache information module. The data in the cache information module is generated regularly by step (1). Choose whether to put the data into the Time Cache part or the Correlation Cache part based on the judgment result. If the data in the request exists in the cache information module, obtain the relevant group of the data from the cache information module, and determine whether each data in the relevant group exists in the Correlation Cache part. If so, insert the data into the same group and add the correlation of the two objects to the group correlation. If not, create a new data group and insert it. If the requested data does not exist in the cache information module, write the data into the Time Cache part in sequence.

[0113] 2. If it is a read request, first determine whether the requested content is in the cache information module. If the requested data exists in the cache information module, first obtain the data from the Correlation Cache. If the data is successfully obtained, the response is successful. If the data cannot be obtained from the Correlation Cache, obtain the data from the Time Cache. If the data is successfully obtained, the response is successful. Otherwise, query the data from the data lake and respond to the data. At the same time, start a background thread to query whether the data exists in the cache information module. If so, all the data in the cache information module will be pre-fetched into the cache. If not, the data will only be inserted into the cache.

[0114] Example 2

[0115] This embodiment relates to a data lake cache optimization device based on data correlation analysis, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a data lake cache optimization method based on data correlation analysis in embodiment 1.

[0116] Example 3

[0117] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a data lake cache optimization method based on data correlation analysis in embodiment 1 is implemented.

[0118] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A data lake cache optimization method based on data correlation analysis, It is characterized in that The following steps are involved: (1) Mining correlation data between data and storing them in the cache information module; (2) The cache management module executes the corresponding response strategy according to the type of request, and executes the cache replacement strategy based on the group eviction mechanism of data relevance when the cache capacity reaches the upper limit.

2. A data lake cache optimization method based on data correlation analysis as claimed in claim 1, It is characterized in that Step (1) specifically includes: (1-1) Data collection and preparation: Collect cache access sequences to mine access correlations between data; (1-2) Frequently accessed object mining: Analyze the access sequence and use FCM to mine objects that are frequently accessed together from the access sequence; FCM sets multiple constraints to evaluate the strength of the relationship between data; the first is the search radius, FCM will only search for related objects of a given object within the search radius; the second is the search depth, if you are searching for related objects of object m, if the search depth is 10, and object m appears N (N>10) times in the access sequence, then FCM will only check the 10 most recent occurrences of the object in the access sequence; the third is support and confidence; using the association rule x→y to indicate that object x is accessed, then y will also be accessed within the search radius of object x. If object x appears N times in the access sequence, and object y appears M times within the search radius of object m, then the confidence of the association rule x→y is M / N and the support is M; the higher the confidence, the higher the correlation between the two objects, and the higher the support, the higher the repetition of the two objects; assuming that a related group C has minimum support and minimum confidence constraints, it must satisfy the following rules: for all C i ∈C and C j ∈C, association rule C i →C j and C j →C i The confidence and support of are not less than the minimum confidence and minimum support; The search process of FCM is as follows:

1. FCM scans the access sequence to count the frequency of each object, determines the appropriate search depth and deletes objects with low access frequency; 2. FCM scans the access sequence to generate candidate association rules. Whenever object j appears in the viewing circle of object i, FCM increases the support of association rule i→j by 1; 3. FCM generates correlation groups according to the group definition. First, it removes the association rules that do not meet the minimum support and minimum confidence, and then scans each object to find potential groups according to the association rules. For example, for object a, it has association rules a→b and a→c. FCM checks the association rules in descending order of confidence value. For association rule a→b, if b→a also exists, the two objects are divided into a group {ab}. Then, FCM continues to check a→c. Object c can be added to group {ab} if and only if association rules c→a, c→b, and b→c also exist. This is based on the definition of the group. Any two objects in the group should be related to each other. (1-3) Mining semantically related objects: Obtain the semantic information of objects and calculate the correlation between objects based on the semantic information. For video, audio, and image file types, first convert these files into text data, use the Whisper model to convert video and audio into text, use the convolutional neural network model to understand the image content, and obtain the text data of the image. Then build a Doc2Vec model to characterize the correlation between texts. Specifically, it includes the following steps: First, data preprocessing, converting objects such as video, audio, and pictures into text; Then the Doc2Vec model is used to infer the vector representation of each text; Next, we calculate the relevance of the text. After obtaining the vector representation of each text, we use cosine similarity to calculate the relevance score between samples. The calculation formula is as follows: (1-4) Metadata-related object mining: Obtain metadata information of objects, and calculate the correlation between objects based on the metadata information; the invention uses the path information in the metadata information and uses a path segmentation algorithm to calculate the path correlation; first calculate the maximum value of the subdirectories of path A and path B as the denominator, and then calculate the intersection of the directories of A and B as the numerator; if A = / home / user1 / paper / a, B = / home / user1 / paper / b, then max(A,B) = 4, A∩B = 3, sim(A,B) = 3 / 4; the calculation formula is as follows: (1-5) Generation of correlation information: Generate the final group of related objects based on the information obtained in steps (1-2)(1-3)(1-4), and store the data in the cache information module; first, based on the frequently accessed object group, calculate the semantic correlation and metadata correlation for the objects in the group, so as to obtain the total correlation between the objects; the invention represents the weight of the correlation as ω, assuming that object B is an element in the frequently accessed object group of object A, then respectively mine the semantic correlation and metadata correlation between object A and object B. If object A and object B are both semantically correlated and metadata correlated, then the correlation weight ω=α between object A and object B is represented by ω=β. If only semantic correlation or metadata correlation exists, the correlation weight is ω=β, otherwise ω=γ.

3. The data lake cache optimization method based on data correlation analysis according to claim 1, It is characterized in that Step (2) specifically includes: 2-1) Cache eviction algorithm: Delete data from the cache by group eviction to make room for new data. Specifically, it includes the following steps:

1. Group utility: Calculate group utility to measure the subsequent impact of evicting or retaining an object group. Object groups with lower utility are more likely to be evicted. The invention defines the utility of an object as the ratio of cost to benefit, where the cost is a miss and the benefit is the freed space multiplied by the time until the next request. Where T o (t) represents the time interval between the last request and the next request, s o Indicates the size of the object; The utility of a group of objects is therefore the sum of the utilities of the individual objects; 2. Feature and model selection: The invention selects the following features for learning: the total number of requests for each object in the group, the number of frequently accessed objects in the group, the write rate when inserting into the cache, the cache miss rate when inserting, the cache request rate when inserting, the average object size in the group, and the age of the group; if the group input to the model belongs to the Correlation Cache part, a feature will be added: the correlation of the group; the invention uses the gradient boosting tree to design a regression algorithm as the objective function.

3. Training: The algorithm uses the training data collected online to train the model and generates new training data by sampling the cached object groups; specifically, it includes the following steps: Sampling: Assuming that there are M groups in the cache and N groups need to be sampled each time, the step length is M / N, and one group is sampled every other step length. The utility of the sampled group is initialized to 0 at the beginning. When an object o in the sampled group is requested, the algorithm calculates the utility of the object using formula (3) and adds the utility to the group utility. Data processing: Convert the features of each group in the sampling into a feature matrix as the X value of the model input, and use the utility of the sampling group as the Y value of the model input; Assume that a group G i , there are n features in the group, denoted as x ij , for group G i , whose corresponding utility value is y i , then the characteristic matrix and utility value of group G can be expressed as: If the sampling contains k groups, each group G i Corresponding to a feature matrix X i and utility value Y i ; The final model input is: X={X 1 ,X 2 ,…,X k } And=(And 1 ,AND 2 ,…,AND k ) T (6) Initialize model: Initialize the model's initial prediction value, which is set to the average value of the sample group's utility value; the calculation formula is: in is the initial prediction value, N is the number of samples, y i is the utility value; Define the loss function: For regression problems, use MSE as the loss function to measure the gap between the predicted value and the actual value; the calculation formula is: Calculate gradients and second-order derivatives: For each iteration, calculate the gradient g of the current model prediction value relative to the loss function i and the second-order derivative h i ; The calculation formula is: where g i and h i are the gradient and second-order derivative of the i-th sample, respectively, and t represents the current iteration round; Building a decision tree: Using the gradient g i and the second-order derivative h i To build a new decision tree; the weight of each leaf node of the new tree is ω j It can be obtained by optimizing the objective function: Among them, λ is the regularization parameter used to prevent the model from overfitting, i∈j means that sample i belongs to leaf node j; Update model: Add the prediction results of the new decision tree to the prediction values ​​of the existing model; the calculation formula is: Among them, f t (x) is the output of the newly constructed decision tree; Repeatedly calculate gradients and second-order derivatives, build decision trees, and update models until a predetermined number of iterations is reached or the model converges; 4. Inference: When the cache needs to be evicted, it predicts the group utility of all objects, sorts these groups, and evicts the N groups of objects with the lowest utility after merging them; 5. Eviction: The invention uses merging-based eviction. Each time an eviction occurs, the least useful N groups of objects are selected for merging. During the merging process, M objects are retained and all other objects are expelled. The retained objects form a new group. The object selection during the merging process uses a simple metric based on the age and size of the object to calculate the utility of a single object, i.e. The age is the time since the last access. If the Correlation Cache is merged, first based on Calculate the object with the greatest utility, and then obtain the weight of the relevance between other objects in the group and this object through the cache information module, and then Sorting is performed to determine the objects that are ultimately retained; the characteristics of the group generated after merging will take the average value of N groups; (2-1) The cache management module receives a content request sent by a user and determines what kind of data request the request is; Specifically, it includes the following:

1. If it is a write request, determine whether the requested content is in the cache information module. The data in the cache information module is generated regularly by step (1); choose whether to put the data into the Time Cache part or the Correlation Cache part according to the determination result; if the data in the request exists in the cache information module, obtain the relevant group of the data from the cache information module, and determine whether each data in the relevant group exists in the Correlation Cache part. If so, insert the data into the same group and add the correlation of the two objects to the group correlation; if not, create a new data group and insert it; if the requested data does not exist in the cache information module, write the data into the Time Cache part in sequence; 2. If it is a read request, first determine whether the requested content is in the cache information module. If the requested data exists in the cache information module, first obtain the data from the Correlation Cache. If the data is successfully obtained, the response is successful. If the data cannot be obtained from the Correlation Cache, obtain the data from the Time Cache. If the data is successfully obtained, the response is successful. Otherwise, query the data from the data lake and respond to the data. At the same time, start a background thread to query whether the data exists in the cache information module. If so, all the data in the cache information module will be pre-fetched into the cache. If not, the data will only be inserted into the cache.

4. A data lake cache optimization device based on data correlation analysis, It is characterized in that It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a data lake cache optimization method based on data correlation analysis according to any one of claims 1 to 3.

5. A computer-readable storage medium, It is characterized in that A program is stored thereon, and when the program is executed by a processor, a data lake cache optimization method based on data correlation analysis as described in any one of claims 1 to 3 is implemented.

Citation Information

Cited By

  • Intelligent data lake server management method and system based on IPMI protocol

    CN121542135A