Cold and hot data identification methods, devices, equipment and storage media

Abstract

This invention discloses a method, apparatus, device, and storage medium for identifying hot and cold data, relating to the field of data processing technology. The method includes: acquiring data tables and data table information from a big data cluster; extracting features from the data table information to obtain feature data, including data access frequency, data generation time, and data partitioning time; training a tree algorithm model based on the feature data to obtain a target tree model, the tree algorithm model being constructed based on the XGBoost algorithm, the loss function during training employing a weighted binary cross-entropy function, and the objective function during training based on a cost function including a regularization term; and using the target tree model to perform attribute identification on the data in the data table to obtain the data identification result. This invention solves the problem of poor accuracy in methods for identifying hot and cold data, achieving automatic identification of hot and cold data based on machine learning, thus improving the accuracy and efficiency of hot and cold data identification.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and storage medium for identifying hot and cold data. Background Technology

[0002] With the rapid popularization of technologies such as 5G networks and the Internet of Things, and the rapid growth of data volume, the tiered storage of massive amounts of data has become a major challenge in the field of big data, including how to identify cold and hot data. Currently, the main method for identifying cold and hot data relies on human experience, with data governance and maintenance personnel judging its cold or hot attribute based on factors such as data usage frequency. This method depends on human experience and has certain limitations, including insufficient accuracy. Moreover, with the growth of data and the rapid iteration of tenant needs, errors are prone to occur. Summary of the Invention

[0003] The main objective of this invention is to provide a method, apparatus, device, and storage medium for identifying hot and cold data, aiming to solve the technical problem of poor accuracy in related technologies for identifying hot and cold data.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] In a first aspect, the present invention provides a method for identifying hot and cold data, the method comprising:

[0006] Retrieve data tables and data table information from a big data cluster;

[0007] Feature extraction is performed on the data table information to obtain feature data; the feature data includes data access frequency, data generation time, and data partitioning time.

[0008] The tree algorithm model is trained based on the feature data to obtain the target tree model. The tree algorithm model is built based on the XGBoost algorithm. The loss function during the training process adopts a weighted binary cross-entropy function, and the target function during the training process is established based on a cost function including a regularization term.

[0009] The target tree model is used to identify the attributes of the data in the data table, and the data identification results are obtained as cold data or hot data.

[0010] Optionally, in the above method for identifying hot and cold data, the data table information includes the access time, generation time, and partitioning time of the data table;

[0011] The steps for extracting features from data table information to obtain feature data include:

[0012] The number of times data in the data table is accessed within a preset time period is determined based on the access time, thus obtaining the data access frequency.

[0013] The data generation time is obtained based on the generation time.

[0014] The data partition time is obtained based on the partition time.

[0015] Optionally, in the above-mentioned hot and cold data identification method, the step of training the tree algorithm model based on feature data to obtain the target tree model includes:

[0016] The feature data is divided into a training set and a test set;

[0017] The tree algorithm model is trained using the training set to obtain the trained tree algorithm model;

[0018] The trained tree algorithm model is used to predict the test set to obtain the prediction results;

[0019] Determine whether the prediction result meets the preset conditions. If the prediction result meets the preset conditions, then the trained tree algorithm model is determined as the target tree model.

[0020] Optionally, in the above-mentioned hot and cold data identification method, before the step of training the tree algorithm model based on the training set to obtain the trained tree algorithm model, the method further includes:

[0021] The XGBoost algorithm is used to construct the model, resulting in a tree algorithm model.

[0022] Based on the tree algorithm model, a loss function is selected;

[0023] Define the objective function based on the loss function;

[0024] The steps to train the tree algorithm model using the training set and obtain the trained tree algorithm model include:

[0025] The tree algorithm model is trained using the training set, loss function, and objective function to obtain the trained tree algorithm model.

[0026] Optionally, in the above method for identifying hot and cold data, in the step of using the XGBoost algorithm to construct the model and obtain the tree algorithm model, the expression of the tree algorithm model is:

[0027]

[0028] Where, x i Let f represent the i-th sample data, k represent the number of decision trees in the tree algorithm model, and f k (x i ) represents the weight of the i-th sample in the k-th decision tree where it falls on a certain leaf. x represents i The predicted value after the t-th iteration, yi This represents the value corresponding to the i-th sample data, where y i When y = 1, it indicates that the sample data is hot data. i When f = 0, it indicates that the sample data is cold data. k Let represent the residual of the k-th round.

[0029] Optionally, in the above-mentioned hot and cold data identification method, the step of selecting the loss function based on the tree algorithm model includes:

[0030] Based on the tree algorithm model, the weighted binary cross-entropy function is used as the loss function, and the expression of the loss function is:

[0031]

[0032] Where Loss represents the loss value of the loss function, n represents the number of sample data, and λ1 and λ2 are weight coefficients.

[0033] Optionally, in the above-mentioned method for identifying hot and cold data, the step of defining the objective function based on the loss function includes:

[0034] Based on the loss function, a cost function including a regularization term is constructed. The expression for the cost function is:

[0035]

[0036] Where l represents the loss function, constant represents the constant term, Ω represents the training complexity, and the regularization term Ω(f) is... t )for:

[0037]

[0038] Where T represents the number of leaf nodes, and w represents the weight value of the leaf nodes. j Let represent the weight value of the j-th leaf node, γ represent a constant used to control the number of leaf nodes T, and λ represent a constant used to control the weight w of the leaf nodes.

[0039] The cost function is optimized to obtain the objective function.

[0040] In a second aspect, the present invention provides a hot / cold data identification device, the device comprising:

[0041] The data acquisition module is used to acquire data tables and data table information from the big data cluster;

[0042] The data processing module is used to extract features from the data table information to obtain feature data; the feature data includes data access frequency, data generation time, and data partitioning time.

[0043] The model training module is used to train the tree algorithm model based on feature data to obtain the target tree model. The tree algorithm model is built based on the XGBoost algorithm. The loss function during training adopts a weighted binary cross-entropy function, and the target function during training is established based on a cost function including a regularization term.

[0044] The data recognition module is used to identify the attributes of data in the data table using the target tree model, and to obtain the data recognition results of whether the data is cold data or hot data.

[0045] Thirdly, the present invention provides a hot and cold data identification device, which includes a processor and a memory. The memory stores a hot and cold data identification program. When the hot and cold data identification program is executed by the processor, it implements the hot and cold data identification method as described above.

[0046] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by one or more processors, implements the hot and cold data identification method described above.

[0047] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects:

[0048] This invention proposes a method, apparatus, device, and storage medium for identifying hot and cold data. It acquires a data table and its information; extracts features from the data table information to obtain feature data; trains a tree algorithm model based on the feature data to obtain a target tree model; and finally uses the target tree model to identify the attributes of the data in the data table, obtaining the data identification result. This achieves automatic identification of hot and cold data based on machine learning. In this invention, a tree algorithm model is constructed based on the XGBoost algorithm, and a weighted binary cross-entropy function is used as the loss function. This optimizes the problem of imbalanced positive and negative sample data, improving identification sensitivity and accuracy, thereby enhancing the precision of hot and cold data identification. Furthermore, a target function is established based on a cost function including a regularization term, which prevents model overfitting and improves model construction efficiency, thus enhancing the efficiency of hot and cold data identification. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these provided drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the hot and cold data identification method of the present invention;

[0051] Figure 2 This is a schematic diagram of the hardware structure of the hot and cold data identification device of the present invention;

[0052] Figure 3 This is a schematic diagram of the functional modules of the hot and cold data identification device of the present invention.

[0053] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0055] It should be noted that in this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element. In this invention, the use of suffixes such as "module," "component," or "unit" to denote elements is merely for illustrative purposes and has no specific meaning in itself. Therefore, "module," "component," or "unit" can be used interchangeably. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. Furthermore, the technical solutions of the various embodiments can be combined with each other; however, this is based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0056] With the rapid popularization of technologies such as 5G networks and the Internet of Things, and the rapid growth of data volume, the tiered storage of massive data has become a major challenge in the field of big data. This includes how to identify cold data and hot data so that hot data can be placed in high-performance storage devices for frequent daily read and write operations, and cold data can be stored in lower-cost OSS (Operation Support Systems) storage devices to reduce data maintenance pressure and costs.

[0057] Analysis of related technologies reveals that currently, the main method for identifying hot and cold data is through human experience, with data governance and maintenance personnel judging the hot or cold attributes of data based on the frequency of data usage.

[0058] The problems with this method are:

[0059] 1. This method relies on human experience, but data tables have many application scenarios, and different use scenarios have different characteristics. Human experience can often only discover some of these characteristics in advance, so it has certain limitations.

[0060] 2. The recognition accuracy is insufficient. The existing model is equivalent to manually setting up multiple classifiers, but there is no connection between each classifier, which may lead to missed cases after data screening. Moreover, with the growth of data and the rapid iteration of tenant needs, manual identification of hot and cold data is prone to errors.

[0061] 3. High manual maintenance costs and difficulty in automation. This task needs to be performed manually on a regular, such as daily, basis. Since the data tables to be processed are not exactly the same each day, it is difficult to fully automate the task using written SQL scripts.

[0062] In view of the technical problem that the methods for identifying hot and cold data in related technologies have poor identification accuracy, the present invention provides a method for identifying hot and cold data, the overall idea of ​​which is as follows:

[0063] The process involves: acquiring data tables and their information from a big data cluster; extracting features from the data table information to obtain feature data, including data access frequency, data generation time, and data partitioning time; training a tree algorithm model based on the feature data to obtain a target tree model; the tree algorithm model is built based on the XGBoost algorithm, with a weighted binary cross-entropy function used as the loss function during training, and the target function established based on a cost function including a regularization term; and using the target tree model to identify the attributes of the data in the data tables, thus determining whether the data is cold or hot data.

[0064] Through the above technical solutions, automatic identification of hot and cold data based on machine learning is realized. In this invention, a tree algorithm model is constructed based on the XGBoost algorithm, and a weighted binary cross-entropy function is used as the loss function. This can optimize the problem of imbalance between positive and negative sample data, improve the identification sensitivity and accuracy, and thus improve the accuracy of hot and cold data identification. Furthermore, an objective function is established based on a cost function including a regularization term, which can prevent the model from overfitting and improve the model construction efficiency, thereby improving the efficiency of hot and cold data identification.

[0065] The cold and hot data identification method, apparatus, device, and storage medium provided by the present invention will be described in detail below with reference to the accompanying drawings and through specific embodiments and implementation methods.

[0066] Example 1

[0067] Reference Figure 1 The flowchart illustrates the first embodiment of the cold and hot data identification method of the present invention, which is applied to a cold and hot data identification device.

[0068] Hot and cold data identification devices refer to terminal devices or network devices that can achieve network connectivity. Hot and cold data identification devices can be terminal devices such as mobile phones, computers, tablets, portable computers, and embedded industrial control computers, or network devices such as servers and cloud platforms.

[0069] like Figure 2 The diagram shown is a hardware structure schematic of a hot / cold data identification device. The hot / cold data identification device may include: a processor 1001, such as a CPU (Central Processing Unit), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005.

[0070] Specifically, the communication bus 1002 is used to realize the connection and communication between these components;

[0071] User interface 1003 is used to connect to the client and communicate data with the client. User interface 1003 may include an output unit and an input unit.

[0072] The network interface 1004 is used to connect to the backend server and communicate data with the backend server. The network interface 1004 may include input / output interfaces.

[0073] The memory 1005 is used to store various types of data, such as instructions for any application or method in the hot / cold data identification device, as well as application-related data. The memory 1005 may be built-in memory; alternatively, the memory 1005 may also be a storage device independent of the processor 1001, continuing with the previous section. Figure 2 The memory 1005 may include an operating system, a network communication module, a user interface module, and a hot / cold data identification program;

[0074] The processor 1001 is used to call the hot / cold data identification program stored in the memory 1005 and perform the following operations:

[0075] Retrieve data tables and data table information from a big data cluster;

[0076] Feature extraction is performed on the data table information to obtain feature data; the feature data includes data access frequency, data generation time, and data partitioning time.

[0077] The tree algorithm model is trained based on the feature data to obtain the target tree model. The tree algorithm model is built based on the XGBoost algorithm. The loss function during the training process adopts a weighted binary cross-entropy function, and the target function during the training process is established based on a cost function including a regularization term.

[0078] The target tree model is used to identify the attributes of the data in the data table, and the data identification results are obtained as cold data or hot data.

[0079] Based on the aforementioned hot and cold data identification equipment, the following will be combined with... Figure 1 The flowchart shown illustrates the hot / cold data identification method of this embodiment in detail. The method may include the following steps:

[0080] Step S100: Obtain the data tables and data table information in the big data cluster.

[0081] Specifically, the hot and cold data identification device can collect data tables containing logs and other data from a big data cluster, along with their corresponding table names. It can also collect relevant information from the data tables and summarize it into table information. This table information can include the table's access time, generation time, and partitioning time.

[0082] Step S200: Extract features from the data table information to obtain feature data; wherein, the feature data includes data access frequency, data generation time, and data partitioning time.

[0083] Specifically, the hot / cold data identification device can extract data access frequency, data generation time, and data partitioning time from data table information as extracted data features. Specifically, it extracts the access frequency, generation time, and partitioning time for each piece of data in the data table.

[0084] Step S300: Train the tree algorithm model based on the feature data to obtain the target tree model; wherein, the tree algorithm model is constructed based on the XGBoost algorithm, the loss function in the training process adopts the weighted binary cross-entropy function, and the target function in the training process is established based on the cost function including the regularization term.

[0085] Specifically, the hot and cold data identification device can train an algorithm model based on the extracted feature data. This algorithm model is a tree algorithm model built using the XGBoost algorithm (Extreme Gradient Boosting, an efficient gradient boosting decision tree algorithm). The XGBoost algorithm supports the use of a custom second-order differentiable function as the loss function. In this embodiment, during the training of the tree algorithm model, a weighted binary cross-entropy function can be used as the loss function to solve the problem of imbalance between positive and negative samples, which can improve the model accuracy. When the business scenario changes in the future, the weights can be directly modified accordingly, so that the model of this method can also be applied to hot and cold data identification in other scenarios. The objective function during the training process is based on a cost function that includes a regularization term. The regularization term can prevent overfitting. Using the XGBoost algorithm to add a regularization term to the cost function can control the complexity of the model and prevent the model from overfitting. The model after training is the target tree model.

[0086] Step S400: Use the target tree model to identify the attributes of the data in the data table to obtain the data identification results of whether the data is cold data or hot data.

[0087] Specifically, after the cold and hot data identification device trains to obtain the target tree model, it uses the target tree model to identify the attributes of the data in the data table, and determines whether the data is cold or hot data, thereby obtaining the data identification result.

[0088] The hot and cold data identification method provided in this embodiment obtains a data table and its information; extracts features from the data table information to obtain feature data; trains a tree algorithm model based on the feature data to obtain a target tree model; and finally uses the target tree model to identify the attributes of the data in the data table to obtain the data identification result, thus realizing automatic hot and cold data identification based on machine learning. In this invention, a tree algorithm model is constructed based on the XGBoost algorithm, and a weighted binary cross-entropy function is used as the loss function, which can optimize the problem of imbalance between positive and negative sample data, improve the identification sensitivity and accuracy, and thus improve the accuracy of hot and cold data identification. Furthermore, a target function is established based on a cost function including a regularization term, which can prevent the model from overfitting and improve the model construction efficiency, thereby improving the efficiency of hot and cold data identification.

[0089] Example 2

[0090] Based on the same inventive concept, a second embodiment of the cold and hot data identification method of the present invention is proposed, which is applied to a cold and hot data identification device. The cold and hot data identification method of this embodiment is described in detail below. The method may include the following steps:

[0091] Step S100: Obtain the data tables and data table information in the big data cluster.

[0092] Specifically, the data table information includes the table's access time, generation time, and partitioning time. The hot / cold data identification device can extract data such as access logs from the big data cluster. Based on this data, the data table can be obtained, and key information such as access time, generation time, and partitioning time can be extracted from the data content to obtain the data table information. Optionally, it can also collect time partitioning information, tenant-subscribed table names, and unstructured data access times from the data table to meet more practical needs.

[0093] Step S200: Extract features from the data table information to obtain feature data; wherein, the feature data includes data access frequency, data generation time, and data partitioning time.

[0094] Specifically, step S200 may include:

[0095] Step S210: Determine the number of times data in the data table is accessed within a preset time period based on the access time, and obtain the data access frequency;

[0096] Step S220: Obtain the data generation time based on the generation time;

[0097] Step S230: Obtain the data partition time based on the partition time.

[0098] In this embodiment, data access frequency can be the number of times each piece of data in the data table is accessed within the most recent week, month, six months, year, or a longer period, serving as the data access popularity characteristic corresponding to that data; data generation time can be the duration between the latest update time of each piece of data in the data table and the time when the current hot and cold data identification device performs hot and cold data identification, serving as the data generation timeliness characteristic corresponding to that data, and the unit can be days; data partition time can be the duration between the partition time of the data in the data table and the current time, serving as the data information timeliness characteristic, and the unit can also be days.

[0099] Some data has low access frequency because it was recently generated, for example, it was just generated and hasn't been called by other programs yet. Therefore, the data generation time is also a characteristic of the data's hot / cold attribute. In this embodiment, after collecting the data generation time in step S100, feature extraction is performed to obtain the data generation time. This data generation time can then be used as a hot / cold attribute feature of data with low access frequency or that hasn't been called yet. Data with a recent generation time can be identified as hot data. However, some data, although with similar generation times, may have been processed from older data. Therefore, this type of data is not necessarily hot data. Thus, the data partitioning time is also a characteristic of the data's hot / cold attribute. In this embodiment, after collecting the data partitioning time in step S100, feature extraction is performed to obtain the data partitioning time. This data partitioning time can be used as a hot / cold attribute feature of data with a recent generation time that has been processed from older data.

[0100] The feature data includes multiple factors such as data access frequency, data generation time, and data partitioning time. Subsequently, hot and cold data can be identified based on multiple features to ensure the accuracy of data identification and improve the comprehensiveness of data identification, making this method widely applicable in the field of big data business.

[0101] Optionally, after obtaining the feature data, the hot and cold data identification device can also write the feature data into a database; or write it into the database in combination with the data table name for later use.

[0102] Step S300: Train the tree algorithm model based on the feature data to obtain the target tree model; wherein, the tree algorithm model is constructed based on the XGBoost algorithm, the loss function in the training process adopts the weighted binary cross-entropy function, and the target function in the training process is established based on the cost function including the regularization term.

[0103] Specifically, step S300 may include:

[0104] Step S310: Divide the feature data into a training set and a test set;

[0105] Step S320: Train the tree algorithm model based on the training set to obtain the trained tree algorithm model;

[0106] Step S330: Use the trained tree algorithm model to predict the test set and obtain the prediction results;

[0107] Step S340: Determine whether the prediction result meets the preset conditions. If the prediction result meets the preset conditions, then determine the trained tree algorithm model as the target tree model.

[0108] In this embodiment, the feature data obtained in the previous step is divided into a training set and a test set. A portion of the feature data is selected as the training set to train the machine learning classification model, i.e., the tree algorithm model built based on the XGBoost algorithm. The loss function during training is a weighted binary cross-entropy function, and the objective function during training is based on a cost function including a regularization term, resulting in the trained tree algorithm model. Then, the trained tree algorithm model is used to predict the test set to obtain the prediction result. Based on the prediction result, it is determined whether further training or optimization is needed, thus obtaining the trained tree algorithm model, which serves as the target tree model.

[0109] In one embodiment, before step S320 "training the tree algorithm model based on the training set to obtain the trained tree algorithm model", the method further includes:

[0110] Step A10: Use the XGBoost algorithm to build the model and obtain the tree algorithm model.

[0111] Specifically, in step A10, the expression for the tree algorithm model is:

[0112]

[0113] Where, x i Let f represent the i-th sample data, k represent the number of decision trees in the tree algorithm model, and f k (x i ) represents the weight of the i-th sample in the k-th decision tree where it falls on a certain leaf. x represents i The predicted value after the t-th iteration, y i This represents the value corresponding to the i-th sample data, where y i When y = 1, it indicates that the sample data is hot data. i When f = 0, it indicates that the sample data is cold data. k Let represent the residual of the k-th round.

[0114] Based on the above expression, we know that the score of the i-th sample is actually the sum of the weights of different leaf nodes. Therefore, to predict whether the data is cold or hot, we need to calculate the tree structure and the weights of all leaf nodes in the above expression.

[0115] Step A20: Select a loss function based on the tree algorithm model.

[0116] Specifically, step A20 may include:

[0117] Step A21: Based on the tree algorithm model, the weighted binary cross-entropy function is used as the loss function. The expression for the loss function is:

[0118]

[0119] Where Loss represents the loss value of the loss function, n represents the number of sample data, and λ1 and λ2 are weight coefficients.

[0120] In general binary classification algorithms, the loss function is often the binary cross-entropy function, which is calculated as follows:

[0121]

[0122] Based on this binary cross-entropy function, when y i =1, when the sample data is identified as hot data, the actual calculation formula is Loss = -log(f(x)). i If f(x) i If f(x) is close to 1, then the loss value is close to 0, and the loss is minimized; conversely, if f(x) is close to 1, then the loss value is close to 0, and the loss is minimized. i If y approaches 0, then the loss value (Loss) approaches infinity. i When the sample data is identified as cold data and equal to 0, the actual calculation formula is Loss = -log(1-f(x)). i If f(x) i If f(x) is close to 0, then the loss value is close to 0, and the loss is minimized; conversely, if f(x) is close to 0, then the loss is minimized. i If the value is close to 1, then the loss value is close to infinity.

[0123] In this embodiment, in order to address the problem of imbalance between hot and cold data samples and the sensitivity of the business to false negatives (hot data being misjudged as cold data), a weighted binary cross-entropy function is used as the loss function, as shown in the expression in step A21 above.

[0124] In the loss function of this embodiment, λ1 and λ2 are weighting coefficients, which can solve the problems of imbalance between positive and negative samples and different importance.

[0125] When λ2 is set to be greater than 1, if y i When the sample data is identified as cold data (i.e., equal to 0), the loss function is actually Loss = -λ²log(1-f(x)). i If f(x) i If f(x) is close to 0, then the loss value is close to 0, and the loss is minimized; conversely, if f(x) is close to 0, then the loss is minimized. i If the value is close to 1, then the loss value will be close to infinity and larger than that in the unweighted binary cross-entropy function. At this time, cold data is less likely to be misjudged as hot data.

[0126] When λ1 is set to be greater than 1, if y iWhen the sample data is identified as hot data (i.e., =1), the loss function is actually Loss = -λ1log(f(x)). i If f(x) i If f(x) is close to 1, then the loss value is close to 0, and the loss is minimized; conversely, if f(x) is close to 1, then the loss value is close to 0, and the loss is minimized. i If the value is close to 0, then the loss value will be close to infinity and larger than that in the unweighted binary cross-entropy function. At this time, hot data is less likely to be misjudged as cold data.

[0127] In practical hot and cold data classification, misclassifying cold data as hot data can put pressure on the data warehouse. However, misclassifying hot data as cold data may cause changes in data access paths, leading to problems such as the unavailability of existing programs. Therefore, the latter situation should be avoided as much as possible in business operations, meaning it's crucial to reduce the misclassification rate of hot data. Thus, in practical applications, a larger weight can be assigned to λ1.

[0128] Introducing a weighted binary cross-entropy loss function can optimize the problem of imbalanced positive and negative samples, preventing the model from having a low accuracy due to the small proportion of one type of data. For example, if there are few cold data samples in the training set, the prior information related to cold data may not be learned during training, which may lead to misjudgment of a certain type of data.

[0129] Step A30: Define the objective function based on the loss function.

[0130] Specifically, step A30 may include:

[0131] Step A31: Construct a cost function including a regularization term based on the loss function. The expression for the cost function is:

[0132]

[0133] Where l represents the loss function, constant represents the constant term, Ω represents the training complexity, and the regularization term Ω(f) is... t )for:

[0134]

[0135] Where T represents the number of leaf nodes, and w represents the weight value of the leaf nodes. j Let represent the weight value of the j-th leaf node, γ represent a constant used to control the number of leaf nodes T, and λ represent a constant used to control the weight w of the leaf nodes.

[0136] Step A32: Optimize the cost function to obtain the objective function.

[0137] In this embodiment, a cost function including a regularization term is established based on the loss function:

[0138]

[0139] Where n represents the number of sample data, and the regularization term... It sums the complexity of all t decision trees and adds it to the objective function as a regularization term to prevent the model from overfitting.

[0140] Therefore, the goal of the XGBoost algorithm is to find a function f t This minimizes the objective function. Therefore, we can continue to optimize the cost function to obtain the objective function. The specific process is as follows:

[0141] Expanding the above cost function using the Taylor formula:

[0142]

[0143] Among them, g i It is the first derivative of the loss function, h i This is the second derivative of the loss function;

[0144] In the above formula, It is a fixed value that has no impact on finding the optimal value and can be removed. Removing the constant term will not affect the finding of the optimal value of the objective function.

[0145] Therefore, the above function can be optimized into the objective function:

[0146]

[0147] In this embodiment, the second derivative is used to accelerate the gradient convergence speed, which improves the efficiency of model building compared to the previous manual judgment method and can be applied to enterprise-level data governance activities.

[0148] Correspondingly, step S320, "training the tree algorithm model based on the training set to obtain the trained tree algorithm model," may include:

[0149] Step S321: Train the tree algorithm model based on the training set, loss function, and objective function to obtain the trained tree algorithm model.

[0150] The specific training process is as follows:

[0151] First, further refine the function definition, defining q(x) as representing which leaf node the sample data x belongs to, w q(x) Let f represent the weight of the leaf node, q represent the tree structure, q(x) be the position of the sample data x in the leaf node, and f be the weight of the leaf node. t (x) can be represented as:

[0152] ft (x)=w q(x) ,w∈R T ,q:R d →{1,2,...K}

[0153] The sample data set I in the j-th leaf node j for:

[0154] I j ={i|q(x i )=j};

[0155] To control model complexity, the XGBoost algorithm was used in the aforementioned process to add a regularization term to the cost function. This regularization term includes the number of leaf nodes in the tree and the sum of the squares of the L2 moduli of the output scores at each leaf node, i.e., the L2 norm. The regularization term Ω(f) t )for:

[0156]

[0157] Where T represents the number of leaf nodes, and w represents the weight value of the leaf nodes. j Let represent the weight value of the j-th leaf node, γ represent a constant used to control the number of leaf nodes T, and λ represent a constant used to control the weight w of the leaf nodes.

[0158] The complexity of a decision tree is determined by both the number of leaf nodes (T) and the L2 norm. In other words, fewer leaf nodes mean a simpler model. Therefore, calculating γ and T can prevent overfitting, and the regularization term Ω(f) is used. t In fact, it is a model complexity function.

[0159] Then, substituting the model complexity function into the objective function defined above, we can obtain the following formula:

[0160]

[0161] set up The above formula is then:

[0162]

[0163] According to the formula for the minimum value of a quadratic function, we can obtain:

[0164]

[0165]

[0166] Next, a greedy algorithm is used to find the optimal split point, which can be determined by the gain value, G. LThis refers to the sum of the g values ​​of all samples in the left node. That is, the feature corresponding to the maximum gain (Gain) is used as the first split point in the decision tree; for example, a feature with a visit frequency exceeding n times within a month is used as the first split point. The maximum gain (Gain) is:

[0167]

[0168] Finally, by analogy, we find the feature corresponding to the next maximum gain (Gain), which determines the second split point. For example, the number of days between the data update time and the current time, i.e., when the data generation time exceeds a certain value, can be used as the second split point. Repeat the above process recursively to build the tree until the end, completing the training and obtaining the trained tree algorithm model.

[0169] In this embodiment, the XGBoost algorithm model of machine learning replaces the existing manual screening method, continuously fits the prediction residual of the previous level, finds the optimal split point, generates multiple trees through different features, and uses multiple classifiers to automatically classify cold and hot data, with low dimensionality cost and less manual intervention.

[0170] Step S400: Use the target tree model to identify the attributes of the data in the data table to obtain the data identification results of whether the data is cold data or hot data.

[0171] Specifically, after obtaining the target tree model, the hot / cold data identification device can apply this model to identify the hot / cold attributes of daily data. Here, it can perform attribute identification on the data in the previously obtained data table to determine whether each data point is hot or cold, thus obtaining the data identification result. The hot / cold data identification device can also write this data identification result into a database for subsequent processing.

[0172] With the rapid growth of data volume, indiscriminately storing all data in Hadoop clusters or other data warehouses would undoubtedly lead to significant storage waste. Furthermore, the need for clusters to maintain large amounts of metadata would degrade cluster performance to some extent. Therefore, cold data migration is a viable solution. This embodiment addresses the accuracy and maintainability issues in identifying hot and cold data, providing a basis for subsequent cold data migration. In the future, it can be widely applied in big data business fields, significantly improving the efficiency of tiered hot and cold data processing and enhancing enterprises' data asset management capabilities.

[0173] For more details on the specific implementation of the above method steps, please refer to the description of the specific implementation in Example 1. For the sake of brevity, these details will not be repeated here.

[0174] The hot and cold data identification method provided in this embodiment automatically classifies data into cold or hot data by using a target tree model trained by machine learning, which solves the problems of repetitive and error-prone manual processing; and uses a weighted binary cross-entropy function as a loss function to solve the problem that imbalance between positive and negative samples may lead to hot data being classified as cold data, resulting in adverse consequences.

[0175] Example 3

[0176] Based on the same inventive concept, referring to Figure 3 The present invention provides a first embodiment of a cold and hot data identification device, which can be a virtual device and applied to cold and hot data identification equipment.

[0177] The following is combined Figure 3 The functional module diagram shown illustrates the cold / hot data identification device provided in this embodiment in detail. The device may include:

[0178] The data acquisition module is used to acquire data tables and data table information from the big data cluster;

[0179] The data processing module is used to extract features from the data table information to obtain feature data; wherein, the feature data includes data access frequency, data generation time, and data partitioning time;

[0180] The model training module is used to train the tree algorithm model based on the feature data to obtain the target tree model; wherein, the tree algorithm model is constructed based on the XGBoost algorithm, the loss function in the training process adopts a weighted binary cross-entropy function, and the target function in the training process is established based on a cost function including a regularization term;

[0181] The data recognition module is used to perform attribute recognition on the data in the data table using the target tree model, and to obtain the data recognition result of whether the data is cold data or hot data.

[0182] Furthermore, the data table information includes the access time, generation time, and partitioning time of the data table; the data processing module may include:

[0183] The first feature extraction unit is used to determine the number of times data in the data table is accessed within a preset time period based on the access time, so as to obtain the data access frequency.

[0184] The second feature extraction unit is used to obtain the data generation time based on the generation time;

[0185] The third feature extraction unit is used to obtain the data partition time based on the partition time.

[0186] Furthermore, the model training module may include:

[0187] A dataset unit is used to divide the feature data into a training set and a test set;

[0188] The model training unit is used to train the tree algorithm model based on the training set to obtain the trained tree algorithm model.

[0189] The data prediction unit is used to predict the test set using the trained tree algorithm model to obtain the prediction result;

[0190] The model determination unit is used to determine whether the prediction result meets the preset conditions. If the prediction result meets the preset conditions, the trained tree algorithm model is determined as the target tree model.

[0191] Furthermore, the model training module may also include:

[0192] The model building unit is used to build a model using the XGBoost algorithm to obtain a tree algorithm model; select a loss function based on the tree algorithm model; and define an objective function according to the loss function.

[0193] Correspondingly, the model training unit is also used to train the tree algorithm model based on the training set, the loss function, and the objective function to obtain the trained tree algorithm model.

[0194] Furthermore, in the model building unit, the expression for the tree algorithm model obtained by using the XGBoost algorithm for model building is:

[0195]

[0196] Where, x i Let f represent the i-th sample data, k represent the number of decision trees in the tree algorithm model, and f k (x i ) represents the weight of the i-th sample in the k-th decision tree where it falls on a certain leaf. x represents i The predicted value after the t-th iteration, y i This represents the value corresponding to the i-th sample data, where y i When y = 1, it indicates that the sample data is hot data. i When f = 0, it indicates that the sample data is cold data. k Let represent the residual of the k-th round.

[0197] Furthermore, in the model building unit, based on the tree algorithm model, the selected loss function includes:

[0198] Based on the tree algorithm model, a weighted binary cross-entropy function is used as the loss function, and the expression of the loss function is as follows:

[0199]

[0200] Where Loss represents the loss value of the loss function, n represents the number of sample data, and λ1 and λ2 are weight coefficients.

[0201] Furthermore, in the model building unit, defining the objective function based on the loss function includes:

[0202] Based on the loss function, a cost function including a regularization term is established, and the expression of the cost function is:

[0203]

[0204] Where l represents the loss function, constant represents the constant term, Ω represents the training complexity, and the regularization term Ω(f) is... t )for:

[0205]

[0206] Where T represents the number of leaf nodes, and w represents the weight value of the leaf nodes. j Let represent the weight value of the j-th leaf node, γ represent a constant used to control the number of leaf nodes T, and λ represent a constant used to control the weight w of the leaf nodes.

[0207] The cost function is then optimized to obtain the objective function.

[0208] It should be noted that the functions and corresponding technical effects of each module in the hot and cold data identification device provided in this embodiment can be referred to the description of the specific implementation methods in the various embodiments of the hot and cold data identification method of this invention. For the sake of brevity, they will not be repeated here.

[0209] Example 4

[0210] Based on the same inventive concept, referring to Figure 2 The hardware structure diagram shows that this embodiment provides a hot and cold data identification device, which may include a processor and a memory. The memory stores a hot and cold data identification program. When the hot and cold data identification program is executed by the processor, it implements all or part of the steps of various embodiments of the hot and cold data identification method of the present invention.

[0211] Specifically, hot and cold data identification equipment refers to terminal devices or network devices that can achieve network connectivity. These can be terminal devices such as mobile phones, computers, tablets, portable computers, and embedded industrial control computers, or network devices such as servers and cloud platforms.

[0212] It is understandable that hot and cold data identification devices may also include communication buses, user interfaces, and network interfaces. The communication bus is used to connect and communicate between these components; the user interface is used to connect to the client and communicate data with the client. The user interface may include output units such as a display screen and speakers, and input units such as a keyboard and microphone; the network interface is used to connect to the backend server and communicate data with the backend server. The network interface may include input / output interfaces, such as standard wired interfaces and wireless interfaces such as Wi-Fi interfaces; the memory is used to store various types of data. This data may include, for example, instructions for any application or method in the hot / cold data identification device, as well as application-related data. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EPROM). The memory may be an EEPROM, magnetic storage, flash memory, disk, or optical disk, etc.; optionally, the memory may also be a storage device independent of the processor; the processor is used to call the cold and hot data identification program stored in the memory and execute the cold and hot data identification method as described above. The processor may be an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components, used to execute all or part of the steps of the various embodiments of the cold and hot data identification method described above.

[0213] It needs to be explained that, Figure 2The hardware structure shown does not constitute a limitation on the hot and cold data identification device of the present invention, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0214] Example 5

[0215] Based on the same inventive concept, this embodiment provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, disk, optical disk, server, etc. The storage medium stores a computer program, which can be executed by one or more processors. When the computer program is executed by the processor, it can implement all or part of the steps of the various embodiments of the cold and hot data identification method of the present invention.

[0216] It should be noted that the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above embodiments are only optional embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made under the inventive concept of the present invention using the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are all included within the patent protection scope of the present invention.

Claims

1. A method for identifying hot and cold data, characterized in that, The method includes: Obtain data tables and data table information from a big data cluster, including the access time, generation time, and partitioning time of the data tables; Feature extraction is performed on the data table information to obtain feature data; wherein, the feature data includes data access frequency, data generation time, and data partitioning time; The tree algorithm model is trained based on the feature data to obtain the target tree model; wherein, the tree algorithm model is constructed based on the XGBoost algorithm, the loss function in the training process adopts the weighted binary cross-entropy function, and the target function in the training process is established based on the cost function including the regularization term; The target tree model is used to identify the attributes of the data in the data table to obtain the data identification results of whether the data is cold data or hot data. The step of extracting features from the data table information to obtain feature data includes: Based on the access time, determine the number of times the data in the data table is accessed within a preset time period to obtain the data access frequency as a feature of data access popularity. Based on the generation time, determine the time elapsed between the most recent update time of the data in the data table and the current time, and obtain the data generation time as a characteristic of the data generation timeliness; Based on the partition time, determine the duration of the partition time of the data in the data table from the current time, and obtain the data partition time as a characteristic of the timeliness of data information.

2. The hot / cold data identification method as described in claim 1, characterized in that, The step of training the tree algorithm model based on the feature data to obtain the target tree model includes: The feature data is divided into a training set and a test set; The tree algorithm model is trained based on the training set to obtain the trained tree algorithm model; The trained tree algorithm model is used to predict the test set to obtain the prediction results; Determine whether the prediction result meets the preset conditions. If the prediction result meets the preset conditions, then determine the trained tree algorithm model as the target tree model.

3. The hot / cold data identification method as described in claim 2, characterized in that, Before the step of training the tree algorithm model based on the training set to obtain the trained tree algorithm model, the method further includes: The XGBoost algorithm is used to construct the model, resulting in a tree algorithm model. Based on the aforementioned tree algorithm model, a loss function is selected; Define the objective function based on the loss function; The step of training the tree algorithm model based on the training set to obtain the trained tree algorithm model includes: The tree algorithm model is trained based on the training set, the loss function, and the objective function to obtain the trained tree algorithm model.

4. The hot / cold data identification method as described in claim 3, characterized in that, In the step of constructing the model using the XGBoost algorithm to obtain the tree algorithm model, the expression of the tree algorithm model is: Where, x i Let f represent the i-th sample data, k represent the number of decision trees in the tree algorithm model, and f k (x i ) represents the weight of the i-th sample in the k-th decision tree where it falls on a certain leaf. x represents i The predicted value after the t-th iteration, y i This represents the value corresponding to the i-th sample data, where y i When y = 1, it indicates that the sample data is hot data. i When f = 0, it indicates that the sample data is cold data. k Let represent the residual of the k-th round.

5. The hot / cold data identification method as described in claim 4, characterized in that, The step of selecting the loss function based on the tree algorithm model includes: Based on the tree algorithm model, a weighted binary cross-entropy function is used as the loss function, and the expression of the loss function is as follows: Where Loss represents the loss value of the loss function, n represents the number of sample data, and λ1 and λ2 are weight coefficients.

6. The hot / cold data identification method as described in claim 5, characterized in that, The step of defining the objective function based on the loss function includes: Based on the loss function, a cost function including a regularization term is established, and the expression of the cost function is: Where l represents the loss function, constant represents the constant term, Ω represents the training complexity, and the regularization term Ω(f) is... t )for: Where T represents the number of leaf nodes, and w represents the weight value of the leaf nodes. j Let represent the weight value of the j-th leaf node, γ represent a constant used to control the number of leaf nodes T, and λ represent a constant used to control the weight w of the leaf nodes. The cost function is then optimized to obtain the objective function.

7. A cold / hot data identification device, characterized in that, The device includes: The data acquisition module is used to acquire data tables and data table information in the big data cluster. The data table information includes the access time, generation time, and partitioning time of the data table. The data processing module is used to extract features from the data table information to obtain feature data; wherein, the feature data includes data access frequency, data generation time, and data partitioning time; The model training module is used to train the tree algorithm model based on the feature data to obtain the target tree model; wherein, the tree algorithm model is constructed based on the XGBoost algorithm, the loss function in the training process adopts a weighted binary cross-entropy function, and the target function in the training process is established based on a cost function including a regularization term; The data recognition module is used to perform attribute recognition on the data in the data table using the target tree model, and to obtain the data recognition result of whether the data is cold data or hot data; The data processing module includes: The first feature extraction unit is used to determine the number of times the data in the data table is accessed within a preset time period based on the access time, and to obtain the data access frequency as a feature of data access popularity. The second feature extraction unit is used to determine the time elapsed between the most recent update time of the data in the data table and the current time based on the generation time, so as to obtain the data generation time as a feature of data generation timeliness. The third feature extraction unit is used to determine the time elapsed between the partition time of the data in the data table and the current time based on the partition time, so as to obtain the data partition time as a feature of the timeliness of data information.

8. A cold / hot data identification device, characterized in that, The device includes a processor and a memory, the memory storing a hot and cold data identification program, which, when executed by the processor, implements the hot and cold data identification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by one or more processors, implements the hot and cold data identification method as described in any one of claims 1 to 6.

Images