Data processing, prediction model training method and device

Abstract

Embodiments of the present application provide a data processing method and device, and a prediction model training method and device. The method comprises: determining feature information according to target data and access records of the target data; inputting the feature information into a prediction model to obtain time information of future access of the target data; the prediction model is trained by using training samples, the training samples comprising sample features, sample time labels and sample types, the sample time labels and the sample types being determined according to whether there are access records of data corresponding to the sample features before and after a random time within a sample sampling period; and identifying cold and hot data of the target data according to the time information. The sample time labels are used as labels of the training samples to train the prediction model, and then the time interval of next access of the target data based on the prediction model is used as a prediction result, and cold and hot data of the target data is identified according to the time interval, so that the accuracy of cold and hot data identification of the target data can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, and in particular to data processing, a prediction model training method and device. BACKGROUND

[0002] With the rapid development of data processing needs, the cost of data storage has greatly increased. It is found in the process of data storage that these data often have distinct hot and cold characteristics, that is, the data in some areas are data with relatively high access frequency, and the data in other areas are rarely in the access state. If a large amount of cold data occupies high-performance devices, it will cause waste of storage resources.

[0003] In the prior art, different types of storage media and storage methods are used to store cold and hot data separately. Before separating the cold and hot data, the mixed data needs to be accurately identified and separated. The identification rules can be used to identify cold and hot data, such as a method of identifying cold and hot data based on LRU / LFU / LIRS / ExponentialDecay rules. Another type is based on machine learning, which predicts whether the data will be accessed in the future based on the history of data access characteristics. However, the accuracy of the identification results obtained by the above methods is relatively low. Therefore, a solution is needed to improve the accuracy of cold and hot data identification. SUMMARY

[0004] To solve or improve the problems existing in the prior art, the embodiments of the present application provide a data processing, a prediction model training method and device.

[0005] In a first aspect, in one embodiment of the present application, a data processing method is provided. The method comprises:

[0006] determining feature information according to target data and access records of the target data;

[0007] inputting the feature information into a prediction model to obtain time information of future access of the target data; wherein the prediction model is obtained by training a training sample, the training sample comprises sample features, sample time labels and sample types, and the sample time labels and sample types are determined by whether there are access records of the sample features corresponding data before and after a random time in a sample sampling period;

[0008] identifying cold and hot data of the target data according to the time information.

[0009] In a second aspect, in one embodiment of the present application, a prediction model training method is provided. The method comprises:

[0010] construct a training sample, wherein the training sample comprises sample features, a sample time label and a sample type, and the sample time label and the sample type are determined according to whether there is an access record of the sample feature corresponding data before and after a random time within a sample sampling period;

[0011] input the training sample into a prediction model to obtain a prediction result;

[0012] optimize parameters in the prediction model according to the prediction result, the sample time label and the sample type;

[0013] The prediction model is used for identifying cold and hot data.

[0014] In a third aspect, in an embodiment of the present application, a non-transitory machine readable storage medium is provided, and the non-transitory machine readable storage medium stores executable codes, and when the executable codes are executed by a processor of an electronic device, the processor executes the data processing method of the first aspect or the prediction model training method of the second aspect.

[0015] In a fourth aspect, in an embodiment of the present application, an electronic device is provided, comprising a memory and a processor; wherein,

[0016] The memory is configured to store a program.

[0017] The processor is coupled with the memory and is configured to execute the program stored in the memory, so as to implement the data processing method of the first aspect or the prediction model training method of the second aspect.

[0018] The technical scheme provided by the embodiment of the present application inputs the feature information of target data into a pre-trained prediction model, and the prediction model can obtain the time information of future access of the target data, that is, the time difference between the future access and the current time. In order to enable the prediction model to accurately predict the time information of future access, the training sample is constructed to include sample features, a sample time label and a sample type, and the sample time label and the sample type are determined according to whether there is an access record of the sample feature corresponding data before and after a random time within a sample sampling period. The prediction model trained by using such a training sample can accurately identify cold and hot data of target data. Through the above scheme, the sample time label is used as the label of the training sample, the prediction model is trained, and then the time interval of the next access of the target data based on the prediction model is used as the prediction result, and the cold and hot data of the target data are identified according to the size of the time interval, so that the accuracy of identifying cold and hot data of the target data can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A flowchart of a data processing method provided for the embodiments of the present application is shown in the figure.

[0021] Figure 2 A flowchart of a method for constructing a training sample provided for the embodiments of the present application is shown in the figure.

[0022] Figure 3 A schematic diagram of sampling period splitting provided for the embodiments of the present application is shown in the figure.

[0023] Figure 4 A schematic diagram of constructing a training sample based on an observation window provided for the embodiments of the present application is shown in the figure.

[0024] Figure 5 A flowchart of a training method of a prediction model provided for the embodiments of the present application is shown in the figure.

[0025] Figure 6 A flowchart of a parameter optimization method of a prediction model provided for the embodiments of the present application is shown in the figure.

[0026] Figure 7 A schematic diagram of a sample type and corresponding relationship matching process provided for the embodiments of the present application is shown in the figure.

[0027] Figure 8 A schematic diagram of cold and hot data identification provided for the embodiments of the present application is shown in the figure.

[0028] Figure 9 A flowchart of a prediction model training method provided for the embodiments of the present application is shown in the figure.

[0029] Figure 10 A schematic diagram of a cold and hot data identification system architecture provided for the embodiments of the present application is shown in the figure.

[0030] Figure 11 A structural schematic diagram of a data processing device provided for the embodiments of the present application is shown in the figure.

[0031] Figure 12 A structural schematic diagram of an electronic device provided for the embodiments of the present application is shown in the figure.

[0032] Figure 13 A structural schematic diagram of a prediction model training device provided for the embodiments of the present application is shown in the figure.

[0033] Figure 14 Another structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] It is found in the data storage process that the data often has distinct hot and cold characteristics, that is, the data in some areas is data with relatively high access frequency, and the data in other areas is rarely in an access state, or the time interval between two accesses is very long (for example, 3 days or a week, a month, half a year, etc.). If a large amount of cold data occupies high-performance devices, it will cause waste of storage resources. In the prior art, different types of storage media and storage methods are used to store hot and cold data separately. Before separating the hot and cold data, the mixed data needs to be accurately identified and separated. Although some schemes use machine learning schemes to predict whether the data will be accessed in the future by using historical access characteristics of the data, the richness of the obtained samples is limited by the size of the observation window (for example, the data observed by the observation window is the data within 1 day or 1 week). Usually, a single sample is generated within a limited observation window, and the data outside the observation window cannot be used as a training sample, nor can the machine learning model be used to well predict future data. Therefore, a scheme is needed to identify hot and cold data without being limited by the size of the observation window and the number of features.

[0035] In order to enable personnel in the technical field to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0036] In some of the processes described in the specification, claims, and above drawings, multiple operations are described in a particular, sequential order. These operations can be performed in different order than the described or in parallel. The operations will be performed in sequential order unless it is indicated that they will be performed in parallel, or in a different order than described. The numbering of the operations described in the specification, claims, and above drawings are merely for identification, and the numbering itself does not indicate any sequence of execution. Additionally, the processes can include more or fewer operations, and the operations can be executed in sequential order or in parallel. It should be noted that the terms "first", "second", etc. in the text are used to distinguish different messages, devices, modules, etc., and do not represent the order of execution, nor do they limit the types of "first" and "second".

[0037] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0038] As Figure 1 A flowchart of a data processing method provided by an embodiment of the present application is shown. The method comprises:

[0039] 101: determining feature information according to target data and access records of the target data.

[0040] 102: inputting the feature information into a prediction model to obtain time information of future access of the target data; wherein the prediction model is obtained by training a training sample, the training sample comprises sample feature, sample time label and sample type, and the sample time label and sample type are determined by whether there is an access record of sample feature corresponding data before and after a random time in a sample sampling period.

[0041] 103: identifying cold and hot data of the target data according to the time information.

[0042] The target data mentioned herein refers to data that needs to be identified as cold and hot data. The feature information mentioned herein comprises access feature of the target data, related data of the target data, semantic information at a database level (such as / domain knowledge (table name, data size, SQL template)), etc. In addition, the content contained in the feature information can be adjusted according to needs.

[0043] The prediction model mentioned herein can be a generated analysis model, which is obtained by pre-training a training sample. The time information mentioned herein can be a time interval between a time of future access of the target data and a current time (or a specified time). The longer the time interval corresponding to the time information is, the lower the frequency of access of the target data is, and the more likely the target data is to be classified as cold data; on the contrary, the shorter the time interval corresponding to the time information is, the higher the frequency of access of the target data is, and the more likely the target data is to be classified as hot data. Therefore, when identifying cold and hot data, the present solution does not directly classify the target data as cold and hot time, but uses the time interval between the time of future access and the current time as the basis for distinguishing cold and hot data.

[0044] The training sample is generated based on the access records and related information of the target data in a historical time period. Specifically, the training sample can be represented as (x i, y i, e i ), where x i represents the sample feature (covariate, such as the access feature), y i represents the sample time label, and e i represents the sample type. It should be noted that the sample time label and the sample type are determined by whether there is an access record corresponding to the sample feature before and after a random time in the sample sampling period. This will be illustrated in the embodiments described below for constructing the sample. The sample type described herein includes the missing type and the non-missing type.

[0045] The length of the sample sampling period is not limited, and the sample sampling can be performed based on a long time period (such as 1 year or half a year). In actual sample sampling, the time period corresponding to the training sample input to the prediction model is also limited due to the observation window. When sampling, the sampling needs to be performed in time periods, which will be explained in the embodiments described below.

[0046] In the training sample, the sample time is used as the label to train the prediction model. Thus, the prediction model can be used to output the time information of the target data as the prediction result.

[0047] The scheme for constructing the training sample will be described in detail below.

[0048] As Figure 2 The flowchart of the method for constructing the training sample provided by the embodiments of the present application is shown in FIG. 1. From Figure 2 It can be seen from FIG. 1 that the construction of the training sample specifically includes the following steps:

[0049] 201: Obtain the access record of the sample data in the sampling period.

[0050] 202: Set a random time in the sampling period to split the sampling period into a feature extraction period and an observation window period.

[0051] 203: Generate the sample feature based on the access record of the sample data set in the feature extraction period.

[0052] 204: Determine whether there is at least one access record for the sample data after the random time.

[0053] 205: When there is at least one access record for the sample data, determine the sample time label according to the random time and the at least one access record, and set the sample type to the non-missing type.

[0054] 206: When there is at least one access record for the sample data, the sample time label is determined according to the random time and the end time of the observation window, and the sample type is set to the censored type.

[0055] As described above, the time range of the sampling period is relatively wide, and a long history period can be regarded as a sampling period. In actual sampling, only part of the period can be sampled. In the technical solution of the present application, the access frequencies of different cold and hot data are not completely the same, so when the access frequency of a certain target data is relatively low, the obtained access record can be relatively scattered, and the access frequency can be unstable (for example, the access time intervals of three consecutive accesses are one month, two months and three months). In the technical solution of the present application, in order to fully utilize the data, the sampling period is split by a random time (that is, Pivot time in Figure 3 ), to obtain a feature extraction period (that is, history phase in Figure 3 ) and an observation window period (that is, observation phase in Figure 3 ). In actual application, there are many historical access data, and the random time can be inserted for the access records corresponding to multiple sample data. In order to facilitate understanding, one target data will be taken as an example in the following embodiment.

[0056] As Figure 3 provided by the embodiments of the present application, the sampling period is split. As can be seen from Figure 3 , the sampling period is split into a feature extraction period and an observation window period.

[0057] The feature extraction period is used to obtain the access features (such as access interval used as a dynamic feature here) of the sample data, and the size of the data belonging to the file, the table name and other static features together constitute the feature information. The longer the sampling period is, the richer the access features we can extract are.

[0058] The observation window corresponding to the observation window period is used to mark the sample time label and the sample type in the training process. The longer the sampling period is, the less the censored data is, and vice versa, which increases the proportion of censored data.

[0059] Therefore, the length of the sample collection period needs to be ensured to be long enough to ensure the quality of the generated training samples, whether it is the feature extraction period or the observation window period. In addition, the sample data in the initial stage and the end stage of the period in the observation window period and the feature extraction period are in an unstable state. For example, in the observation window period, 20% of the window length in the initial stage and 20% of the window length in the end stage, that is, 20% to 80% of the middle period trace, a random time is randomly generated in the middle to divide, and then the data corresponding to each random time is combined as a full data set.

[0060] Specifically, as Figure 4 The schematic diagram for constructing a training sample based on an observation window is provided for the embodiments of the present application. As Figure 4 The access records in the sampling period include T0, T1, T2, T3, T4, and T5 in sequence, that is, the access times of the sample data in the sampling period are 6, and the corresponding time points are T0-T5. The start time of the observation window in the observation window period is Ts, and the end time is Te. From Figure 3 As can be seen from the above, T5 is not in the observation window, in other words, the access event of T5 cannot be observed. Although T5 is not observed, the sample data still has an access event at a future time outside the observation window. Next, a random time can be uniformly set or randomly set in the observation window. In the observation window based on the same sample data, the more random times set, the more training samples that can be obtained.

[0061] For example, the random time t can be set between T0 and T1, or between T1 and T2, or between T2 and T3, or between T3 and T4, or between T4 and T5. Since Te is less than T5 and greater than T4, if the random time t is the current observation time, and t is set between T4 and T5, it means that the access event occurring after T5 cannot be observed within the observation window range.

[0062] In actual application, the random time is adjusted, and it is determined whether there is at least one access record for the sample data after the adjusted random time, so as to determine the sample time label and the sample type according to the determination result. Specifically:

[0063] After setting the random time, it is further determined whether at least one access record for the sample data can be found after the random time. For example,

[0064] If the random time t is set between T0 and T1, 4 access records can be observed in the observation window, and training sample A1 is obtained.

[0065] If the random time t is set between T1 and T2, 3 access records can be observed in the observation window, obtaining the training sample A2.

[0066] If the random time t is set between T2 and T3, 2 access records can be observed in the observation window, obtaining the training sample A3.

[0067] If the random time t is set between T3 and T4, 1 access record can be observed in the observation window, obtaining the training sample A4.

[0068] At this time, the sample types corresponding to the samples A1, A2, A3, and A4 can be set as no deletion types respectively.

[0069] If the random time t is set between T4 and T5, 0 access record can be observed in the observation window, and the access record at T5 cannot be observed, obtaining the training sample A5. At this time, the sample type corresponding to the training sample A5 can be set as a deletion type.

[0070] As can be seen, by controlling the number of generated random time points, training sets of training samples of different sizes can be obtained. Generally, the larger the size of the training set, the better the algorithm index of the prediction model, and the more accurate the prediction result.

[0071] The way of determining the sample time label corresponding to step 206 will be specifically illustrated below. The specific way of determining the sample time label includes the following steps: if the last access time in the observation window is later than the random time, marking the first time difference between the random time and the nearest access time after the random time as the sample time label, and marking the target event type as a non-deletion event.

[0072] In actual application, when setting the random time, the sample time label is further calculated according to the random time as the label of the training sample. Since the access event can be successfully observed after the random time in the observation window, there is an access record, which means that the access event can be observed, and there is no problem of data deletion, therefore, the sample type corresponding to the training sample is set as a no-deletion type. The corresponding sample time label is the first time difference between the random time and the nearest access time after the random time. Continuing to take the example, the sample time label Y1 of the training sample A1 is T2-t, the sample time label Y2 of the training sample A2 is T2-t, the sample time label Y3 of the training sample A3 is T3-t, and the sample time label Y4 of the training sample A4 is T4-t.

[0073] The way of determining the sample time label corresponding to step 207 will be specifically illustrated below. The specific way of determining the sample time label includes the following steps: if the last access time in the observation window period is earlier than the random time, marking the second time difference between the window end time and the random time as the sample time label, and marking the target event type as a missing event.

[0074] In actual application, when setting the random time, the sample time label is further calculated according to the random time as the label of the training sample. Since no access event occurs after the random time in the observation window, there is no access record, which means that the access event cannot be observed, and the data missing problem occurs, therefore, the sample type corresponding to the training sample is set as the missing type. The corresponding sample time label is the second time difference between the second time between the window end time and the random time. Continuing to take the training sample A5 as an example, the sample time label Y5 of the training sample A5 is Te-t.

[0075] Based on the above embodiment scheme, when constructing the training sample, the unobserved access record can be introduced into the training sample in the form of the first time difference and the non-missing type or the second time difference and the missing type. Although the length of the observation window is limited, the problem of sample loss does not occur in the case of missing data (some data access records are not observed). After the prediction model is trained based on the comprehensive training sample, the trained model can better predict the time information of the next access after the current observation time, thereby improving the prediction effect of the survival analysis model on future access. The scheme can be applied to a database or a cluster to identify cold and hot data in each node, and then store according to the identification results. When the trained model is used for prediction, the time information of the next access output by the prediction model is used to judge the data cold and hot category.

[0076] After the training sample is obtained through the above embodiment, the prediction model can be trained. The following will be specifically illustrated with examples.

[0077] As Figure 5 The flowchart of the training method of the prediction model provided by the embodiment of the present application is shown in FIG. 5. Figure 5 It can be seen from FIG. 5 that the method specifically includes the following steps:

[0078] 501: Constructing a training sample. 502: Inputting the training sample into a prediction model to obtain a prediction result. 503: Optimizing parameters in the prediction model according to the prediction result, a sample time label and a sample type, wherein the prediction model is used for identifying cold and hot data.

[0079] As described above, the training sample contains (x_i, y_i, e_i), where x_i represents sample features (covariates, such as visit features), y_i represents sample time label, and e_i represents sample type. In the process of constructing the training sample, different random times can be set based on the same sample data to obtain a set of training samples. Further, based on multiple sample data, a set of training samples can be obtained. The prediction model (such as a survival analysis model) is trained using the set of training samples. In the training process, y_i is used as the training label. It is easy to understand that although the prediction model is well trained by obtaining a more comprehensive training sample, the prediction model needs to be continuously optimized based on the training sample in the training process. The specific optimization process is as follows:

[0080] As Figure 6 A flowchart of a parameter optimization method of a prediction model provided by an embodiment of the present application is shown in FIG. 6. As can be seen from Figure 6 , the method specifically includes the following steps:

[0081] 601: Determine the correspondence between the prediction result and the sample time label.

[0082] 602: According to the matching result of the sample type and the correspondence, optimize the parameters in the prediction model.

[0083] It should be noted that the prediction result referred to herein is the time information of the time difference between the next visit and the current time and the sample type. In other words, the correspondence between the prediction result and the sample time label specifically includes that the prediction result is earlier than the sample time label, or the prediction result is later than the sample time label. The sample label type includes non-censored type and censored type. The specific matching result determination process will be described in the following embodiments. Figure 7 with examples.

[0084] As described above, in the training sample, in addition to the sample time label as the training sample, the sample type is also contained. The step 602 will be described in detail below with reference to the accompanying drawings. As Figure 7 A schematic diagram of the sample type and correspondence matching process provided by an embodiment of the present application is shown in FIG. 7. As can be seen from Figure 7 , the method specifically includes the following steps:

[0085] 701: If the sample type is a non-censored type, and the correspondence is that the time information corresponding to the prediction result is less than the sample time label, it is determined that the sample type and the correspondence match.

[0086] For example, assume that the training sample Y6 is (x_1, y_1, e_1) in particular, assume that y_1 = ty1 and e_1 = 0 (representing a non-censored type). The prediction result is tx, and after comparison, the corresponding relationship is that tx is less than ty1. In other words, the prediction result is that the access event to the target data will occur tx time difference after the current time t. Since tx is less than ty1, that is, the event can be observed, no censored data is generated, which is consistent with the sample type (non-censored type). That is, the sample type matches the corresponding relationship.

[0087] 702: If the sample type is a censored type and the corresponding relationship is that the time information corresponding to the prediction result is greater than the sample time label, it is determined that the sample label type matches the corresponding relationship.

[0088] For example, assume that the training sample Y7 is (x_2, y_2, e_2) in particular, assume that y_2 = ty2 and e_2 = 1 (representing a censored type). The prediction result is tx, and after comparison, the corresponding relationship is that tx is greater than ty2. In other words, the prediction result is that the access event to the target data will occur tx time difference after the current time t. Since tx is greater than ty2, that is, the event cannot be observed, censored data is generated, which is consistent with the sample type (censored type). That is, the sample label type matches the corresponding relationship.

[0089] 703: If the sample type is a non-censored type and the corresponding relationship is that the time information corresponding to the prediction result is greater than the sample time label, it is determined that the sample type does not match the corresponding relationship.

[0090] For example, assume that the training sample Y6 is (x_1, y_1, e_1) in particular, assume that y_1 = ty1 and e_1 = 0 (representing a non-censored type). The prediction result is tx, and after comparison, the corresponding relationship is that tx is greater than ty1. In other words, the prediction result is that the access event to the target data will occur tx time difference after the current time t. Since tx is greater than ty1, that is, the event cannot be observed, censored data is generated, which is not consistent with the sample type (non-censored type). That is, the sample type does not match the corresponding relationship.

[0091] 704: If the sample type is a censored type and the corresponding relationship is that the time information corresponding to the prediction result is less than the sample time label, it is determined that the sample label type does not match the corresponding relationship.

[0092] For example, assume that the training sample Y7 is (x_2, y_2, e_2) in particular, assume that y_2 = ty2 and e_2 = 1 (indicating no missing type). The prediction result is tx, and it can be known through comparison that the corresponding relationship is that tx is less than ty2. In other words, the prediction result is that the event of accessing the target data will occur after a time difference of tx from the current time t. Since tx is less than ty2, that is, the event can be observed, no missing data is generated, which is inconsistent with the sample type (no missing type). That is, the sample label type does not match the corresponding relationship.

[0093] As an optional embodiment of optimizing a pair of prediction models, the quality of the prediction model can be measured by the prediction result of the model when optimization is performed:

[0094] Due to the existence of missing data, the c-index is generally used to measure the effect of the prediction model. The c-index refers to the proportion of consistent pairs of the prediction result and the actual result in all sample pairs.

[0095] The calculation steps are as follows:

[0096] 1. All training samples are matched to obtain sample pairs. For example, if there are n samples, n*(n-1) / 2 sample pairs are generated;

[0097] 2. If the sample pair has a smaller sample time corresponding to a missing type (which means that the sample data is missing data) or both samples in the sample pair are missing data, it is considered to be an invalid pair, and the remaining samples are useful pairs.

[0098] 3. Calculate the number of pairs in which the prediction result and the actual result are consistent, that is, the individual actual sample time of the sample with a longer sample time is longer.

[0099] 4. c-index = consistent pair number / useful pair number; the range of c-index is between 0 and 1, and the closer to 1, the stronger the ability of the model to distinguish cold and hot data. In the training optimization process, the training sample is used to continuously train the prediction model so that the c-index of the prediction model is closer to 1.

[0100] After the time information is output by the prediction model, the target data will be further identified as cold and hot data according to the time information. The following will be specifically explained.

[0101] As Figure 8 The schematic diagram of cold and hot data identification provided by the embodiment of the present application can be seen from Figure 8 It can be seen from the following steps:

[0102] 801: Obtain the time information corresponding to the target data respectively.

[0103] 802: Perform hot and cold data identification on the target data according to the size of the time information.

[0104] As described in step 802, when comparing according to the size of the time information, a sorting method can be used. For example, four time information is obtained, tx1 is 10 minutes, tx2 is 20 minutes, tx3 is 30 minutes, and tx4 is 40 minutes. When sorting, the order can be from large to small or from small to large. Assuming that the order obtained by sorting from small to large is tx1, tx2, tx3, and tx4. The first 50% or the first 25% can be divided into hot data, and the remaining is divided into cold data. The specific proportion can be set according to the actual situation (such as the size of the hot data storage space). This is only an example and does not limit the technical solutions of the present application.

[0105] In addition to the sorting method, a threshold size comparison method can also be used. Specifically as follows:

[0106] Mark the target data with time information greater than the first time threshold as cold data, and store the target data to the first storage medium supporting low access performance.

[0107] Mark the target data with time information less than or equal to the first time threshold as hot data, and store the target data to the second storage medium supporting high access performance.

[0108] For example, four time information is obtained, tx1 is 10 minutes, tx2 is 20 minutes, tx3 is 30 minutes, and tx4 is 40 minutes. Assuming that the first time threshold is 25 minutes, it can be known that since tx3 and tx4 are greater than 25 minutes, the target data is cold data; since tx1 and tx2 are less than 25 minutes, the target data is hot data.

[0109] In actual application, the time information output by the prediction model can also be used to adjust the storage state of hot and cold data. Specifically, the cold data is stored and timed. Based on the difference between the storage timing and the time information corresponding to the cold data, the remaining time is determined. When the remaining time is less than a second time threshold, the cold data is migrated from the first storage medium to the second storage medium.

[0110] For example, after the prediction model outputs time information (such as 24 hours), the target data is identified as cold data and stored in the first storage medium (such as an HDD medium). The storage time for the cold data is started, and as time goes on, the storage time accumulates to 23 hours. At this time, the difference between the time information and the storage time (that is, the remaining time) is only 1 hour, and the threshold is set to 1 hour. When the remaining time is 1 hour or 59 minutes, the remaining time is less than the second time threshold, indicating that the target data corresponding to the cold data will be accessed. In order to improve the access speed of the target data, the cold data can be transferred from the first storage medium to the second storage medium (such as an SSD medium) for storing hot data.

[0111] Based on the same idea, the embodiment of the present application also provides a prediction model training method. As Figure 9 The flowchart of the prediction model training method provided by the embodiment of the present application is shown in the figure. From Figure 9 It can be seen that the specific steps include the following steps:

[0112] 901: Construct a training sample, wherein the training sample includes sample features, sample time labels, and sample types, and the sample time labels and sample types are determined by whether there is an access record of the sample feature corresponding data before and after a random time in the sample sampling period.

[0113] 902: Input the training sample into the prediction model to obtain a prediction result.

[0114] 903: Optimize the parameters in the prediction model according to the prediction result, the sample time label, and the sample type; wherein the prediction model is used to identify cold and hot data.

[0115] As described in step 903, the process of optimizing the parameters in the prediction model is as follows: determine the correspondence between the prediction result and the sample time label; and optimize the parameters in the prediction model according to the matching result of the sample type and the correspondence.

[0116] In step 901, the construction process of the training sample includes the following steps:

[0117] Obtain the access record of the sample data in the sampling period;

[0118] Set a random time in the sampling period to split the sampling period into a feature extraction period and an observation window period;

[0119] Generate sample features based on the access record in the feature extraction period of the sample data set;

[0120] searching whether there is at least one access record for the sample data after the random time point;

[0121] when there is at least one access record for the sample data, determining the sample time label according to the random time point and the at least one access record, and setting the sample type as a non-censored type;

[0122] when there is at least one access record for the sample data, determining the sample time label according to the random time point and the end time point of the observation window, and setting the sample type as a censored type.

[0123] Specifically, reference can be made to Figures 1 to 8 For corresponding embodiments, details are not repeated here.

[0124] For the convenience of understanding, the overall process of cold and hot data identification will be specifically illustrated by taking the prediction model as a survival analysis model. For example, Figure 10 The cold and hot data identification system architecture diagram illustrated by the embodiments of the present application. From Figure 10 It can be seen that the survival analysis server and the application end are included. The survival analysis server includes an object storage service (OSS), a relational database service (RDS), and a survival analysis model. In the model, a light survival analysis model can be constructed based on cox, rsf, etc. The model is simpler and has a higher model index (c-index) than the neural network survival analysis algorithm. The historical data of the application end is used as sample data, which is more Figures 1 to 9 Corresponding embodiments generate training samples, and then train the survival analysis model. The model obtained by training is optimized. Next, the target data to be identified can be identified, and cold and hot classification storage is performed according to the nodes in the shared storage.

[0125] There are two important functions in survival analysis, one is the survival function (Survival function), that is, the probability that the event does not occur before t time: S(t) = Pr(T >= t), and the other is the risk function:

[0126]

[0127] Described is the probability that the event occurs at t time

[0128] The model based on survival analysis is to fit the risk function. Taking the cox model as an example, the cox model assumes that the log-hazard is a linear relationship with the covariate (that is, the feature), that is,

[0129] h(t,X) = h0(t)exp(β1X1+β2X2+...+β k X k )

[0130] where X = (X1, X2, X3,..., X k ) is k risk factors that affect the survival time t.

[0131] Next, the maximum likelihood estimator of β can be obtained by establishing the partial likelihood function of the Cox risk model, taking the logarithm of both sides of the partial likelihood function, and then taking the partial derivative of β.

[0132] We convert the prediction of hot and cold data into a regression problem, i.e., time-to-event prediction. Due to the limited observation window of historical information, there will be a lot of censored data, i.e., the next access of some samples is not observed within the observation window, but this does not mean that the sample will not be accessed after the access window. For those samples that are not accessed after the inference time, traditional machine learning cannot utilize these samples. Survival analysis is naturally suitable for handling censored data, and is very suitable for modeling data hot and cold using survival analysis to achieve better results by fitting the CHF (cumulative hazard function).

[0133] Based on the same idea, the embodiment of the present application also provides a data processing apparatus. As Figure 11 The structure diagram of a data processing apparatus provided by the embodiment of the present application. The data processing apparatus comprises:

[0134] The determining module 1101 is configured to determine feature information according to target data and access records of the target data.

[0135] The input module 1102 is configured to input the feature information into a prediction model to obtain time information of future access of the target data; wherein the prediction model is obtained by training a training sample, the training sample comprises sample features, sample time labels and sample types, and the sample time labels and sample types are determined according to whether there is an access record of the sample features corresponding data before and after a random time in a sample sampling period.

[0136] The recognition module 1103 is configured to perform hot and cold data recognition on the target data according to the time information.

[0137] Optionally, the method further comprises a training module 1104 configured to: construct a training sample; input the training sample into a prediction model to obtain a prediction result; and optimize parameters in the prediction model according to the prediction result, a sample time label, and a sample type, wherein the prediction model is configured to identify cold and hot data.

[0138] Optionally, the training module 1104 is further configured to determine a correspondence between the prediction result and the sample time label.

[0139] The parameters in the prediction model are optimized according to a matching result of the sample type and the correspondence.

[0140] Optionally, the training module 1104 is further configured to determine that the sample type matches the correspondence if the sample type is a non-censored type and the correspondence is that time information corresponding to the prediction result is less than the sample time label.

[0141] Optionally, the training module 1104 is further configured to determine that the sample type matches the correspondence if the sample type is a censored type and the correspondence is that time information corresponding to the prediction result is greater than the sample time label.

[0142] Optionally, the method further comprises a sample construction module 1105 configured to: obtain access records of sample data in a sampling period; set a random time in the sampling period to split the sampling period into a feature extraction period and an observation window period; generate sample features based on access records of the sample data set in the feature extraction period; and determine a sample time label according to the random time and at least one access record of the sample data if there is at least one access record of the sample data after the random time, and set a sample type as a non-censored type; and determine the sample time label according to the random time and a termination time of the observation window and set the sample type as a censored type if there is at least one access record of the sample data after the random time.

[0143] Optionally, the sample construction module 1105 is further configured to adjust the random time.

[0144] The method further comprises searching for whether there is at least one access record of the sample data after the adjusted random time, so as to determine the sample time label and the sample type according to a searching result.

[0145] Optionally, the sample construction module 1105 is further configured to, if the last access time within the observation window is later than the random time point, mark a first time difference between the random time point and the last access time after the random time point as the sample time label, and mark the target event type as a non-censored event.

[0146] Optionally, the sample construction module 1105 is further configured to, if the last access time within the observation window is earlier than the random time point, mark a second time difference between the end time of the window and the random time point as the sample time label, and mark the target event type as a censored event.

[0147] Optionally, the identification module 1103 is further configured to obtain the time information corresponding to the target data respectively, and identify the target data as cold data or hot data according to the size of the time information.

[0148] Optionally, the identification module 1103 is further configured to mark the target data with the time information greater than a first time threshold as cold data, and store the target data to a first storage medium supporting low access performance; and mark the target data with the time information less than or equal to the first time threshold as hot data, and store the target data to a second storage medium supporting high access performance.

[0149] Optionally, the migration module is further configured to store a time for the cold data, determine a remaining time based on a difference between the time for storing and the time information corresponding to the cold data, and migrate the cold data from the first storage medium to the second storage medium when the remaining time is less than a second time threshold.

[0150] In an embodiment of the present application, a non-transitory machine readable storage medium is provided, and the non-transitory machine readable storage medium has stored executable codes, when the executable codes are executed by a processor of an electronic device, the processor executes the data processing method as Figures 1 to 8 The data processing method can be specifically referred to the above embodiments.

[0151] An embodiment of the present application further provides an electronic device. As shown in Figure 12 An embodiment of the present application provides a structural schematic diagram of an electronic device. The electronic device comprises a memory 1201, a processor 1202 and a communication component 1203; wherein,

[0152] The memory 1201 is configured to store a program;

[0153] The processor 1202 is coupled with the memory, and is configured to execute the program stored in the memory, so as to:

[0154] determine feature information according to the target data and the access record of the target data;

[0155] input the feature information into a prediction model to obtain time information of future access of the target data, wherein the prediction model is obtained by training a training sample, the training sample includes sample feature, sample time label and sample type, and the sample time label and the sample type are determined according to whether there is an access record of the sample feature corresponding data before and after a random time in a sample time period.

[0156] According to the time information, the hot and cold data of the target data is identified.

[0157] The memory 1201 can be configured to store other various data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0158] Further, the processor 1202 in the embodiment can be specifically: a programmable switching processing chip, wherein a data replication engine is configured in the programmable switching processing chip, and the data replication engine can replicate the received data.

[0159] In addition to the above functions, the processor 1202 can also implement other functions when executing the program in the memory. For details, refer to the description of the foregoing embodiments. Further, as shown in Figure 12 The electronic device also includes a power supply component 1204 and other components.

[0160] Based on the same idea, the embodiment of the present application also provides a prediction model training device. As Figure 13 A structure diagram of a prediction model training device provided by the embodiment of the present application. The other data processing device includes:

[0161] The sample construction module 1301 is configured to construct a training sample, wherein the training sample includes sample feature, sample time label and sample type, and the sample time label and the sample type are determined according to whether there is an access record of the sample feature corresponding data before and after a random time in a sample time period.

[0162] The input module 1302 is configured to input the training sample into a prediction model to obtain a prediction result.

[0163] The optimization module 1303 is configured to optimize parameters in the prediction model according to the prediction result, the sample time label, and the sample type, wherein the prediction model is used to identify cold and hot data.

[0164] Optionally, the optimization module 1303 is further configured to determine a correspondence between the prediction result and the sample time label, and optimize the parameters in the prediction model according to a matching result of the sample type and the correspondence.

[0165] Optionally, the sample construction module 1301 is further configured to obtain access records of sample data in a sampling period, obtain access records of sample data in a sampling period, set a random time in the sampling period to split the sampling period into a feature extraction period and an observation window period, generate sample features based on the access records of the sample data set in the feature extraction period, and determine the sample time label according to the random time and the at least one access record of the sample data, and set the sample type as a non-censored type. When there is at least one access record of the sample data, the sample time label is determined according to the random time and the termination time of the observation window, and the sample type is set as a censored type.

[0166] In an embodiment of the present application, a non-transitory machine-readable storage medium is provided, and the non-transitory machine-readable storage medium stores executable code. When the executable code is executed by a processor of an electronic device, the processor executes the method for training a prediction model as Figure 9 The prediction model training method. For details, please refer to the above embodiments.

[0167] An embodiment of the present application also provides an electronic device. The electronic device is a backup node electronic device in a computing unit. As Figure 14 Another structural schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device includes a memory 1401, a processor 1402, and a communication component 1403; wherein,

[0168] The memory 1401 is configured to store a program.

[0169] The processor 1402 is coupled with the memory and is configured to execute the program stored in the memory, so as to:

[0170] Construct a training sample, wherein the training sample includes sample features, sample time labels, and sample types, and the sample time labels and the sample types are determined according to whether there is an access record of sample feature corresponding data before and after a random time in a sample sampling period.

[0171] inputting the training sample into a prediction model to obtain a prediction result;

[0172] optimizing parameters in the prediction model according to the prediction result, the sample time label and the sample type;

[0173] The prediction model is used for identifying cold and hot data.

[0174] The memory 1401 can be configured to store other various data to support operations on the electronic device. Examples of the data include instructions for any application or method operating on the electronic device. The memory can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disc or optical disc.

[0175] Further, the processor 1402 in the embodiment can be specifically: a programmable switching processing chip, which is configured with a data replication engine and can replicate received data.

[0176] In addition to the above functions, the processor 1402 can also implement other functions when executing programs in the memory. For details, refer to the descriptions in the foregoing embodiments. Further, as shown in Figure 14 The electronic device also includes a power supply component 1404 and other components.

[0177] Based on the above embodiments, according to the feature information of the target data input into the pre-trained prediction model, the prediction model can obtain the time information of the future access of the target data, that is, the time difference between the future access and the current time. In order to enable the prediction model to accurately predict the time information of the future access, the training sample is constructed to include sample features, sample time labels and sample types, and the sample time labels and sample types are determined by whether there is an access record of the sample feature corresponding data before and after a random time in a sample sampling period. The prediction model trained by using such training samples can accurately identify the cold and hot data of the target data. Through the above scheme, the sample time label is used as the label of the training sample to train the prediction model, and then the time interval of the next access of the target data based on the prediction model is used as the prediction result, and the cold and hot data of the target data are identified according to the size of the time interval, so that the accuracy of identifying the cold and hot data of the target data can be effectively improved.

[0178] It should be noted that the data processing apparatus provided in the above embodiments can implement the technical solutions described in the above method embodiments, and the principles of the implementation of the above modules or units can be referred to the corresponding content in the above method embodiments, which will not be described here.

[0179] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0180] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and the necessary general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiment.

[0181] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, the method comprising: determining feature information according to target data and access records of the target data; inputting the feature information into a model based on survival analysis to obtain time information of future access of the target data; wherein the model based on survival analysis is trained by training samples, the training samples include sample features generated according to access records of sample data before a random time in a sample sampling period, and sample time labels and sample types determined by whether there is an access record of the sample data after the random time in the sample sampling period, the sample types include a non-censored type and a censored type; and performing cold and hot data identification on the target data according to the time information.

2. The method of claim 1, wherein the training method of the model based on survival analysis comprises: constructing training samples; inputting the training samples into the model based on survival analysis to obtain a prediction result; and optimizing parameters in the model based on survival analysis according to the prediction result, sample time labels and sample types; wherein the model based on survival analysis is used for identifying cold and hot data.

3. The method of claim 2, wherein the optimizing parameters in the model based on survival analysis according to the prediction result, sample time labels and sample types comprises: determining a correspondence between the prediction result and the sample time labels; and optimizing the parameters in the model based on survival analysis according to a matching result of the sample types and the correspondence.

4. The method of claim 3, wherein the matching result of the sample types and the correspondence comprises: if the sample type is the non-censored type and the correspondence is that the time information corresponding to the prediction result is less than the sample time label, determining that the sample type matches the correspondence; and if the sample type is the censored type and the correspondence is that the time information corresponding to the prediction result is greater than the sample time label, determining that the sample label type matches the correspondence.

5. The method of claim 2, wherein the constructing training samples comprises: obtaining access records of sample data in a sampling period; setting a random time in the sampling period to split the sampling period into a feature extraction period and an observation window period; generating sample features based on access records of the sample data set in the feature extraction period; finding whether there is at least one access record for the sample data after the random time; when there is at least one access record for the sample data, determining the sample time label according to the random time and the at least one access record, and setting the sample type as the non-censored type; and when there is at least one access record for the sample data, determining the sample time label according to the random time and a termination time of the observation window, and setting the sample type as the censored type.

6. The method of claim 5, further comprising: adjusting the random time. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ searching whether there is at least one access record of the sample data after the adjusted random time point, so as to determine a sample time label and a sample type according to a searching result.

7. The method of claim 5, wherein the determining the sample time label according to the random time point and the at least one access record comprises: if the last access time within the observation window is later than the random time point, marking a first time difference between the random time point and the last access time after the random time point as the sample time label, and marking the target event type as a non-censored event.

8. The method of claim 5, wherein the determining the sample time label according to the random time point and a termination time point of the observation window comprises: if the last access time within the observation window is earlier than the random time point, marking a second time difference between the termination time point of the observation window and the random time point as the sample time label, and marking the target event type as a censored event.

9. The method of claim 1, wherein the identifying the target data as cold or hot data according to the time information comprises: obtaining the time information corresponding to the target data respectively; and identifying the target data as cold or hot data according to a size of the time information.

10. The method of claim 1 or 9, further comprising: marking the target data with the time information greater than a first time threshold as cold data, and storing the target data to a first storage medium supporting low access performance; and marking the target data with the time information less than or equal to the first time threshold as hot data, and storing the target data to a second storage medium supporting high access performance.

11. The method of claim 10, further comprising: timing storage of the cold data; determining a remaining time based on a difference between the storage timing and the time information corresponding to the cold data; and migrating the cold data from the first storage medium to the second storage medium when the remaining time is less than a second time threshold.

12. A model training method based on survival analysis, comprising: constructing a training sample, wherein the training sample comprises a sample feature generated according to an access record of sample data before a random time within a sample sampling period, and a sample time label and a sample type determined by whether there is an access record of the sample data after the random time within the sample sampling period, the sample type comprising a non-censored type and a censored type; inputting the training sample into a model based on survival analysis to obtain a prediction result; and optimizing a parameter in the model based on survival analysis according to the prediction result, the sample time label and the sample type; wherein the model based on survival analysis is used for identifying cold or hot data. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 13. A non-transitory machine-readable storage medium having stored thereon executable code to cause a processor of an electronic device to perform the method of any of claims 1-11, or the method of claim 12.

14. An electronic device, comprising: a memory; and a processor; wherein: the memory is configured to store a program; and the processor is configured to execute the program stored in the memory to implement the method of any of claims 1-11, or the method of claim 12. ​ ​

Images