A data processing method, apparatus, and storage medium

Abstract

This application provides a data processing method, apparatus, and storage medium. The method includes: acquiring multiple sets of data to be detected generated by multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold; processing a first set of data to be detected in the multiple sets of data to be detected using a preset data prediction model to obtain a predicted data set corresponding to a second set of data to be detected in the multiple sets of data to be detected; the first set of data to be detected and the second set of data to be detected are sets of data to be detected corresponding to different data types in the multiple sets of data to be detected; sequentially comparing each set of data to be detected in the multiple sets of data to be detected with a corresponding predicted data set to obtain a comparison result, and determining abnormal data from the multiple sets of data to be detected based on the comparison result. Through the above technical solution, the accuracy of detecting abnormal data is improved.

Description

Technical Field

[0001] This application relates to the field of deep learning, and more particularly to a data processing method, apparatus, and storage medium. Background Technology

[0002] As businesses increasingly prioritize refined operations, there is a widespread need for monitoring core data and detecting anomalies. In various scenarios, including advertising and e-commerce, the ability to quickly and accurately pinpoint abnormal data is crucial.

[0003] Currently, anomaly detection relies on custom detection rules. Different rules are established for different data types, such as setting comparison thresholds. The system compares the data against these rules, and if the data does not meet the criteria, it is considered anomaly. However, when using fixed rules to locate anomalies, if no pre-defined detection rules exist for a particular data type, existing anomaly detection methods may fail to detect such anomalies, resulting in low accuracy. Summary of the Invention

[0004] This application provides a data processing method, apparatus, and storage medium that can improve the accuracy of detecting abnormal data.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a data processing method, the method comprising:

[0007] Multiple groups of data to be detected are obtained from multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold.

[0008] By using a preset data prediction model, the first data group to be detected in the plurality of data groups to be detected is processed to obtain the predicted data group corresponding to the second data group to be detected in the plurality of data groups to be detected; the first data group to be detected and the second data group to be detected are data groups to be detected corresponding to different data types in the plurality of data groups to be detected.

[0009] Each of the plurality of data groups to be detected is compared with a corresponding prediction data group in turn to obtain a comparison result, and abnormal data is determined from the plurality of data groups to be detected based on the comparison result.

[0010] In the above data processing method, before processing the first data group to be detected among the plurality of data groups to be detected using a preset data prediction model to obtain the predicted data group corresponding to the second data group to be detected among the plurality of data groups to be detected, the method further includes:

[0011] Multiple sample data groups are acquired within a sampling time period; each sample data group corresponds to a preset data type, and the sample data in each sample data group is assigned a data label, which includes normal data labels and abnormal data labels.

[0012] The first sample data group in the plurality of sample data groups is processed by the initial data prediction model to obtain the predicted sample data group corresponding to the second sample data group in the plurality of sample data groups; the first sample data group and the second sample data group are sample data groups corresponding to different preset data types in the plurality of sample data groups.

[0013] The sample data sets and the corresponding predicted sample data sets are compared to obtain the sample comparison results.

[0014] The initial data prediction model is trained using the sample comparison results and the normal data labels and / or abnormal data labels corresponding to the sample data in each sample data group to obtain the preset data prediction model.

[0015] In the above data processing method, after acquiring multiple sample data groups within the sampling time period, and before processing the first sample data group among the multiple sample data groups using an initial data prediction model to obtain the predicted sample data group corresponding to the second sample data group among the multiple sample data groups, the method further includes:

[0016] Determine the multiple preset data types corresponding to the multiple sample data groups;

[0017] Based on the reordering of the multiple preset data types, the multiple sample data groups are reordered to obtain multiple sorted sample data groups.

[0018] The step of processing the first sample data group among the plurality of sample data groups using an initial data prediction model to obtain the predicted sample data group corresponding to the second sample data group among the plurality of sample data groups includes:

[0019] The third sample data group in the sorted plurality of sample data groups is processed by the initial data prediction model to obtain the predicted sample data group corresponding to the fourth sample data group in the plurality of sample data groups. The third sample data group is at least one sample data group in the sorted plurality of sample data groups that precedes the fourth sample data group.

[0020] In the above data processing method, the first data group to be detected is at least one data group to be detected; the step of processing the first data group to be detected among the plurality of data groups to be detected using a preset data prediction model to obtain the predicted data group corresponding to the second data group to be detected among the plurality of data groups to be detected includes:

[0021] From the at least one group of data to be detected, at least one data point corresponding to each preset time point within the preset time period is determined sequentially;

[0022] Using the preset data prediction model, at least one data point to be detected corresponding to each preset time point is used to predict one predicted data point corresponding to each preset time point in the second data group to be detected, thereby obtaining the predicted data group corresponding to the second data group to be detected.

[0023] In the above data processing method, the step of sequentially comparing each of the plurality of data groups to be detected with a corresponding predicted data group to obtain a comparison result, and determining abnormal data from the plurality of data groups to be detected based on the comparison result, includes:

[0024] From the plurality of data to be detected, find the plurality of data to be detected corresponding to each preset time point within the preset time period, and combine the plurality of data to be detected into a first combination of data;

[0025] From the multiple predicted data groups corresponding to the multiple data groups to be detected, find multiple predicted data corresponding to each preset time point within the preset time period, and combine the multiple predicted data into a second combined data;

[0026] If the difference between the first combined data and the corresponding second combined data is greater than a first preset difference threshold, then the plurality of data to be detected corresponding to the first combined data are determined as the abnormal data.

[0027] In the above data processing method, the step of sequentially comparing each of the plurality of data groups to be detected with a corresponding predicted data group to obtain a comparison result, and determining abnormal data from the plurality of data groups to be detected based on the comparison result, includes:

[0028] Sequentially, from each group of data to be detected and its corresponding group of predicted data, find one data to be detected and one predicted data corresponding to each preset time point within the preset time period;

[0029] If the difference between the first data to be detected and the corresponding first predicted data is greater than a second preset difference threshold, then the first data to be detected is determined to be the abnormal data.

[0030] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0031] The acquisition unit is used to acquire multiple groups of data to be detected generated by multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold.

[0032] The data processing unit is used to process the first data group to be detected in the plurality of data groups to be detected through a preset data prediction model to obtain the predicted data group corresponding to the second data group to be detected in the plurality of data groups to be detected; the first data group to be detected and the second data group to be detected are data groups to be detected corresponding to different data types in the plurality of data groups to be detected.

[0033] The comparison unit is used to sequentially compare each of the plurality of data groups to be detected with a corresponding prediction data group to obtain a comparison result;

[0034] A determining unit is used to determine abnormal data from the plurality of data groups to be detected based on the comparison results.

[0035] In the aforementioned data processing apparatus, the apparatus further includes: a model training unit;

[0036] The acquisition unit is also used to acquire multiple sample data groups within the sampling time period; each sample data group in the multiple sample data groups corresponds to a preset data type, and the sample data in each sample data group corresponds to a set data label, the data label including normal data label and abnormal data label;

[0037] The data processing unit is further configured to process the first sample data group among the plurality of sample data groups using an initial data prediction model to obtain a predicted sample data group corresponding to the second sample data group among the plurality of sample data groups; the first sample data group and the second sample data group are sample data groups corresponding to different preset data types among the plurality of sample data groups.

[0038] The comparison unit is also used to compare the plurality of sample data groups and the plurality of predicted sample data groups corresponding to the plurality of sample data groups respectively, and to obtain sample comparison results;

[0039] The model training unit is used to train the initial data prediction model using the sample comparison results and the normal data labels and / or abnormal data labels corresponding to the sample data in each sample data group, so as to obtain the preset data prediction model.

[0040] Thirdly, embodiments of this application provide a data processing device, characterized in that the device includes: a processor, a memory, and a communication bus; the processor executes a running program stored in the memory to implement the data processing method as described in any of the above claims.

[0041] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the data processing method as described in any of the preceding claims.

[0042] This application provides a data processing method, apparatus, and storage medium. The method includes: acquiring multiple sets of data to be detected generated by multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold; processing a first set of data to be detected in the multiple sets of data to be detected using a preset data prediction model to obtain a predicted data set corresponding to a second set of data to be detected in the multiple sets of data to be detected; the first set of data to be detected and the second set of data to be detected are sets of data to be detected corresponding to different data types in the multiple sets of data to be detected; sequentially comparing each set of data to be detected in the multiple sets of data to be detected with a corresponding predicted data set to obtain a comparison result, and determining abnormal data from the multiple sets of data to be detected based on the comparison result; and using the above... The proposed solution first inputs the data sets to be detected corresponding to different data types into a preset data prediction model. This model, based on the type correlation between different data types, predicts data sets of different data types using data sets corresponding to at least one data type, thus obtaining the predicted data set corresponding to that data set. Then, the data set to be detected is compared with the corresponding predicted data set. Finally, based on the comparison result, the abnormal data in the data set to be detected is determined. This solution enables the detection of abnormal data based on the type correlation between different data types. Therefore, it eliminates the need to preset detection rules for different data types as in existing technologies, thereby improving the accuracy of detecting abnormal data. Attached Figure Description

[0043] Figure 1 A flowchart of a data processing method provided in an embodiment of this application;

[0044] Figure 2A flowchart illustrating an exemplary data processing method provided in this application embodiment;

[0045] Figure 3 An exemplary flowchart for training an initial data prediction model is provided for embodiments of this application;

[0046] Figure 4 This is a schematic diagram of the composition structure of a data processing device provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the composition structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the scope of this application.

[0049] This application provides a data processing method applied to a data processing device. Figure 1 A flowchart of a data processing method provided in an embodiment of this application is shown below. Figure 1 As shown, data processing methods may include:

[0050] S101. Obtain multiple groups of data to be detected generated by multiple data types within a preset time period; the type correlation between multiple data types is greater than a preset correlation threshold.

[0051] In this embodiment, the data processing device acquires multiple groups of data to be detected generated by multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold.

[0052] The data processing method proposed in this application can be applied to scenarios where abnormal data is searched from data of different data types.

[0053] In this application embodiment, an exemplary data processing method flowchart is provided, such as... Figure 2 As shown, the data processing method comprises three parts: data processing, model building and training, and anomaly detection. The data processing part segments the sample data according to the sampling time period, obtaining multiple sample data groups corresponding to each sampling time period. These multiple sample data groups are then stored as a dictionary, and in practice, the dictionary index can be used as input for subsequent model training. The model building and training part trains an initial data prediction model based on the data correlations between multiple sample data groups and between sample data groups corresponding to different data types, thus obtaining a preset data prediction model. The anomaly detection part uses the trained preset data prediction model to find abnormal data in the data to be detected.

[0054] It should be noted that, in the embodiments of this application, the data types can be the temperature of the central processing unit (CPU), the operating speed of the CPU, and the read / write speed of the hard disk, etc. The type correlation between different data types is greater than a preset correlation threshold. For example, if the CPU temperature is too high, it will lead to a decrease in the operating speed of the CPU and the read / write speed of the hard disk. The specific data types and preset correlation thresholds are determined according to the actual situation, and this embodiment of the application does not make specific limitations here.

[0055] It should be noted that the preset time period can be 1 minute, 30 seconds, or 10 seconds. The preset time period includes multiple preset time points. Assuming that the time point unit is seconds, then 1 minute includes 60 preset time points, 30 seconds includes 30 preset time points, and 10 seconds includes 10 preset time points. The specific preset time period and preset time point unit can be determined according to the actual situation. This application embodiment does not make specific limitations here.

[0056] It should be noted that in this embodiment of the application, it is assumed that there are 3 data types and the preset time period is 10 seconds. The number of data groups to be detected is 3, and each data group to be detected contains 10 data to be detected. One preset time point corresponds to one data to be detected.

[0057] S102. By using a preset data prediction model, the first data group to be detected in the multiple data groups to be detected is processed to obtain the predicted data group corresponding to the second data group to be detected in the multiple data groups to be detected; the first data group to be detected and the second data group to be detected are data groups to be detected corresponding to different data types in the multiple data groups to be detected.

[0058] In this embodiment of the application, after acquiring multiple sets of data to be detected, the data processing device processes the first set of data to be detected in the multiple sets of data to be detected through a preset data prediction model to obtain the predicted data set corresponding to the second set of data to be detected in the multiple sets of data to be detected; the first set of data to be detected and the second set of data to be detected are sets of data to be detected corresponding to different data types in the multiple sets of data to be detected.

[0059] In this embodiment, the preset data prediction model is obtained by training an initial data prediction model based on the sample data set. Figure 3 An exemplary flowchart for training an initial data prediction model is provided for embodiments of this application. The specific training method is as follows:

[0060] S201. Obtain multiple sample data groups within the sampling time period; each sample data group corresponds to a preset data type, and the sample data in each sample data group corresponds to a data label, which includes normal data labels and abnormal data labels.

[0061] In this embodiment, the data processing device acquires multiple sample data groups within a sampling time period; each sample data group corresponds to a preset data type, and the sample data in each sample data group is assigned a data label, which includes normal data labels and abnormal data labels.

[0062] It should be noted that the initial data set used to train the prediction model is quite large, potentially reaching 20,000 or even 30,000 samples. Directly inputting this data into the initial prediction model would make it overly complex during training, making it difficult to converge to the desired result. Therefore, it is necessary to... Figure 2 The data processing section preprocesses the sample data and divides it into multiple sample data sets according to the sampling time period. Each sample data set corresponding to a sampling time period includes multiple sample data groups.

[0063] It should be noted that the sampling time period can be 1 minute, 30 seconds, or 10 seconds. The sampling time period includes multiple sampling time points. Assuming that the time point unit is seconds, then 1 minute includes 60 sampling time points, 30 seconds includes 30 sampling time points, and 10 seconds includes 10 sampling time points. The specific sampling time period and sampling time point unit can be determined according to the actual situation. This application embodiment does not make specific limitations here.

[0064] It should be noted that when segmenting sample data, the sample data can be segmented at each sampling time interval, or it can be segmented at every sampling time point, with the sampling time interval as the unit. For example, suppose we obtain sample data M generated by two data types A and B within 10 seconds, as shown in Formula 1:

[0065]

[0066] In the above formula (1), the first row is the sample data generated by data type A within 10 seconds, and the second row is the sample data generated by data type B within 10 seconds.

[0067] Assuming the sampling time period is 5 seconds, the sample data M is divided into two sample data groups M1 and M2 every sampling time period, as shown in formula (2):

[0068]

[0069] Assuming the sampling time period is 5 seconds, the sample data M is divided into 6 sample data groups M1, M2, M3, M4, M5 and M6 at each sampling time point, as shown in formula (3):

[0070]

[0071] It should be noted that the specific segmentation method can be determined according to the actual situation, and the embodiments of this application do not impose specific limitations here.

[0072] It should be noted that after the sample data is segmented to obtain multiple sample data sets, a numerical index is created for each sample data set and its corresponding sampling time period. The numerical index and the corresponding sample data set are stored in a dictionary. When inputting the model later, the corresponding sample data set can be directly selected as the model input sample data based on the numerical index.

[0073] It should be noted that, in the embodiments of this application, one sampling time period corresponds to multiple sample data groups. For example, multiple sample data groups are defined as X = {x} 1 ,x 2 ,...,x N}, where N represents the number of subsequences contained in X, i.e., the number of preset data types, and a sample data group x in X. i It is a sequence of length T. Where T represents the number of sampling time points within the sampling time period.

[0074] It should be noted that, assuming the sample data can be divided into L sample data sets according to the sampling time period, then X = {X...} [t:t+T] ,X [t+T:t+2T] ,...,X [t+(L-1)T:t+LT]}, where t is the starting sampling time of the sample data, and the sampling time point is in seconds.

[0075] S202. Using the initial data prediction model, the first sample data group in the multiple sample data groups is processed to obtain the predicted sample data group corresponding to the second sample data group in the multiple sample data groups; the first sample data group and the second sample data group are sample data groups corresponding to different preset data types in the multiple sample data groups.

[0076] In this embodiment of the application, after obtaining multiple sample data groups, the data processing device processes the first sample data group among the multiple sample data groups through an initial data prediction model to obtain the predicted sample data group corresponding to the second sample data group among the multiple sample data groups; the first sample data group and the second sample data group are sample data groups corresponding to different preset data types among the multiple sample data groups.

[0077] It should be noted that, in the embodiments of this application, the initial data prediction model is a deep autoregressive model based on the attention mechanism. The specific initial data prediction model can be selected according to the actual situation, and this application does not make any specific limitations here.

[0078] It should be noted that after the data processing device obtains multiple sample data groups corresponding to a sampling time period, it inputs the multiple sample data groups into the initial data prediction model. The initial data prediction model uses the first sample data group to predict the second sample data group, thus obtaining the predicted sample data group corresponding to the second sample data group.

[0079] It should be noted that, since there is data correlation between sample data groups corresponding to multiple different preset data types, it is possible to predict sample data groups corresponding to another preset data type using sample data groups corresponding to one preset data type.

[0080] It should be noted that the number of the first sample data set can be one or more, and the specific number can be determined according to the actual situation. This application does not make a specific limitation here; for example, assume that the multiple sample data sets input to the initial data prediction model are X = {x 1 ,x 2 ,...,x N In this embodiment of the application, a preset number of sample data groups can be used to predict subsequent sample data groups, that is, using x 1 Predict x 2 x 1 and x 2 Predict x 3 x 1 x 2 and x 3 Predict x 4 And so on, until multiple predicted sample data sets are obtained corresponding to multiple sample data sets.

[0081] In this embodiment of the application, when using the initial data prediction model for prediction, if only a preset number of sample data groups before the multiple sample data groups are used to predict the subsequent sample data groups, the influence of the preceding and following sample data groups on the intermediate sample data groups will be lost, resulting in a decrease in the prediction accuracy of the initial data prediction model. Therefore, before inputting the multiple sample data groups into the initial data prediction model, the multiple sample data groups can be randomly sorted. Specifically, multiple preset data types corresponding to the multiple sample data groups are determined; based on the reordering of the multiple preset data types, the multiple sample data groups are reordered to obtain the sorted multiple sample data groups.

[0082] It should be noted that, for example, we assume multiple sample data sets are X = {x} 1 ,x 2 ,x 3 ,x 4}, where x 1 x 2 x3 and x 4 These correspond to data types 1, 2, 3, and 4 respectively. By reordering the data types, we obtain the sorted data types 1, 3, 2, and 4. At this point, X = {x} 1 ,x 2 ,x 3 ,x 4 Then, reordering yields X = {x} 1 ,x 3 ,x 2 ,x 4}

[0083] In this embodiment of the application, after reordering multiple sample data groups, the sorted multiple sample data groups are input into the initial data prediction model; specifically, the initial data prediction model processes the third sample data group in the sorted multiple sample data groups to obtain the predicted sample data group corresponding to the fourth sample data group in the multiple sample data groups, where the third sample data group is at least one sample data group in the sorted multiple sample data groups that precedes the fourth sample data group.

[0084] It should be noted that, assuming the sorted sample data sets are X = {x} 1 ,x 3 ,x 2 ,x 4}, then when X = {x 1 ,x 3 ,x 2 ,x 4 After inputting the initial data and predicting the model, use x 1 Predict x 3 Using x 1 and x 3 Predict x 2 That is, predicting x 2 The model considered the data types before and after the initial data, thereby improving the accuracy of the prediction model based on the initial data.

[0085] It should be noted that in the actual training process of the initial data prediction model, it is necessary to concatenate the sampling time point and sample data corresponding to each sample data within the sampling time period to obtain multiple sorted and concatenated sample data groups, and then re-input the multiple sorted and concatenated sample data groups into the initial data prediction model.

[0086] It should be noted that since the initial data prediction model is executed in parallel, by concatenating the sampling time point sequence and the sample data, the sample prediction data can be trained based on the sample data corresponding to the sampling time points during the execution of the initial data prediction model. For example, suppose the sample data groups M1 and M2 are concatenated with their corresponding sampling time point sequences, as shown in Equation 4:

[0087]

[0088] In equation (4) above, a1, a2, a3, a4, a5, a6, a7, a8, a9 and a 10 The sampling times for data type A are t1, t2, t3, t4, t5, t6, t7, t8, t9, and t1. 10 Sample data collected at location b1, b2, b3, b4, b5, b6, b7, b8, b9, and b 10 The sampling times for data type A are t1, t2, t3, t4, t5, t6, t7, t8, t9, and t1. 10 Sample data collected at the location.

[0089] In this embodiment of the application, after obtaining multiple sorted and concatenated sample data groups, the multiple sorted and concatenated sample data groups are input into the initial data prediction model for prediction.

[0090] It should be noted that, assuming the multiple sorted and concatenated sample data sets M3 are as shown in Formula 5:

[0091]

[0092] In the above formula (5), c1, c2, c3, c4 and c5 are the sample data of data type C collected at sampling time points t1, t2, t3, t4 and t5, respectively. If it is necessary to predict the second spliced ​​sample data group in the second row through the first spliced ​​sample data group in the first row of M3, specifically, the predicted data of each sampling time point in the second row is predicted by using one sample data corresponding to each sampling time point in the first row, or the predicted data of each sampling time point in the third row is predicted by using two sample data corresponding to each sampling time point in the first row and the second row.

[0093] S203. Compare multiple sample data groups and multiple predicted sample data groups corresponding to multiple sample data groups respectively to obtain sample comparison results.

[0094] In this embodiment of the application, after obtaining multiple predicted sample data groups corresponding to multiple sample data groups, the data processing device compares the multiple sample data groups and the multiple predicted sample data groups corresponding to the multiple sample data groups respectively to obtain sample comparison results.

[0095] It should be noted that a sample data and its corresponding predicted sample data can be compared sequentially. In this embodiment, the mean squared error is used as the objective function. By analyzing the mean squared error between a sample data and a predicted sample data, a sample data is determined to be abnormal data if the mean squared error is greater than a preset value.

[0096] S204. Using the sample comparison results and the normal data labels and / or abnormal data labels corresponding to the sample data in each sample data group, train the initial data prediction model to obtain the preset data prediction model.

[0097] In this embodiment of the application, after obtaining the sample comparison results between multiple sample data groups and multiple predicted sample data groups, the data processing device uses the sample comparison results, the normal data labels and / or abnormal data labels corresponding to the sample data in each sample data group to train the initial data prediction model and obtain the preset data prediction model.

[0098] It should be noted that after identifying abnormal data from the sample data based on the initial data prediction model, the prediction is successful if the data label corresponding to the abnormal data is an abnormal data label, and it fails if the data label corresponding to the abnormal data is a normal data label. In this case, the initial data prediction model needs to be trained again to finally obtain the preset data prediction model.

[0099] In this embodiment of the application, after obtaining the preset data prediction model, the first data group to be detected is processed using the preset data prediction model to predict the second data group to be detected. Specifically, from at least one data group to be detected, at least one data to be detected corresponding to each preset time point within a preset time period is determined sequentially. Using the preset data prediction model, at least one data to be detected corresponding to each preset time point is used to predict one predicted data corresponding to each preset time point in the second data group to be detected, thereby obtaining the predicted data group corresponding to the second data group to be detected.

[0100] It should be noted that, assuming multiple sets of data to be detected, M4, are as shown in Formula 6:

[0101]

[0102] In equation (6) above, a 11 a 12 a 13 a 14 and a 15 For data type A at a preset time point t 11 t 12 t13 t 14 and t 15 The data to be detected generated at the location; b 11 b 12 b 13 b 14 and b 15 For data type B at the preset time point t 11 t 12 t 13 t 14 and t 15 The data to be detected generated at the location; c 11 c 12 c 13 c 14 and c 15 For data type C at a preset time point t 11 t 12 t 13 t 14 and t 15 The data to be detected is generated at the location; if the first data group to be detected is the data group to be predicted in the first and second rows of M4, and the second data group to be detected is the data group to be predicted in the third row of M4, then after inputting M4 into the preset data prediction model, the data group to be detected in the third row is predicted by the data group to be predicted in the first and second rows of M4. Specifically, the predicted data for each preset time point in the data group to be detected in the first and second rows is predicted by the two data groups to be detected at each preset time point in the third row, so as to obtain the predicted data group corresponding to the data group to be detected in the third row.

[0103] S103. Each of the multiple sets of data to be detected is compared with a corresponding set of predicted data in turn to obtain the comparison results, and abnormal data is identified from the multiple sets of data to be detected based on the comparison results.

[0104] In this embodiment of the application, after obtaining the predicted data group corresponding to the data group to be detected, the data processing device sequentially compares each data group to be detected with a corresponding predicted data group to obtain a comparison result, and determines abnormal data from the multiple data groups to be detected based on the comparison result.

[0105] In one optional embodiment, the specific method for determining abnormal data is as follows: from multiple groups of data to be detected, find multiple data to be detected corresponding to each preset time point within a preset time period, and combine the multiple data to be detected into a first group of data; from multiple groups of predicted data corresponding to the multiple groups of data to be detected, find multiple predicted data corresponding to each preset time point within a preset time period, and combine the multiple predicted data into a second group of data; if the difference between the first group of data and the corresponding second group of data is greater than a first preset difference threshold, then the multiple data to be detected corresponding to the first group of data are determined to be abnormal data.

[0106] It should be noted that in the embodiments of this application, the data combination is an addition calculation method. The specific data combination method can be determined according to the actual situation, and the embodiments of this application do not make specific limitations here.

[0107] It should be noted that multiple data types are sequentially combined at each preset time point to obtain multiple first combination data and multiple second combination data. When it is determined that the difference between the first combination data and the second combination data at the first preset time point is greater than the second preset threshold, it is determined that the data to be detected is abnormal at the first preset time point. Therefore, the multiple data to be detected at the first preset time point are all abnormal data.

[0108] In another optional embodiment, the specific method for determining abnormal data is as follows: sequentially from each group of data to be detected and a corresponding group of predicted data, find a data to be detected and a predicted data corresponding to each preset time point within a preset time period; if the difference between the first data to be detected and the corresponding first predicted data is greater than a second preset difference threshold, then the first data to be detected is determined to be abnormal data.

[0109] It should be noted that, in the embodiments of this application, the difference between the first data to be detected and the first predicted data is the difference between the first data to be detected and the first predicted data. The specific rules for calculating the difference can be determined according to the actual situation, and the embodiments of this application do not make specific limitations here.

[0110] It should be noted that the first preset difference threshold and the second preset difference threshold can be determined according to the actual situation, and the embodiments of this application do not impose specific limitations here.

[0111] This application provides a data processing method, which includes: acquiring multiple sets of data to be detected generated by multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold; processing a first set of data to be detected in the multiple sets of data to be detected using a preset data prediction model to obtain a predicted data set corresponding to a second set of data to be detected in the multiple sets of data to be detected; the first set of data to be detected and the second set of data to be detected are sets of data to be detected corresponding to different data types in the multiple sets of data to be detected; sequentially comparing each set of data to be detected in the multiple sets of data to be detected with a corresponding predicted data set to obtain a comparison result, and determining abnormal data from the multiple sets of data to be detected based on the comparison result; the above implementation method... The proposed solution first inputs the data sets to be detected corresponding to different data types into a preset data prediction model. The preset data prediction model can predict the data sets to be detected that are of different data types based on the type correlation between different data types, using the data sets to be detected corresponding to at least one data type. This yields the predicted data sets corresponding to the data sets to be detected. Then, the data sets to be detected are compared with the corresponding predicted data sets. Finally, based on the comparison results, the abnormal data in the data sets to be detected are determined. Through the technical solution of this application, abnormal data can be detected based on the type correlation between different data types. Therefore, it is not necessary to preset detection rules for different data types as in existing technologies, thereby improving the accuracy of detecting abnormal data.

[0112] Based on the above embodiments, in another embodiment of this application, a data processing device 1 is provided. Figure 4 This application provides a schematic diagram of the composition of a data processing device, as shown below. Figure 4 As shown, the data processing device 1 includes:

[0113] Acquisition unit 10 is used to acquire multiple groups of data to be detected generated by multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold;

[0114] Data processing unit 11 is used to process the first data group to be detected in the plurality of data groups to be detected through a preset data prediction model to obtain the predicted data group corresponding to the second data group to be detected in the plurality of data groups to be detected; the first data group to be detected and the second data group to be detected are data groups to be detected corresponding to different data types in the plurality of data groups to be detected.

[0115] Comparison unit 12 is used to sequentially compare each of the plurality of data groups to be detected with a corresponding prediction data group to obtain a comparison result;

[0116] The determining unit 13 is used to determine abnormal data from the plurality of data groups to be detected based on the comparison results.

[0117] Optionally, the data processing device 1 further includes: a model training unit;

[0118] The acquisition unit 10 is also used to acquire multiple sample data groups within the sampling time period; each sample data group in the multiple sample data groups corresponds to a preset data type, and the sample data in each sample data group corresponds to a set data label, the data label including normal data label and abnormal data label;

[0119] The data processing unit 11 is further configured to process the first sample data group among the plurality of sample data groups through an initial data prediction model to obtain a predicted sample data group corresponding to the second sample data group among the plurality of sample data groups; the first sample data group and the second sample data group are sample data groups corresponding to different preset data types among the plurality of sample data groups.

[0120] The comparison unit 12 is also used to compare the plurality of sample data groups and the plurality of predicted sample data groups corresponding to the plurality of sample data groups respectively to obtain sample comparison results;

[0121] The model training unit is used to train the initial data prediction model using the sample comparison results and the normal data labels and / or abnormal data labels corresponding to the sample data in each sample data group, so as to obtain the preset data prediction model.

[0122] Optionally, the data processing device 1 further includes: a sorting unit;

[0123] The determining unit 13 is also used to determine multiple preset data types corresponding to the multiple sample data groups;

[0124] The sorting unit is used to reorder the multiple sample data groups based on the reordering of the multiple preset data types, so as to obtain multiple sorted sample data groups.

[0125] The data processing unit 11 is further configured to process the third sample data group in the sorted plurality of sample data groups through an initial data prediction model to obtain the predicted sample data group corresponding to the fourth sample data group in the plurality of sample data groups, wherein the third sample data group is at least one sample data group in the sorted plurality of sample data groups that precedes the fourth sample data group.

[0126] Optionally, the determining unit 13 is further configured to sequentially determine at least one data point to be detected corresponding to each preset time point within the preset time period from the at least one group of data to be detected;

[0127] The data processing unit 11 is further configured to use the preset data prediction model to predict a predicted data corresponding to each preset time point in the second data group to be detected, using at least one data to be detected corresponding to each preset time point, so as to obtain the predicted data group corresponding to the second data group to be detected.

[0128] Optionally, the data processing device 1 further includes: a lookup unit;

[0129] The search unit is used to search for multiple data to be detected corresponding to each preset time point within the preset time period from the multiple data to be detected groups, and combine the multiple data to be detected into a first combination data; and to search for multiple predicted data corresponding to each preset time point within the preset time period from the multiple data to be detected groups, and combine the multiple predicted data into a second combination data.

[0130] The determining unit 13 is further configured to determine the plurality of data to be detected corresponding to the first combined data as the abnormal data if the difference between the first combined data and the corresponding second combined data is greater than a first preset difference threshold.

[0131] Optionally, the search unit is further configured to sequentially search for a piece of data to be detected and a piece of data to be predicted at each preset time point within the preset time period from each group of data to be detected and a corresponding group of predicted data.

[0132] The determining unit 13 is further configured to determine the first data to be detected as the abnormal data if the difference between the first data to be detected and the corresponding first predicted data is greater than a second preset difference threshold.

[0133] This application provides a data processing apparatus, comprising: an acquisition unit, a data processing unit, a comparison unit, and a determination unit. The acquisition unit is configured to acquire multiple sets of data to be detected generated from multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold. The data processing unit is configured to process a first set of data to be detected from the multiple sets of data to be detected using a preset data prediction model to obtain a predicted data set corresponding to a second set of data to be detected from the multiple sets of data to be detected; the first set of data to be detected and the second set of data to be detected are sets of data to be detected corresponding to different data types from the multiple sets of data to be detected. The comparison unit is configured to sequentially compare each set of data to be detected with a corresponding predicted data set from the multiple sets of data to be detected to obtain a comparison result. The determination unit is configured to determine the result based on the comparison. The result is that abnormal data is identified from multiple sets of data to be detected. Using the above implementation scheme, the sets of data to be detected corresponding to different data types are first input into a preset data prediction model. This preset data prediction model can predict the data sets of different data types based on the type correlation between different data types, using the data sets of at least one data type to obtain the predicted data sets corresponding to the data sets of the data sets to be detected. Then, the data sets to be detected are compared with the corresponding predicted data sets, and finally, the abnormal data in the data sets to be detected is determined based on the comparison results. Through the technical solution of this application, abnormal data can be detected based on the type correlation between different data types. Therefore, it is not necessary to preset detection rules for different data types as in existing technologies, thus achieving the goal of improving the accuracy of detecting abnormal data.

[0134] Figure 5 This is a schematic diagram of the composition structure of a data processing device provided in an embodiment of this application. In practical applications, based on the same disclosed concept of the above embodiments, such as... Figure 5 As shown, the data processing device 2 in this embodiment includes: a processor 20, a memory 21, and a communication bus 22.

[0135] In a specific embodiment, the acquisition unit 10, data processing unit 11, comparison unit 12, determination unit 13, model training unit, sorting unit, and search unit described above can be implemented by a processor 20 located on the data processing device 2. The processor 20 can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different data processing devices, the electronic devices used to implement the above processor functions can also be other types; this embodiment does not specifically limit them.

[0136] In this embodiment, the communication bus 22 is used to establish a connection between the processor 20 and the memory 21; when the processor 20 executes the running program stored in the memory 21, it implements the following data processing method:

[0137] Multiple groups of data to be detected are obtained from multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold.

[0138] By using a preset data prediction model, the first data group to be detected in the plurality of data groups to be detected is processed to obtain the predicted data group corresponding to the second data group to be detected in the plurality of data groups to be detected; the first data group to be detected and the second data group to be detected are data groups to be detected corresponding to different data types in the plurality of data groups to be detected.

[0139] Each of the plurality of data groups to be detected is compared with a corresponding prediction data group in turn to obtain a comparison result, and abnormal data is determined from the plurality of data groups to be detected based on the comparison result.

[0140] Optionally, the processor 20 is further configured to acquire multiple sample data groups within a sampling time period; each sample data group corresponds to a preset data type, and the sample data in each sample data group is assigned a data label, which includes normal data labels and abnormal data labels; the first sample data group in the multiple sample data groups is processed through an initial data prediction model to obtain a predicted sample data group corresponding to a second sample data group in the multiple sample data groups; the first sample data group and the second sample data group are sample data groups corresponding to different preset data types in the multiple sample data groups; the multiple sample data groups and the multiple predicted sample data groups corresponding to the multiple sample data groups are compared respectively to obtain a sample comparison result; the initial data prediction model is trained using the sample comparison result, the normal data labels and / or abnormal data labels corresponding to the sample data in each sample data group to obtain the preset data prediction model.

[0141] Optionally, the processor 20 is further configured to determine multiple preset data types corresponding to the multiple sample data groups; reorder the multiple sample data groups based on the reordering of the multiple preset data types to obtain multiple sorted sample data groups; the step of processing the first sample data group in the multiple sample data groups through the initial data prediction model to obtain the predicted sample data group corresponding to the second sample data group in the multiple sample data groups includes: processing the third sample data group in the sorted multiple sample data groups through the initial data prediction model to obtain the predicted sample data group corresponding to the fourth sample data group in the multiple sample data groups, wherein the third sample data group is at least one sample data group in the sorted multiple sample data groups that precedes the fourth sample data group.

[0142] Optionally, the processor 20 is further configured to sequentially determine at least one data point corresponding to each preset time point within the preset time period from the at least one data group to be detected; and predict a predicted data point corresponding to each preset time point in the second data group to be detected using the preset data prediction model, thereby obtaining a predicted data group corresponding to the second data group to be detected.

[0143] Optionally, the processor 20 is further configured to: search for multiple data points corresponding to each preset time point within the preset time period from the multiple data groups to be detected, and combine the multiple data points to be detected into a first combined data; search for multiple predicted data points corresponding to each preset time point within the preset time period from the multiple predicted data groups corresponding to the multiple data groups to be detected, and combine the multiple predicted data into a second combined data; if the difference between the first combined data and the corresponding second combined data is greater than a first preset difference threshold, then the multiple data points to be detected corresponding to the first combined data are determined as the abnormal data.

[0144] Optionally, the processor 20 is further configured to sequentially search for a piece of data to be detected and a piece of data to be predicted at each preset time point within the preset time period from each group of data to be detected and a corresponding group of predicted data; if the difference between the first piece of data to be detected and the corresponding first predicted data is greater than a second preset difference threshold, then the first piece of data to be detected is determined to be the abnormal data.

[0145] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors and applied in a data processing device. The computer program implements the data processing method described above.

[0146] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the data processing methods described in the various embodiments of this disclosure.

[0148] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Multiple groups of data to be detected are obtained from multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold. By using a preset data prediction model, the first data group to be detected in the plurality of data groups to be detected is processed to obtain the predicted data group corresponding to the second data group to be detected in the plurality of data groups to be detected; the first data group to be detected and the second data group to be detected are data groups to be detected corresponding to different data types in the plurality of data groups to be detected; wherein, the data type includes at least the temperature of the central processing unit, the speed of the central processing unit, and the read and write speed of the hard disk; Each of the plurality of data groups to be detected is compared with a corresponding prediction data group in turn to obtain a comparison result, and abnormal data is determined from the plurality of data groups to be detected based on the comparison result.

2. The method according to claim 1, characterized in that, Before processing the first data group to be detected among the plurality of data groups to be detected using a preset data prediction model to obtain the predicted data group corresponding to the second data group to be detected among the plurality of data groups to be detected, the method further includes: Multiple sample data groups are acquired within a sampling time period; each sample data group corresponds to a preset data type, and the sample data in each sample data group is assigned a data label, which includes normal data labels and abnormal data labels. The first sample data group in the plurality of sample data groups is processed by the initial data prediction model to obtain the predicted sample data group corresponding to the second sample data group in the plurality of sample data groups; the first sample data group and the second sample data group are sample data groups corresponding to different preset data types in the plurality of sample data groups. The sample data sets and the corresponding predicted sample data sets are compared to obtain the sample comparison results. The initial data prediction model is trained using the sample comparison results and the normal data labels and / or abnormal data labels corresponding to the sample data in each sample data group to obtain the preset data prediction model.

3. The method according to claim 2, characterized in that, After acquiring multiple sample data groups within the sampling time period, and before processing the first sample data group among the multiple sample data groups using an initial data prediction model to obtain the predicted sample data group corresponding to the second sample data group among the multiple sample data groups, the method further includes: Determine the multiple preset data types corresponding to the multiple sample data groups; Based on the reordering of the multiple preset data types, the multiple sample data groups are reordered to obtain multiple sorted sample data groups. The step of processing the first sample data group among the plurality of sample data groups using an initial data prediction model to obtain the predicted sample data group corresponding to the second sample data group among the plurality of sample data groups includes: The third sample data group in the sorted plurality of sample data groups is processed by the initial data prediction model to obtain the predicted sample data group corresponding to the fourth sample data group in the plurality of sample data groups. The third sample data group is at least one sample data group in the sorted plurality of sample data groups that precedes the fourth sample data group.

4. The method according to claim 1, characterized in that, The first data group to be detected is at least one data group to be detected; the step of processing the first data group to be detected among the plurality of data groups to be detected through a preset data prediction model to obtain the predicted data group corresponding to the second data group to be detected among the plurality of data groups to be detected includes: From the at least one group of data to be detected, at least one data point corresponding to each preset time point within the preset time period is determined sequentially; Using the preset data prediction model, at least one data point to be detected corresponding to each preset time point is used to predict one predicted data point corresponding to each preset time point in the second data group to be detected, thereby obtaining the predicted data group corresponding to the second data group to be detected.

5. The method according to claim 1, characterized in that, The step of sequentially comparing each of the plurality of data groups to be detected with a corresponding predicted data group to obtain a comparison result, and determining abnormal data from the plurality of data groups to be detected based on the comparison result, includes: From the plurality of data to be detected, find the plurality of data to be detected corresponding to each preset time point within the preset time period, and combine the plurality of data to be detected into a first combination of data; From the multiple predicted data groups corresponding to the multiple data groups to be detected, find multiple predicted data corresponding to each preset time point within the preset time period, and combine the multiple predicted data into a second combined data; If the difference between the first combined data and the corresponding second combined data is greater than a first preset difference threshold, then the plurality of data to be detected corresponding to the first combined data are determined as the abnormal data.

6. The method according to claim 1, characterized in that, The step of sequentially comparing each of the plurality of data groups to be detected with a corresponding predicted data group to obtain a comparison result, and determining abnormal data from the plurality of data groups to be detected based on the comparison result, includes: Sequentially, from each group of data to be detected and its corresponding group of predicted data, find one data to be detected and one predicted data corresponding to each preset time point within the preset time period; If the difference between the first data to be detected and the corresponding first predicted data is greater than a second preset difference threshold, then the first data to be detected is determined to be the abnormal data.

7. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire multiple groups of data to be detected generated by multiple data types within a preset time period; the type correlation between the multiple data types is greater than a preset correlation threshold. The data processing unit is used to process the first data group to be detected in the plurality of data groups to be detected through a preset data prediction model to obtain the predicted data group corresponding to the second data group to be detected in the plurality of data groups to be detected; the first data group to be detected and the second data group to be detected are data groups to be detected corresponding to different data types in the plurality of data groups to be detected; wherein, the data type includes at least the temperature of the central processing unit, the speed of the central processing unit, and the read and write speed of the hard disk; The comparison unit is used to sequentially compare each of the plurality of data groups to be detected with a corresponding prediction data group to obtain a comparison result; A determining unit is used to determine abnormal data from the plurality of data groups to be detected based on the comparison results.

8. The apparatus according to claim 7, characterized in that, The device further includes: a model training unit; The acquisition unit is also used to acquire multiple sample data groups within the sampling time period; each sample data group in the multiple sample data groups corresponds to a preset data type, and the sample data in each sample data group corresponds to a set data label, the data label including normal data label and abnormal data label; The data processing unit is further configured to process the first sample data group among the plurality of sample data groups using an initial data prediction model to obtain a predicted sample data group corresponding to the second sample data group among the plurality of sample data groups; the first sample data group and the second sample data group are sample data groups corresponding to different preset data types among the plurality of sample data groups. The comparison unit is also used to compare the plurality of sample data groups and the plurality of predicted sample data groups corresponding to the plurality of sample data groups respectively, and to obtain sample comparison results; The model training unit is used to train the initial data prediction model using the sample comparison results and the normal data labels and / or abnormal data labels corresponding to the sample data in each sample data group, so as to obtain the preset data prediction model.

9. A data processing device, characterized in that, The device includes: a processor, a memory, and a communication bus; when the processor executes the running program stored in the memory, it implements the method as described in any one of claims 1-6.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images