Data processing method and device, electronic equipment and storage medium
By determining the data set prediction target and using the data prediction model to process historical received processing data, the problem of inefficient maintenance of multiple timing data sets is solved, and efficient prediction and maintenance is achieved.
Patent Information
- Application Number
- CN202411045491.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-12
AI Technical Summary
When multiple time series data sets are not unique, the data calibers are not exactly the same, which increases maintenance difficulty and complexity and affects maintenance efficiency.
By determining the data set prediction target, obtaining the historical received processing data of the time series data set to be predicted, and making predictions based on the appropriate data prediction model, ensuring prediction accuracy and efficiency.
It improves the maintenance efficiency of predicted time series data sets, ensures the standardization and prediction accuracy of historical received processing data, and reduces maintenance complexity.
Smart Images

Figure CN118939973A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art
[0002] In the related art, the data in the time series data set is arranged in chronological order. Such time series data sets are often used to process tasks with a chronological order, such as task allocation in a workflow system, processing of network requests, and the like.
[0003] When maintaining multiple time series data sets, it is impossible to perform unified maintenance because the sources of each time series data set are not unique, resulting in low maintenance efficiency. Summary of the invention
[0004] In view of this, the present disclosure provides a data processing method, device, electronic device and storage medium to solve the problem of low maintenance efficiency for multiple time series data sets.
[0005] In a first aspect, the present disclosure provides a data processing method, the method comprising:
[0006] Determine the prediction target of the data set according to the data processing requirements of the current task. The prediction target of the data set includes the target data granularity, target time granularity, and target observation duration.
[0007] Obtain historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted;
[0008] Based on the first data distribution of the historical received processing volume data at the target time granularity, the target time distribution length corresponding to the historical received processing volume data is determined, and the amount of data to be processed of the predicted time series data set at the target observation time length is predicted through the data prediction model corresponding to the target time distribution length to obtain the target prediction result, which is used to process the current task.
[0009] In a second aspect, the present disclosure provides a data processing device, the device comprising:
[0010] The first processing module is used to determine the data set prediction target according to the data processing requirements of the current task, and the data set prediction target includes the target data granularity, the target time granularity and the target observation duration;
[0011] An acquisition module is used to acquire historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted;
[0012] The second processing module is used to determine the target time distribution length corresponding to the historical received processing volume data based on the first data distribution of the historical received processing volume data at the target time granularity, and predict the amount of data to be processed of the predicted time series data set at the target observation time length through the data prediction model corresponding to the target time distribution length to obtain the target prediction result, which is used to process the current task.
[0013] In a third aspect, the present disclosure provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0014] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof.
[0015] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof.
[0016] The data processing method provided in this embodiment determines the data set prediction target according to the data processing requirements of the current task, and then obtains the historical received processing volume data of the time series data set to be predicted according to the data set prediction target, which can ensure the standardization of the historical received processing volume data, and then subsequently use a suitable data prediction model to perform prediction based on the target time distribution duration corresponding to the historical received processing volume data, which can improve the prediction efficiency while ensuring the prediction accuracy, thereby helping to improve the efficiency of maintaining the time series data set to be predicted. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 is a flowchart of a data processing method according to some embodiments of the present disclosure;
[0019] Figure 2 is a flowchart of another data processing method according to some embodiments of the present disclosure;
[0020] Figure 3 is a schematic diagram of a model structure of a data prediction model according to some embodiments of the present disclosure;
[0021] Figure 4 is a flowchart of another data processing method according to some embodiments of the present disclosure;
[0022] Figure 5 is a flowchart of yet another data processing method according to some embodiments of the present disclosure;
[0023] Figure 6 is a flowchart of a data processing method according to some other embodiments of the present disclosure;
[0024] Figure 7 is a schematic diagram of a process for determining a target shift according to some embodiments of the present disclosure;
[0025] Figure 8 is a structural block diagram of a data processing device according to an embodiment of the present disclosure;
[0026] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0028] In the related technology, when the sources of multiple time series data sets are not unique, the data calibers corresponding to the multiple time series data sets will not be exactly the same, and thus there will be differences in the meaning, definition, and calculation method of the corresponding data. If it is necessary to perform unified maintenance and processing on multiple time series data sets at this time, it will increase the difficulty and complexity of maintenance, thereby affecting maintenance efficiency.
[0029] In view of this, the embodiments of the present disclosure provide a data processing method, which can determine the data set prediction target according to the data processing requirements of the current task, and perform unified and standardized processing on the historical received processing volume data of the time series data set to be predicted. Furthermore, in the prediction process, when a suitable data prediction model is used for processing, the prediction efficiency can be improved while ensuring the prediction accuracy, thereby effectively improving the efficiency of maintaining the time series data set to be predicted.
[0030] According to an embodiment of the present disclosure, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0031] In this embodiment, a data processing method is provided, which can be used in the above-mentioned electronic device, such as a tablet computer, etc. Figure 1 is a flow chart of a data processing method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:
[0032] Step S101, determining the data set prediction target according to the data processing requirements of the current task.
[0033] The current task refers to the task that currently needs to perform batch prediction on the data of the time series data set to be predicted. The specific task can be determined according to actual needs. In order to make the subsequent prediction process more targeted and effectively improve the prediction efficiency, the data set prediction target is determined according to the data processing requirements of the current task, so as to use the data set prediction target for targeted prediction. Among them, the data set prediction target includes the target data granularity, target time granularity and target observation duration.
[0034] The target data granularity is used to clarify the level of detail or refinement of the data to be predicted, so that trends and changes in the data can be captured during the prediction process, and the subsequent target prediction results can better meet the data processing requirements of the current task. For example, in the e-commerce scenario, the target data granularity can be divided based on consumption areas, products, brands and other dimensions.
[0035] The target time granularity is used to specify the granularity for time division during the forecasting process, for example, forecasting in hours, days, weeks, or months.
[0036] The target observation duration refers to the length of time used for prediction, which is convenient for standardizing the time range for prediction. In some examples, the target observation duration may be greater than or equal to the duration required for prediction in the data processing requirements. Preferably, the target observation duration is greater than the duration required for prediction in the data processing requirements, which helps to reduce interference and improve the accuracy of data prediction. For example: If the data processing requirement of the current task is to predict the amount of data of the time series data set to be predicted in the next week (7 days), the target observation duration is determined to be 9 days. Alternatively, if the data processing requirement of the current task is to predict the amount of data of the time series data set to be predicted in the next month (30 days), the target observation duration is determined to be 40 days.
[0037] Step S102, obtaining historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted.
[0038] The time series data set to be predicted refers to the sequence that needs to be predicted. These data are usually arranged in chronological order. Since the source of the time series data set to be predicted is not unique, the data granularity of the corresponding historical received processing volume data may differ from the target data granularity during the actual prediction process. For example, if the current task is a live broadcast task, the historical received processing volume data refers to the number of audits of live broadcast slices. If the current task is a picture and text maintenance task, the historical received processing volume data refers to the number of pictures.
[0039] Therefore, in order to improve the prediction efficiency and unify the unified standard of data granularity, the historical received processing volume data of the time series to be predicted is obtained according to the target data granularity to ensure data consistency, which is convenient for subsequent analysis and helps to improve the processing efficiency of the data prediction model. In some examples, if the actual data granularity of the historical received processing volume data of the time series data set to be predicted is inconsistent with the target data granularity, the historical received processing volume data at the actual data granularity is adjusted in a targeted manner according to the target data granularity, thereby obtaining the historical received processing volume data corresponding to the target data granularity.
[0040] Step S103, based on the first data distribution of the historical received processing volume data at the target time granularity, determine the target time distribution length corresponding to the historical received processing volume data, and predict the amount of data to be processed of the predicted time series data set at the target observation time length through the data prediction model corresponding to the target time distribution length to obtain the target prediction result.
[0041] According to the first data distribution of the historical received processing volume data at the target time granularity, the distribution law of the received processing volume data at the target time granularity can be clarified, and then the relationship between the change trend of the historical received processing volume data and time can be understood.
[0042] The target time distribution length refers to the time distribution length covered by the historical received processing volume data. For example, if the historical received processing volume data is data between January 1 and January 31, the target time distribution length is 31 days.
[0043] In order to ensure the accuracy and reliability of the prediction, a data prediction model that matches the target time distribution length is used to predict the amount of data to be processed in the time series data set to be predicted within the target observation time length, so as to obtain the target prediction result that can handle the current task, which helps to ensure the reliability and scientificity of the prediction.
[0044] The data processing method provided in this embodiment determines the data set prediction target according to the data processing requirements of the current task, and then obtains the historical received processing volume data of the time series data set to be predicted according to the data set prediction target, which can ensure the standardization of the historical received processing volume data, and then subsequently use a suitable data prediction model to perform prediction based on the target time distribution duration corresponding to the historical received processing volume data, which can improve the prediction efficiency while ensuring the prediction accuracy, thereby helping to improve the efficiency of maintaining the time series data set to be predicted.
[0045] In this embodiment, a data processing method is provided, which can be used in the above-mentioned electronic devices, such as mobile phones, tablet computers, etc. Figure 2 is a flow chart of a data processing method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:
[0046] Step S201, determining the data set prediction target according to the data processing requirements of the current task.
[0047] Step S202, obtaining historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted.
[0048] Specifically, the above step S202 includes:
[0049] Step S2021, obtaining the initial historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted.
[0050] The initial historical received processing volume data refers to the original data in the time series data set to be predicted that corresponds to the target data granularity and has not been processed.
[0051] Step S2022: pre-process the initial historical receiving processing volume data to obtain a first pre-processing result of the initial historical receiving processing volume data.
[0052] In order to ensure the quality of the data, the initial historical received processing volume data is preprocessed to improve the availability of the initial historical received processing volume data, thereby obtaining a first preprocessing result of the initial historical received processing volume data. The preprocessing operations include but are not limited to data cleaning, conversion, formatting, etc. For example: logarithmic transformation processing, noise removal, missing value filling, standardization, etc.
[0053] In some optional implementations, the above step S2022 includes:
[0054] Step a1, performing outlier detection on the initial historical receiving processing volume data to obtain the detection result;
[0055] Step a2: If the detection result indicates that the initial historical received processing volume data has an abnormal value, the position of the abnormal value in the initial historical received processing volume data is determined, and a specified value is added to the position to eliminate the abnormal value, thereby obtaining a first preprocessing result.
[0056] Among them, an outlier refers to an observation value that deviates significantly from other data points. An outlier may be caused by data entry errors, measurement errors, or other abnormal conditions. In order to reduce the interference of abnormal data, an outlier detection is performed on the initial historical received processing volume data to discover possible abnormal conditions in the data, thereby ensuring the data quality and reliability of the initial historical received processing volume data.
[0057] If the test result indicates that there are outliers in the initial historical received processing volume data, it is necessary to determine the specific location of the outlier in the data, and then add the specified value to the location of the outlier to replace the original outlier and eliminate the impact of the outlier on the data, thereby ensuring the integrity and continuity of the initial historical received processing volume data, avoiding the loss of important information, and helping to provide reliable data support for subsequent predictions. For example, the specified value can be 0.
[0058] Step S2023: determining the time characteristics of the initial historical received processing volume data based on the historical processing time corresponding to the initial historical received processing volume data.
[0059] The historical processing time refers to the specific time point or time period when the initial historical received processing volume data is processed. In order to determine the relationship between the received processing volume data and time of the time series data set to be predicted, feature engineering processing is performed based on the historical processing time of each data in the initial historical received processing volume data, thereby obtaining the time characteristics of the initial historical received processing volume data.
[0060] For example, according to the historical processing time corresponding to the initial historical receiving and processing volume data, the historical processing time is classified, and then the initial historical receiving and processing volume data under various time types are determined, and then the time characteristics that can characterize the trend changes of the initial historical receiving and processing volume data under different time types are obtained, so as to have a deeper understanding of the relationship between the receiving and processing volume data and time of the time series data set to be predicted, and provide strong support for subsequent analysis and prediction. Among them, the time types include but are not limited to: holidays, designated activity days, working days, morning / midday / late / evening, etc.
[0061] Step S2024: Obtain historical receiving processing volume data based on the first preprocessing result and the time characteristics.
[0062] Combining the first preprocessing result with the time characteristics can fully tap the potential value of the data, thereby obtaining historical receiving and processing volume data that can provide a more comprehensive understanding of the time series data set to be predicted, thereby providing favorable support for subsequent analysis and decision-making.
[0063] In some optional implementations, if the current task is a target task, the above step S2024 includes:
[0064] Step b1, obtaining task feature data corresponding to the target task;
[0065] Step b2, preprocessing the task feature data to obtain a second preprocessing result;
[0066] Step b3, integrating the first preprocessing result, the time feature and the second preprocessing result to obtain historical receiving processing volume data.
[0067] Specifically, the target task refers to a pre-specified task that needs to be processed in a targeted manner in combination with the task characteristics. In order to determine the relationship between the received processing volume data of the time series data set to be predicted and the target task, the task feature data corresponding to the target task is also obtained before predicting the processing data to be predicted, so as to characterize the task content of the target task through the task feature data. Among them, the task feature data may include but is not limited to the domain feature data and upstream data corresponding to the target task. For example: taking an e-commerce task as an example, the domain feature data may be the amount of advertising consumption, etc. Upstream data may include diversion strategies, human review ratios, etc.
[0068] In order to ensure the quality of the task feature data, the task feature data is preprocessed to improve the usability of the task feature data, thereby obtaining a second preprocessing result. The preprocessing includes but is not limited to any one or more of the following operations: mean or variance processing.
[0069] By integrating the first preprocessing result, time characteristics and the second preprocessing result, the historical received processing volume data can be made more reliable, and the review status of the historical received processing volume data of the time series data set to be predicted can be jointly characterized by combining dimensions such as time and task type, thereby helping to make subsequent predictions using the historical received processing volume data more scientific and reliable.
[0070] Step S203, based on the first data distribution of the historical received processing volume data at the target time granularity, determine the target time distribution length corresponding to the historical received processing volume data, and predict the amount of data to be processed of the predicted time series data set at the target observation time length through the data prediction model corresponding to the target time distribution length to obtain the target prediction result.
[0071] The data processing method provided in this embodiment can preprocess and feature engineer the initial historical received processing volume data of the time series data set to be predicted, so that the obtained historical received processing volume data can more comprehensively characterize the relationship between the received processing volume data and time, and then when the historical received processing volume data is used for prediction, the accuracy and effect of the prediction can be better improved, thereby making the obtained target prediction result more reliable and more conducive to processing the current task.
[0072] In some optional implementations, based on the first data distribution of the historical received processing volume data at the target time granularity, the process of determining the time distribution length corresponding to the historical received processing volume data includes the following steps:
[0073] Step c1, determining the mapping relationship between the target time granularity and the time step;
[0074] Step c2, summarizing the first data distribution of the historical received processing volume data at the target time granularity according to the mapping relationship, and determining the second data distribution of the historical received processing volume data at multiple time steps;
[0075] Step c3, determining the time distribution length corresponding to the historical received processing volume data according to the second data distribution under multiple time steps.
[0076] Specifically, in order to facilitate the determination of the time distribution length involved in the historical received processing volume data, the mapping relationship between the target time granularity and the time step is determined to facilitate targeted aggregation. For example, if one day corresponds to one time step, and the unit of the target time granularity is hour, the mapping relationship between the target time granularity and the time step is the mapping relationship between hours and days.
[0077] In order to discover the potential rules and trends of the historical received processing volume data over time, the first data distribution of the historical received processing volume data at the target time granularity is summarized according to the mapping relationship, and then the second data distribution of the historical received processing volume data at each time step is obtained. For example: determine the data distribution of the historical received processing volume data on each day.
[0078] By determining the time distribution length corresponding to the historical received processing volume data based on the second data distribution under multiple time steps, the time characteristics of the historical received processing volume data can be more clearly understood. In subsequent predictions, the data information of the historical received processing volume data can be fully utilized to improve data utilization, thereby helping to select a suitable data prediction model for targeted prediction and improve the accuracy and efficiency of the prediction.
[0079] In some optional examples, the above step c3 includes:
[0080] Step c31, performing a null value rate detection on the second data distribution at the current time step to obtain a null value rate detection result;
[0081] Step c32: If the null value rate represented by the null value rate detection result is less than or equal to the specified null value rate threshold, the second data distribution under the current time step is determined to be a valid distribution, and combined with the time steps corresponding to other valid distributions, the time distribution length corresponding to the historical received processing volume data is obtained.
[0082] Specifically, in order to determine the data integrity of the second data distribution at the current time step, a null value rate test is performed on the second data distribution to clarify the data quality, and then obtain a null value rate test result.
[0083] If the detected void rate is less than or equal to the specified void rate threshold, the second data distribution under the current time step is considered to be valid, and the current time step is combined with the time steps of other valid distributions to more accurately determine the time distribution length of the historical received processing volume data, thereby providing more valuable information for subsequent analysis and decision-making.
[0084] In some optional implementations, the process of determining the data prediction model includes:
[0085] Step d1, matching the time distribution length with the time distribution intervals corresponding to multiple candidate data prediction models respectively to obtain matching results;
[0086] Step d2: If the matching result indicates that the current time distribution interval contains the entire or part of the time distribution length, the candidate data prediction model corresponding to the current time distribution interval is used as the data prediction model.
[0087] Specifically, multiple candidate data prediction models are preconfigured, and each candidate data prediction model has its corresponding time distribution interval. In order to ensure the reliability of the target prediction result, the determined time distribution length is matched with each time distribution interval respectively to determine the time distribution interval that can contain the time distribution length or part of the time distribution length, and then the corresponding candidate data prediction model is used as the data prediction model for prediction, so that the quality of data analysis and decision-making can be improved in subsequent analysis. Among them, the number of data prediction models is at least 1.
[0088] In some optional implementation scenarios, the time distribution intervals corresponding to the multiple candidate data prediction models are: 0-90 days, 91-180 days and 181-365 days. If the time distribution length is 75 days, the candidate data prediction model corresponding to 0-90 days is selected as the data prediction model for prediction. If the time distribution length is 180 days, the candidate data prediction model corresponding to 0-90 days and the candidate data prediction model corresponding to 91-180 days are selected as the data prediction models for prediction.
[0089] In some optional implementations, when there are multiple data prediction models, the process of predicting the amount of data to be processed of the time series data set to be predicted under the target observation duration by the data prediction model to obtain the target prediction result may be as follows:
[0090] Step e1, determining input data corresponding to the current data prediction model from the second data distribution under multiple time steps according to the time distribution interval of the current data prediction model;
[0091] Step e2, based on the input data, predicting the amount of data to be processed for the time series data set to be predicted under the target observation time length, and obtaining the first prediction result of the current data prediction model;
[0092] Step e3, fusing the first prediction result of the current data prediction model with the first prediction results of other data prediction models to obtain a target prediction result.
[0093] Specifically, to ensure that the prediction can be carried out in an orderly manner, the input data corresponding to the current data prediction model is determined from the second data distribution under multiple time steps according to the time distribution interval of the current data prediction model to avoid interference from invalid data.
[0094] The determined input data is input into the current data prediction model, and a targeted prediction is performed using the change trend of the input data, thereby obtaining a first prediction result output by the current data prediction model.
[0095] Since there is a one-to-one correspondence between the data prediction model and the time distribution interval, in order to reduce the prediction error, the first prediction result of the current data prediction model is fused with the first prediction results of other data prediction models to improve the prediction coverage, thereby making the obtained target prediction result more reliable and accurate. The fusion method includes but is not limited to summation, weighted average, etc.
[0096] In some optional examples, when the actual quantity dimension of the input data is smaller than the data dimension of the current data prediction model, in order to ensure that the reliability of the current data prediction model is not affected by insufficient data, the input data is padded according to the data dimension of the current data prediction model to ensure that the dimension of the input data is complete, which is conducive to the current data prediction model to perform effective prediction.
[0097] In some optional implementations, if the void rate obtained by characterizing the void rate detection result is greater than the specified void rate threshold, the second data distribution characterizing the current time step cannot well reflect the distribution of the received processing volume data, and is likely to affect the accuracy of data analysis and decision-making. Therefore, in order to ensure the integrity of the historical received processing volume data, the second data distribution at the current time step is used as a supplementary distribution, and it is integrated with the first prediction results of multiple data prediction models to use the supplementary distribution to provide the distribution of the received processing volume data at the current time step, so that the target prediction result can be more comprehensive and representative, more in line with the actual situation, thereby helping to improve the robustness of the data prediction model.
[0098] In some optional implementation scenarios, the time distribution intervals corresponding to the multiple candidate data prediction models are in a gradient-growing and non-intersecting relationship, and the model structure of each candidate data prediction model is the same.
[0099] By setting candidate data prediction models for different time distribution intervals, it is possible to better adapt to prediction needs at different time scales, so that each candidate data prediction model can focus on data analysis and prediction within the corresponding time distribution interval, thereby capturing the changing trends and characteristics of each time series to be predicted at different stages, thereby providing more accurate predictions. In addition, by setting multiple candidate data prediction models in the above manner, the time distribution interval corresponding to each candidate data prediction model can be adjusted according to demand, thereby making the prediction method more flexible and more scalable, and better performing targeted selection to meet the data processing requirements of multiple tasks, thereby better improving the efficiency and accuracy of the prediction.
[0100] In some optional implementations, for each data prediction model, the process of performing prediction using the data prediction model may be as follows:
[0101] Step f1, identifying stable data and unstable data in the input data;
[0102] Step f2, performing linear fitting processing on the stable data to obtain a first fitting result, and using the first fitting result, predicting the amount of data to be processed for the time series data set to be predicted under the target observation time length to obtain a first prediction result;
[0103] Step f3, performing fitting deviation processing on the unstable data to obtain a second fitting result, and using the second fitting result, predicting the amount of data to be processed for the time series data set to be predicted under the target observation time length to obtain a second prediction result;
[0104] Step f4, integrating the first prediction result and the second prediction result to obtain the first prediction result.
[0105] Specifically, in order to reduce the prediction error, the data characteristics of the input data are used to identify the stable data and unstable data in the input data. The expression of the input data can be as follows: y(t) = g(t) + r(t) + δ, where g(t) represents stable data, r(t) represents unstable data, and δ is a constant. Among them, stable data refers to a numerical sequence with a relatively stable trend or regularity in the input data, which can be modeled and predicted by methods such as linear fitting. Unstable data refers to a numerical sequence in the input data that does not have a regularity in its changes.
[0106] Since the change trend of stable data is relatively stable, a linear fitting process is performed on it to obtain a first fitting result, and then the first fitting result is used to predict the amount of data to be processed for the predicted time series data set under the target observation time length to obtain a first prediction result corresponding to the stable data.
[0107] In order to facilitate the mining of the changing characteristics of non-stable data, fitting deviation processing is performed on it to convert it into a predictable form to obtain a second fitting result. The second fitting result is then used to predict the amount of data to be processed in the target observation time of the time series data set to be predicted, and a second prediction result corresponding to the stable data is obtained.
[0108] The first prediction result and the second prediction result are integrated to synthesize the prediction results of each part of the data, thereby obtaining the first prediction result of the current data prediction model. For example, the integration process can be an equalization or weighted average processing process.
[0109] Through the above prediction method, different characteristics of input data can be used to process and predict stable data and unstable data respectively, so that the first prediction result obtained is finally more accurate.
[0110] In some optional implementations, the model structure of the data prediction model is as follows: Figure 3As shown. The data prediction model includes a linear fitting prediction model for predicting stable data, and two deviation fitting prediction models for predicting unstable data. Preferably, the deviation fitting prediction model can be a tree structure deviation fitting model, which can effectively capture the nonlinear relationship in the unstable data and improve the accuracy of the prediction, and the tree structure deviation fitting model can automatically evaluate the importance of the features during operation, thereby effectively improving the prediction efficiency. Furthermore, in order to ensure the reliability of the second fitting result, two different tree structure deviation fitting models are selected for prediction, which can effectively eliminate the instability of the prediction for the unstable data and reduce the error.
[0111] In some optional implementation scenarios, during the process of training the data prediction model, in order to make the prediction results of each deviation fitting prediction model more stable, the loss function in the training model adopts the mape_ln function (a loss function used to smooth the influence of different magnitudes), so that the influence of different magnitudes can be smoothed out during the training process.
[0112] Performing fitting deviation processing on the unstable data to obtain a second fitting result includes: performing fitting deviation processing on the unstable data through a built-in first tree model to obtain a first deviation fitting result; performing fitting deviation processing on the unstable data through a built-in second tree model to obtain a second deviation fitting result; performing mean processing on the first deviation fitting result and the second deviation fitting result to obtain a second fitting result. That is, the second fitting result = (first deviation fitting result + second prediction result) / 2.
[0113] As one or more specific application embodiments of the embodiments of the present disclosure, if the current task is not the target task, the process of predicting the time series data set to be predicted can be Figure 4 shown.
[0114] Determine the dataset prediction target based on the data processing requirements of the current task. The dataset prediction target includes the target data granularity, target time granularity, and target observation duration;
[0115] Obtain the initial historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted.
[0116] An abnormal value detection process is performed on the initial historical received processing volume data to add a specified value to a position of the abnormal value in the initial historical received processing volume data to obtain a first preprocessing result.
[0117] Based on the historical processing time corresponding to the initial historical receiving and processing volume data, the time characteristics of the initial historical receiving and processing volume data under different time types are determined, wherein the time types include but are not limited to: holidays, designated activity days, working days, morning / noon / late / evening, etc.
[0118] The first preprocessing result is combined with the time feature to obtain the historical receiving processing volume data to be processed.
[0119] Determine the first data distribution of the historical received processing volume data at the target time granularity. According to the mapping relationship between the target time granularity and the time step, summarize the first data distribution of the historical received processing volume data at the target time granularity, determine the second data distribution of the historical received processing volume data at multiple time steps, and then determine the time distribution length corresponding to the historical received processing volume data according to the second data distribution at multiple time steps.
[0120] Perform a null value rate test on the second data distribution at each time step to obtain the null value rate test result. Take the second data distribution at the time step where the null value rate is less than or equal to the specified null value rate threshold as the valid distribution, and combine the time steps corresponding to other valid distributions to obtain the time distribution length corresponding to the historical received processing volume data. Then determine the data prediction model corresponding to the target time distribution length from the preset multiple candidate data prediction models, and predict the amount of data to be processed for the time series data set to be predicted under the target observation time length to obtain the intermediate fusion result. Among them, the data granularity of the intermediate fusion result is the target data granularity.
[0121] The second data distribution at the time step where the null value rate is greater than the specified null value rate threshold is used as the supplementary distribution. The supplementary distribution is combined with the prior knowledge corresponding to the current task and the intermediate fusion result to obtain the target prediction result. The data granularity of the target prediction result is the target data granularity.
[0122] Display target prediction results according to target data granularity.
[0123] As another one or more specific application embodiments of the present disclosure, if the current task is a target task, then Figure 5 The process shown in the figure further obtains the task feature data corresponding to the target task, and preprocesses the task feature data to obtain a second preprocessing result. The preprocessing process includes: mean and variance processing. The task feature data can be the domain feature data and upstream data corresponding to the target task. For example, taking an e-commerce task as an example, the domain feature data can be the amount of advertising consumption, etc. Upstream data can include diversion strategies, human review ratios, etc.
[0124] The first preprocessing result, the time characteristics and the second preprocessing result are integrated to obtain the historical receiving processing volume data, and then the following is obtained: Figure 5The process shown in FIG. 1 is as follows. The process of using the data prediction model to predict the amount of data to be processed of the time series data set to be predicted under the target observation time is similar to the above prediction process and will not be described in detail here.
[0125] In some optional implementations, after the target prediction result is obtained, it is displayed so that the target prediction result can be intuitively displayed to the data processing personnel, so that the data processing personnel can quickly understand the future trend of the time series data set to be predicted, and then perform targeted maintenance in a timely manner. For example, it can be displayed in the form of a chart. The types of icons include but are not limited to line charts, bar charts, pie charts, scatter plots, box plots, radar charts, etc.
[0126] In this embodiment, a data processing method is provided, which can be used in the above-mentioned electronic devices, such as mobile phones, tablet computers, etc. Figure 6 is a flow chart of a data processing method according to an embodiment of the present disclosure, such as Figure 6 As shown, the process includes the following steps:
[0127] Step S601, determining the data set prediction target according to the data processing requirements of the current task.
[0128] Step S602, obtaining historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted.
[0129] Step S603, based on the first data distribution of the historical received processing volume data at the target time granularity, determine the target time distribution length corresponding to the historical received processing volume data, and predict the amount of data to be processed of the predicted time series data set at the target observation time length through the data prediction model corresponding to the target time distribution length to obtain the target prediction result.
[0130] Step S604, displaying the target prediction result.
[0131] Step S605, sending the target prediction result to the scheduling system to determine the target shift group for maintaining the time series data set to be predicted.
[0132] The scheduling system is a system used to maintain and arrange data processing personnel to perform targeted maintenance on the predicted time series data set. In order to improve maintenance efficiency, the target prediction results are sent to the scheduling system so that the scheduling system can determine the maintenance workload required for each predicted time series data set based on the target prediction results and allocate it to the appropriate target shift, thereby ensuring that resources can be reasonably utilized and avoiding over-allocation or under-allocation, which can effectively reduce maintenance time and cost and help avoid maintenance conflicts.
[0133] The data processing method provided in this embodiment performs targeted prediction on the time series data set to be predicted according to the data processing requirements, and sends the target prediction results to the scheduling system, which helps to improve the efficiency and effect of maintaining the time series data set to be predicted. In addition, in the prediction process, through the analysis of historical received processing volume data and the application of data prediction models, the amount to be reviewed can be estimated more accurately, thereby providing valuable reference information for the scheduling system.
[0134] In some optional implementations, the target team can also be adjusted based on the target prediction results and prior knowledge to make resource allocation more reasonable, which will help improve the work efficiency and quality of data processing personnel, and can also better respond to emergencies and ensure the smooth progress of work.
[0135] In some optional implementation scenarios, based on the target prediction results of the time series data set to be predicted, the process of determining the target shift can be as follows: Figure 7 shown.
[0136] The time series data set to be predicted is obtained through a data processing system. The data processing system can clarify the data set information of each time series data set, and then identify whether the time series data set needs to be predicted for the receiving processing volume, thereby clarifying the time series data set to be predicted. Among them, the data set information in the data processing system can be sent through the business end of a designated organization / team, and which time series data sets need to be targeted for prediction are marked in advance before sending.
[0137] After clarifying the time series data set to be predicted, the prediction target of the data set is determined according to the data processing requirements of the current task. The data set prediction target includes the target data granularity, target time granularity and target observation duration. The historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted is obtained through the data maintenance platform.
[0138] Based on the first data distribution of the historical received processing volume data at the target time granularity, the target time distribution length corresponding to the historical received processing volume data is determined, and the amount of data to be processed of the predicted time series data set at the target observation time length is predicted through the data prediction model corresponding to the target time distribution length to obtain the target prediction result.
[0139] The target prediction results are input into the display module (eg, a dashboard) and the scheduling system. In the scheduling system, scheduling is performed based on the received target prediction results, thereby obtaining a target shift group for maintaining the time series data set to be predicted.
[0140] In some examples, the business end can also adjust the target team based on the target prediction results displayed by the display module and prior knowledge to make resource allocation more reasonable.
[0141] In the present embodiment, a data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0142] This embodiment provides a data processing device, such as Figure 8 As shown, including:
[0143] The first processing module 801 is used to determine the data set prediction target according to the data processing requirements of the current task, and the data set prediction target includes the target data granularity, the target time granularity and the target observation duration;
[0144] An acquisition module 802 is used to acquire historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted;
[0145] The second processing module 803 is used to determine the target time distribution length corresponding to the historical received processing volume data based on the first data distribution of the historical received processing volume data at the target time granularity, and predict the amount of data to be processed of the predicted time series data set at the target observation time length through the data prediction model corresponding to the target time distribution length to obtain the target prediction result, which is used to process the current task.
[0146] In some optional implementations, the acquisition module 802 includes:
[0147] A first acquisition unit is used to acquire initial historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted;
[0148] A first processing unit, configured to preprocess the initial historical received processing volume data to obtain a first preprocessing result of the initial historical received processing volume data;
[0149] A second processing unit, configured to determine a time feature of the initial historical received processing volume data based on a historical processing time corresponding to the initial historical received processing volume data;
[0150] The third processing unit is used to obtain historical receiving processing volume data based on the first preprocessing result and the time feature.
[0151] In some optional implementations, the first processing unit includes:
[0152] A first detection unit is used to perform abnormal value detection on the initial historical receiving processing volume data to obtain a detection result;
[0153] The first execution unit is used to determine the position of the abnormal value in the initial historical received processing volume data if the detection result indicates that the initial historical received processing volume data has an abnormal value, and add a specified value to the position to eliminate the abnormal value to obtain a first preprocessing result.
[0154] In some optional implementations, if the current task is a target task, the third processing unit includes:
[0155] A second acquisition unit is used to acquire task feature data corresponding to the target task;
[0156] A first data processing unit is used to preprocess the task characteristic data to obtain a second preprocessing result;
[0157] The first integration unit is used to integrate the first preprocessing result, the time feature and the second preprocessing result to obtain historical receiving processing volume data.
[0158] In some optional implementations, the second processing module 803 includes:
[0159] A fourth processing unit, used to determine a mapping relationship between a target time granularity and a time step;
[0160] A second integration unit is used to summarize the first data distribution of the historical received processing volume data at the target time granularity according to the mapping relationship, and determine the second data distribution of the historical received processing volume data at multiple time steps;
[0161] The fifth processing unit is used to determine the time distribution length corresponding to the historical received processing volume data according to the second data distribution under multiple time steps.
[0162] In some optional implementations, the fifth processing unit includes:
[0163] A second detection unit is used to perform a null value rate detection on the second data distribution at the current time step to obtain a null value rate detection result;
[0164] The second execution unit is used to determine that the second data distribution under the current time step is a valid distribution if the null value rate represented by the null value rate detection result is less than or equal to the specified null value rate threshold, and to obtain the time distribution length corresponding to the historical received processing volume data in combination with the time steps corresponding to other valid distributions.
[0165] In some optional implementations, the data prediction model determination device includes:
[0166] A matching module, used to match the time distribution length with the time distribution intervals corresponding to multiple candidate data prediction models respectively to obtain a matching result;
[0167] The third processing module is used to use the candidate data prediction model corresponding to the current time distribution interval as the data prediction model if the matching result indicates that the current time distribution interval contains the entire or partial time distribution length, and the number of data prediction models is at least 1.
[0168] In some optional implementations, if there are multiple data prediction models, the second processing module includes:
[0169] A second data processing unit, configured to determine input data corresponding to the current data prediction model from the second data distribution at multiple time steps according to the time distribution interval of the current data prediction model;
[0170] A prediction unit, configured to predict the amount of data to be processed of the time series data set to be predicted under the target observation time length based on the input data, and obtain a first prediction result of the current data prediction model;
[0171] The fusion unit is used to fuse the first prediction result of the current data prediction model with the first prediction results of other data prediction models to obtain a target prediction result.
[0172] In some optional implementations, the prediction unit includes:
[0173] A recognition unit, used for recognizing stable data and unstable data in the input data;
[0174] A sixth processing unit is used to perform linear fitting processing on the stable data to obtain a first fitting result, and predict the amount of data to be processed of the time series data set to be predicted under the target observation time length according to the first fitting result to obtain a first prediction result;
[0175] The seventh processing unit is used to perform fitting deviation processing on the unstable data to obtain a second fitting result, and predict the amount of data to be processed of the time series data set to be predicted under the target observation time length according to the second fitting result to obtain a second prediction result;
[0176] The third integration unit is used to integrate the first prediction result and the second prediction result to obtain the first prediction result.
[0177] In some optional implementations, the seventh processing unit includes:
[0178] A third execution unit is used to perform fitting deviation processing on the unstable data through the built-in first tree model to obtain a first deviation fitting result;
[0179] A fourth execution unit is used to perform fitting deviation processing on the unstable data through a built-in second tree model to obtain a second deviation fitting result;
[0180] The fifth execution unit is used to perform mean processing on the first deviation fitting result and the second deviation fitting result to obtain a second fitting result.
[0181] In some optional implementations, the fifth processing unit further includes:
[0182] A sixth execution unit, configured to use the second data distribution at the current time step as a supplementary distribution if the null value rate represented by the null value rate detection result is greater than a specified null value rate threshold;
[0183] The fusion unit includes:
[0184] A first fusion subunit, used to fuse the first prediction result of the current data prediction model with the first prediction results of other data prediction models to obtain an intermediate fusion result;
[0185] The second fusion subunit is used to fuse the supplementary distribution with the intermediate fusion result to obtain the target prediction result.
[0186] In some optional implementations, the time distribution intervals corresponding to the multiple candidate data prediction models are in a gradient-growing and non-intersecting relationship, and the model structure of each candidate data prediction model is the same.
[0187] In some optional embodiments, the device further comprises:
[0188] The display module is used to display the target prediction results.
[0189] In some optional embodiments, the device further comprises:
[0190] The sending module is used to send the target prediction results to the scheduling system to determine the target shift group for maintaining the time series data set to be predicted.
[0191] In some optional embodiments, the device further comprises:
[0192] The adjustment module is used to adjust the target team based on the target prediction results and prior knowledge.
[0193] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0194] The data processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0195] The present disclosure also provides an electronic device having the above Figure 8 The data processing device shown.
[0196] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present disclosure, such as Fig. 9 As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.
[0197] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0198] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0199] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0200] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0201] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.
[0202] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0203] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0204] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0205] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0206] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.
[0207] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0208] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0209] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A data processing method, It is characterized in that The method comprises: Determine the data set prediction target according to the data processing requirements of the current task, wherein the data set prediction target includes the target data granularity, the target time granularity and the target observation duration; Acquire historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted; Based on the first data distribution of the historical received processing volume data at the target time granularity, determine the target time distribution length corresponding to the historical received processing volume data, and use the data prediction model corresponding to the target time distribution length to predict the amount of data to be processed of the time series data set to be predicted at the target observation time length to obtain a target prediction result, which is used to process the current task.
2. The method according to claim 1, It is characterized in that The obtaining of historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted includes: Acquire initial historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted; Preprocessing the initial historical received processing volume data to obtain a first preprocessing result of the initial historical received processing volume data; Determining the time characteristics of the initial historical received processing volume data based on the historical processing time corresponding to the initial historical received processing volume data; The historical receiving processing volume data is obtained based on the first preprocessing result and the time feature.
3. The method according to claim 2, It is characterized in that The preprocessing of the initial historical received processing volume data to obtain a first preprocessing result of the initial historical received processing volume data includes: Performing outlier detection on the initial historical received processing volume data to obtain a detection result; If the detection result indicates that an abnormal value exists in the initial historical received processing volume data, the position of the abnormal value in the initial historical received processing volume data is determined, and a specified value is added to the position to eliminate the abnormal value to obtain the first preprocessing result.
4. The method according to claim 2 or 3, It is characterized in that If the current task is a target task, obtaining the historical received processing volume data based on the first preprocessing result and the time feature includes: Acquire task feature data corresponding to the target task; Preprocessing the task characteristic data to obtain a second preprocessing result; The first preprocessing result, the time feature and the second preprocessing result are integrated to obtain the historical receiving processing volume data.
5. The method according to claim 1, It is characterized in that The determining, based on the first data distribution of the historical received processing volume data at the target time granularity, a time distribution length corresponding to the historical received processing volume data includes: Determine a mapping relationship between the target time granularity and the time step; According to the mapping relationship, a first data distribution of the historical received processing volume data at the target time granularity is summarized to determine a second data distribution of the historical received processing volume data at multiple time steps; According to the second data distribution under the multiple time steps, the time distribution length corresponding to the historical received processing volume data is determined.
6. The method according to claim 5, It is characterized in that The determining, according to the second data distribution under the multiple time steps, the time distribution length corresponding to the historical received processing volume data includes: Performing a null value rate detection on the second data distribution at the current time step to obtain a null value rate detection result; If the null value rate represented by the null value rate detection result is less than or equal to the specified null value rate threshold, the second data distribution under the current time step is determined to be a valid distribution, and combined with the time steps corresponding to other valid distributions, the time distribution length corresponding to the historical received processing volume data is obtained.
7. The method according to claim 6, It is characterized in that The process of determining the data prediction model includes: Matching the time distribution length with the time distribution intervals corresponding to multiple candidate data prediction models respectively to obtain matching results; If the matching result indicates that the current time distribution interval includes the entire or part of the time distribution length, the candidate data prediction model corresponding to the current time distribution interval is used as the data prediction model, and the number of the data prediction models is at least 1.
8. The method according to claim 7, It is characterized in that If there are multiple data prediction models, the data amount to be processed of the time series data set to be predicted under the target observation time length is predicted by the data prediction model to obtain the target prediction result, including: Determining input data corresponding to the current data prediction model from the second data distribution under the multiple time steps according to the time distribution interval of the current data prediction model; Based on the input data, predict the amount of data to be processed of the time series data set to be predicted under the target observation duration to obtain a first prediction result of the current data prediction model; The first prediction result of the current data prediction model is merged with the first prediction results of other data prediction models to obtain the target prediction result.
9. The method according to claim 8, It is characterized in that The step of predicting the amount of data to be processed of the time series data set to be predicted under the target observation duration based on the input data to obtain a first prediction result of the current data prediction model includes: Identify stable data and unstable data in the input data; Performing linear fitting processing on the stable data to obtain a first fitting result, and predicting the amount of data to be processed of the time series data set to be predicted under the target observation duration based on the first fitting result to obtain a first prediction result; Perform fitting deviation processing on the non-stable data to obtain a second fitting result, and predict the amount of data to be processed of the time series data set to be predicted under the target observation duration based on the second fitting result to obtain a second prediction result; The first prediction result and the second prediction result are integrated to obtain the first prediction result.
10. The method according to claim 9, It is characterized in that The performing fitting deviation processing on the unstable data to obtain a second fitting result includes: Perform fitting deviation processing on the unstable data through a built-in first tree model to obtain a first deviation fitting result; Perform fitting deviation processing on the unstable data through a built-in second tree model to obtain a second deviation fitting result; The first deviation fitting result and the second deviation fitting result are averaged to obtain the second fitting result.
11. The method according to claim 8, It is characterized in that The determining, according to the second data distribution under the multiple time steps, the time distribution length corresponding to the historical received processing volume data further includes: If the null value rate characterized by the null value rate detection result is greater than the specified null value rate threshold, the second data distribution at the current time step is used as a supplementary distribution; The fusing the first prediction result of the current data prediction model with the first prediction results of other data prediction models to obtain the target prediction result includes: Fusing the first prediction result of the current data prediction model with the first prediction results of other data prediction models to obtain an intermediate fusion result; The supplementary distribution is fused with the intermediate fusion result to obtain the target prediction result.
12. The method according to claim 7, It is characterized in that The time distribution intervals corresponding to the multiple candidate data prediction models are in a gradient-growing and non-intersecting relationship, and the model structure of each candidate data prediction model is the same.
13. The method according to claim 1, It is characterized in that The method further comprises: The target prediction result is displayed.
14. The method according to claim 1 or 13, It is characterized in that The method further comprises: The target prediction result is sent to the scheduling system to determine the target shift group for maintaining the time series data set to be predicted.
15. The method according to claim 14, It is characterized in that The method further comprises: The target shift is adjusted based on the target prediction result and prior knowledge.
16. A data processing device, It is characterized in that The device comprises: A first processing module is used to determine a data set prediction target according to the data processing requirements of the current task, wherein the data set prediction target includes a target data granularity, a target time granularity, and a target observation duration; An acquisition module, used to acquire historical received processing volume data corresponding to the target data granularity in the time series data set to be predicted; The second processing module is used to determine the target time distribution length corresponding to the historical received processing volume data based on the first data distribution of the historical received processing volume data at the target time granularity, and predict the amount of data to be processed of the time series data set to be predicted at the target observation time length through a data prediction model corresponding to the target time distribution length to obtain a target prediction result, which is used to process the current task.
17. An electronic device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data processing method according to any one of claims 1 to 15 by executing the computer instructions.
18. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data processing method according to any one of claims 1 to 15.
19. A computer program product, It is characterized in that The method comprises computer instructions for causing a computer to execute the data processing method according to any one of claims 1 to 15.
Citation Information
Cited By
Transaction data processing method and device, equipment, storage medium and program product
CN119671718A
