Data processing method and apparatus

Abstract

The application discloses a data processing method and device. The method comprises the following steps: in the case that a parent node corresponding to a target node in a target workflow is run completely, obtaining a first variable according to a running time length of the parent node in a current running process of the target workflow; obtaining a second variable according to a running time length of the target node in a historical running process of the target workflow; and obtaining a predicted running time length of the target node according to at least the first variable and the second variable.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and apparatus. Background Technology

[0002] A workflow is a series of nodes that execute different programs sequentially or in parallel. The types of nodes and their execution order are fixed for each workflow. Some nodes perform tasks based on the amount of business data that the current workflow needs to process, and their runtime will vary depending on the amount and complexity of data to be calculated.

[0003] Currently, predicting node runtime and monitoring for timeouts requires considering the size and complexity of the data. However, these metrics are only available after the entire workflow and all nodes have completed their runs.

[0004] Therefore, the above scheme for predicting node runtime suffers from low prediction efficiency. Summary of the Invention

[0005] In view of the above, this application provides a data processing method and apparatus, as follows:

[0006] A data processing method, comprising:

[0007] If the parent node corresponding to the target node in the target workflow has completed its execution, the first variable is obtained based on the runtime of the parent node in the current execution flow of the target workflow.

[0008] The second variable is obtained based on the runtime of the target node in the historical execution process of the target workflow;

[0009] The predicted runtime of the target node is obtained based at least on the first variable and the second variable.

[0010] The above method, preferably, obtains the first variable based on the runtime of the parent node in the current running process of the target workflow, including:

[0011] Principal component analysis is performed on the runtime of the parent node in the current running process of the target workflow to obtain the first principal component corresponding to the parent node;

[0012] Obtain the principal component score corresponding to the first principal component of the parent node, and use the principal component score as the first variable.

[0013] The above method, preferably, involves performing principal component analysis on the runtime of the parent node in the current running process of the target workflow to obtain the first principal component corresponding to the parent node, including:

[0014] Based on the runtime of the parent node in the current running process of the target workflow, the principal component loading coefficient matrix corresponding to the parent node is obtained. The matrix elements in the principal component loading coefficient matrix represent the degree of influence of the corresponding principal component on the runtime of the corresponding parent node.

[0015] The first principal component is obtained based on the matrix elements in the principal component loading coefficient matrix, and the degree of influence of the first principal component on the runtime of the parent node satisfies the first screening condition.

[0016] The above method, preferably, involves obtaining the principal component score of the parent node corresponding to the first principal component, including:

[0017] Obtain the first loading coefficient matrix corresponding to the first principal component, where the matrix elements in the first loading coefficient matrix characterize the degree of influence of the first principal component on the runtime of the corresponding parent node.

[0018] The runtime of the parent node is processed based on the matrix elements in the first load coefficient matrix to obtain the principal component score of the parent node corresponding to the first principal component.

[0019] In the above method, preferably, the parent node is the parent node in the target workflow whose correlation coefficient with the target node regarding runtime satisfies the second screening condition.

[0020] The above method, preferably, obtains a second variable based on the runtime of the target node in the historical execution flow of the target workflow, including:

[0021] Obtain the partial autocorrelation coefficient between the runtime of the target node in the first running process and the runtime of the target node in at least one second running process, wherein the first running process is the most recent running process of the target workflow, and the second running process is the Nth running process before the first running process, where N is a positive integer greater than or equal to 1.

[0022] The second variable is obtained at least based on the partial autocorrelation coefficient.

[0023] The above method, preferably, at least based on the partial autocorrelation coefficient, obtains the second variable, including:

[0024] Based on the partial autocorrelation coefficient, at least one target runtime is obtained. The target runtime is the runtime of the target node in the third running process. The third running process is the running process in the second running process in which the corresponding partial autocorrelation coefficient satisfies the third screening condition.

[0025] The second variable is obtained based on the target runtime and its corresponding partial autocorrelation coefficient.

[0026] Preferably, after obtaining the predicted runtime of the target node based at least on the first variable and the second variable, the method further includes:

[0027] Obtain the performance parameters of the running database corresponding to the target node;

[0028] The predicted runtime is adjusted based on the performance parameters.

[0029] The above method, preferably, involves obtaining the performance parameters of the running database corresponding to the target node, including:

[0030] Obtain the index value of the running database corresponding to the target node on at least one performance metric;

[0031] Based on the index value, an expansion coefficient is obtained, which characterizes the performance parameters of the running database corresponding to the target node.

[0032] A data processing apparatus, comprising:

[0033] The first variable acquisition unit is used to obtain the first variable based on the running time of the parent node in the current running process of the target workflow when the parent node corresponding to the target node in the target workflow has completed running.

[0034] The second variable acquisition unit is used to obtain a second variable based on the runtime of the target node in the historical operation process of the target workflow;

[0035] A runtime prediction unit is used to obtain the predicted runtime of the target node based at least on the first variable and the second variable.

[0036] As can be seen from the above technical solutions, in the data processing method and apparatus disclosed in this application, the runtime of the target node is predicted based on the runtime of the parent node that has already completed its operation in the current process and the runtime of the target node in the historical process. Therefore, this application can predict the runtime of nodes in the target workflow using the runtime of the already completed parent node without waiting for all nodes in the workflow to complete, thereby improving prediction efficiency. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart of a data processing method provided in Embodiment 1 of this application;

[0039] Figure 2 Example diagram of the target workflow;

[0040] Figure 3 , Figure 4 and Figure 5 These are partial flowcharts of a data processing method provided in Embodiment 1 of this application;

[0041] Figure 6 Another flowchart of a data processing method provided in Embodiment 1 of this application;

[0042] Figure 7 This is a schematic diagram of the structure of a data processing device provided in Embodiment 2 of this application;

[0043] Figure 8 This is another structural schematic diagram of a data processing device provided in Embodiment 2 of this application;

[0044] Figure 9 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of this application;

[0045] Figure 10 For the purposes of this application Figure 2 The flowchart shown illustrates the workflow for runtime prediction and early warning.

[0046] Figure 11 Here is a sample diagram of a specific workflow in the JDA Planning system;

[0047] Figure 12 This is a schematic diagram of the running time of four nodes with strong correlation coefficients in this application;

[0048] Figure 13 This is an example diagram of the order selected by the partial autocorrelation coefficient in this application;

[0049] Figure 14 A diagram showing the comparison between the predicted and actual values ​​of this application;

[0050] Figure 15 This diagram illustrates the comparison between the predicted values ​​before and after correction using database performance metrics, and the actual values.

[0051] Figure 16 This is a diagram illustrating the average fluctuation trend of the relevant performance metrics of the database running on the node. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] refer to Figure 1 The diagram shown is a flowchart illustrating the implementation of a data processing method according to Embodiment 1 of this application. This method can be applied to electronic devices capable of data processing, such as computers or servers. The electronic device can be a device executing a target workflow, or it can be a device with a data connection established with the device executing the target workflow. The technical solution in this embodiment is mainly used to improve the efficiency of predicting node runtime.

[0054] Specifically, the method in this embodiment may include the following steps:

[0055] Step 101: Monitor whether the parent node corresponding to the target node in the target workflow has completed its operation. If the parent node corresponding to the target node in the target workflow has completed its operation, proceed to step 102.

[0056] The target node is the node whose runtime needs to be predicted, and the parent node corresponding to the target node is the node in the target workflow that needs to be completed before the target node can be executed.

[0057] For example, such as Figure 2 As shown, the target workflow includes nodes Node_A, Node_B, Node_C, Node_D, Node_F, Node_G, Node_H, and Node_I. The node whose runtime needs to be predicted is Node_G, i.e., the target node, and its parent nodes are Node_A, Node_C, Node_D, and Node_F. Based on this, in this embodiment, for Node_G, we monitor whether its parent nodes Node_A, Node_C, Node_D, and Node_F have all completed their execution.

[0058] Step 102: Obtain the first variable based on the runtime of the parent node in the current running process of the target workflow.

[0059] In this embodiment, the current record information corresponding to the target workflow in the current running process can be read first. The current record information includes the start time and end time of the nodes that have been completed in the target workflow. Based on this, the start time and end time of the parent node of the target node can be obtained. Then, the difference between the end time and the start time is calculated to obtain the running time of each parent node in the current running process. Finally, based on the running time of these parent nodes in the current running process, the first variable is obtained, denoted by PC1.

[0060] For example, such as Figure 2 As shown, for the target node Node_G, after all its parent nodes have finished running, but before Node_G runs, the runtime of its parent nodes in the current running process is obtained. That is, based on the current record information of the target workflow, the runtime of each of Node_A, Node_C, Node_D, and Node_F is calculated.

[0061] It should be noted that in this embodiment, the first variable represents the degree of influence of one or more target factors affecting the runtime of a node in the current running process on the runtime of the parent node in the current running process, and the first variable represents the degree of influence of the target factors on the runtime of the target node in the current running process.

[0062] Among them, the target factors can be factors such as data processing volume and processing business type, which will affect the runtime of the node in the target workflow.

[0063] Step 103: Obtain the second variable based on the runtime of the target node in the historical execution process of the target workflow.

[0064] In this embodiment, the historical records of the target workflow in at least one historical process can be read first. Each historical record corresponds to a historical process, and each historical record contains the start and end times of each node in the target workflow in its corresponding historical process. Based on this, the start and end times of the target node in each historical process are obtained. Then, the difference between the end time and the start time is calculated to obtain the runtime of the target node in each historical process. Finally, based on the runtime of the target node in these historical processes, a second variable, denoted by z, is obtained.

[0065] It should be noted that the second variable can characterize the changing trend of the target node's runtime in the historical operation process.

[0066] In this embodiment, it can be as follows: Figure 1The step 102 is executed first, then step 103. Alternatively, step 103 can be executed first, then step 102, or both steps 102 and 103 can be executed simultaneously. Different technical solutions resulting from different execution orders of steps 102 and 103 are all within the scope of the technical solutions in this application.

[0067] Step 104: Obtain the predicted runtime of the target node based at least on the first and second variables.

[0068] Specifically, in this embodiment, the first and second variables can be substituted into the binary linear regression model. The binary linear regression model uses the second and first variables as independent variables and the predicted runtime of the target node as the dependent variable. Based on this, the predicted runtime of the target node is obtained through the binary linear regression model.

[0069] In the binary linear regression model, a first coefficient and a second coefficient are set for the first and second variables, along with a constant. These first and second coefficients, as well as the constant, can be set based on historical experience or obtained through training with historical data.

[0070] For example, taking the binary linear regression model y = a × PC1 + b × z + c as an example, a is the first coefficient, b is the second coefficient, and c is a constant. Based on this, in this embodiment, the first variable PC1 and the second variable z are substituted into the model to obtain the predicted runtime y of the target node.

[0071] As can be seen from the above technical solutions, in the data processing method provided in Embodiment 1 of this application, the runtime of the target node is predicted based on the runtime of the parent node that has already completed running in the current running process and the runtime of the target node in the historical running process. Therefore, in this embodiment, it is not necessary to wait for all nodes in the target workflow to complete running; the runtime of the completed parent node can be used to predict the runtime of the nodes, thereby improving prediction efficiency.

[0072] In one implementation, when obtaining the first variable in step 102 based on the parent node's runtime in the current running process of the target workflow, it can be achieved in the following way: Figure 3 As shown:

[0073] Step 301: Perform principal component analysis on the runtime of the parent node in the current running process of the target workflow to obtain the first principal component corresponding to the parent node.

[0074] Among them, the principal components are the factors that affect the runtime of the nodes, and the first principal component is the target factor whose influence on the runtime of the parent node meets the first screening condition.

[0075] Specifically, in this embodiment, the first principal component can be obtained in the following way:

[0076] First, based on the runtime of the parent node in the current running process of the target workflow, the principal component load coefficient matrix corresponding to the parent node is obtained. The matrix elements in the principal component load coefficient matrix represent the degree of influence of the corresponding principal component on the runtime of the corresponding parent node.

[0077] Then, based on the matrix elements in the principal component loading coefficient matrix, the first principal component is obtained, and the degree of influence of the first principal component on the runtime of the parent node satisfies the first screening condition.

[0078] Specifically, in the principal component loading coefficient matrix, the principal components are arranged in rows and the parent nodes in columns. The principal component loading coefficient matrix can have one or more rows and one or more columns. Each element in the principal component loading coefficient matrix represents the degree of influence of the principal component in its row on the runtime of the parent node in its column.

[0079] The first screening criterion can be: the top M principal components whose influence on the runtime of the parent node is ranked from largest to smallest among all principal components, where M is a positive integer greater than or equal to 1. Based on this, in this embodiment, rows can be screened according to the variance of the matrix elements in each row, and the top M rows whose variances are ranked from largest to smallest can be selected. In this embodiment, M is set to 1. Thus, the principal component corresponding to the row with the largest variance of the matrix elements is the first principal component.

[0080] Step 302: Obtain the principal component score corresponding to the first principal component of the parent node, and use the principal component score as the first variable.

[0081] Specifically, step 302 can obtain the principal component scores in the following way:

[0082] First, obtain the first loading coefficient matrix corresponding to the first principal component, which is the matrix composed of the matrix elements of the row corresponding to the first principal component in the principal component loading coefficient matrix. The matrix elements in the first loading coefficient matrix represent the degree of influence of the first principal component on the runtime of the corresponding parent node.

[0083] Then, based on the matrix elements in the first load coefficient matrix, the runtime of the parent node is processed to obtain the principal component score of the parent node corresponding to the first principal component.

[0084] Specifically, in this embodiment, the matrix elements in the first load coefficient matrix can be used as the principal component weight values ​​corresponding to the parent node. Based on this, the runtime of the corresponding parent node is weighted and summed or weighted and averaged using the matrix elements in the first load coefficient matrix, that is, multiplied and then added or multiplied and then averaged to obtain the weighted value of the parent node corresponding to the first principal component, that is, the principal component score, which is the first variable.

[0085] In one implementation, the parent node in steps 102 and 103 is the parent node in the target workflow whose correlation coefficient with the target node regarding runtime satisfies the second screening condition.

[0086] Specifically, before steps 102 and 103, this embodiment first filters the parent nodes corresponding to the target node in the target workflow. The filtering method is as follows:

[0087] First, obtain the runtime of the target node and all its parent nodes in the historical execution process of the target workflow;

[0088] Then, using the runtime of the target node and all its parent nodes as variables, the correlation coefficient between the target node and each parent node is obtained. For example, Spearman correlation analysis is performed between the target node and each parent node to obtain the corresponding correlation coefficient.

[0089] Finally, the parent nodes whose correlation coefficients meet the second screening criteria are selected to obtain the first and second variables.

[0090] The second filtering condition can be: the correlation coefficients between the parent node and the target node are ranked from largest to smallest, with X being a positive integer greater than or equal to 1, and X can be set according to requirements. Alternatively, the second filtering condition can be: the correlation coefficient between the parent node and the target node is greater than or equal to a preset first threshold, and the first threshold can be set according to requirements. Based on this, in this embodiment, parent nodes can be filtered according to the second filtering condition to obtain the first variable and the second variable in steps 102 and 103.

[0091] In one implementation, when obtaining the second variable in step 103 based on the runtime of the target node in the historical execution flow of the target workflow, it can be specifically implemented in the following way: Figure 4 As shown:

[0092] Step 401: Obtain the partial autocorrelation coefficient between the runtime of the target node in the first running process and the runtime of the target node in the second running process.

[0093] Here, the first execution flow is the most recent execution flow of the target workflow, and the second execution flow is the Nth execution flow preceding the first execution flow, where N is a positive integer greater than or equal to 1. For example, such as... Figure 5 As shown, there are multiple second running processes, namely: the running process preceding the first running process, the running process 2nd before the first running process, and so on, as well as the running process Nth before the first running process. Based on this, in this embodiment, for the target node, the partial autocorrelation coefficient is calculated for its runtime in the first running process and the runtime in each second running process, to obtain N partial autocorrelation coefficients.

[0094] Step 402: Obtain the second variable based at least on the partial autocorrelation coefficient.

[0095] In this embodiment, the runtime of the target node in the second running process can be processed by the partial autocorrelation coefficient to obtain a trend that can characterize the change of the runtime of the target node in the historical running process.

[0096] Specifically, in step 402, the second variable can be obtained in the following way, such as... Figure 5 As shown:

[0097] Step 501: Obtain at least one target runtime based on the partial autocorrelation coefficient.

[0098] The target runtime is the runtime of the target node in the third running process, and the third running process is the running process in the second running process whose corresponding partial autocorrelation coefficient satisfies the third screening condition.

[0099] Specifically, in this embodiment, the partial autocorrelation coefficient of the target node with respect to runtime between the first running process and each second running process is screened to select the second running process corresponding to the partial autocorrelation coefficient that meets the third screening condition, which is denoted as the third running process. Based on this, the runtime of the target node in the third running process is determined as the target runtime.

[0100] There can be one or more partial autocorrelation coefficients that satisfy the third screening condition, and correspondingly, there can be one or more target runtimes. In this case, the number of times the first running process precedes the third running process is the lag order of the first running process relative to the third running process. That is, in this embodiment, the lag order satisfying the third screening condition within N is determined by the partial autocorrelation coefficient, and the runtime of the target node in the historical running process corresponding to that lag order is recorded as the target runtime.

[0101] It should be noted that the third screening condition can be: the partial autocorrelation coefficient is greater than the second threshold. The second threshold can be set according to the requirements. For example, the second threshold can be set to 0.2. Thus, the second running process corresponding to the partial autocorrelation coefficient greater than 0.2 is recorded as the third running process, and the running time of the target node in the third running process is recorded as the target running time.

[0102] Alternatively, the third filtering criterion can be: the partial autocorrelation coefficients are sorted from largest to smallest, with the top Y values ​​being positive integers greater than or equal to 1. Y can be set according to requirements. Therefore, the historical execution flow corresponding to the top Y partial autocorrelation coefficients is recorded as the third execution flow, and the runtime of the target node in this third execution flow is recorded as the target runtime.

[0103] Step 502: Obtain the second variable based on the target runtime and its corresponding partial autocorrelation coefficient.

[0104] Specifically, in this embodiment, the partial autocorrelation coefficient can be used as the weight value of the target runtime. The target runtime is weighted and summed using the partial autocorrelation coefficient to obtain the second variable, so that the second variable can characterize the change trend of the target node's runtime in the historical process.

[0105] In addition, in this embodiment, outliers in the target runtime can be removed before step 502 to improve the accuracy of the second variable.

[0106] Specifically, in this embodiment, outliers in the target runtime can be removed using at least one of the following methods:

[0107] One approach involves removing abnormally large or small values ​​from the target runtime. For example, all target runtimes are sorted from smallest to largest, then the quartile Q3 is selected based on the sorted target runtimes, and the interquartile range IQR is calculated. Finally, values ​​greater than Q3 + 1.5 × IQR in the target runtime are deleted.

[0108] In another approach, target runtimes with missing values ​​are excluded.

[0109] In one implementation, after obtaining the predicted runtime of the target node in step 104, the method in this embodiment may further include the following process, such as... Figure 6 As shown:

[0110] Step 105: Obtain the performance parameters of the running database corresponding to the target node.

[0111] In this embodiment, the performance metrics of the running database corresponding to the target node can be processed on one or more performance indicators, and the processing results of these performance metrics are used as performance parameters of the running database. These performance parameters are related to the execution time of the target workflow and characterize the performance fluctuation trend of the running database.

[0112] Specifically, in this embodiment, the performance parameters of the running database can be obtained in the following way:

[0113] First, obtain the index value of the running database corresponding to the target node on at least one performance metric; for example, in this embodiment, during the operation of the target workflow, each performance metric of the running database can be sampled at multiple sampling time points to obtain the index value corresponding to these performance metrics at each sampling time point.

[0114] Next, based on the index values, the inflation coefficient is obtained. The inflation coefficient characterizes the performance parameters of the running database corresponding to the target node. For example, in this embodiment, for each performance index, the average value of the index value at each sampling time point can be calculated. After obtaining the average value of each performance index at each sampling time point, the index value of each performance index at each sampling time point is then multiplied by the average value corresponding to that performance index. The resulting product is used as the inflation coefficient to characterize the performance parameters of the running database.

[0115] For example, the target node's running database has performance metrics KPI1, KPI2, KPI3, KPI4, and KPI5. Correspondingly, after sampling once per second within one hour, the corresponding metric values ​​of these five performance metrics at 60 sampling time points are obtained, which can be represented by KPI1, KPI2, KPI3, KPI4, and KPI5. The average value of each performance metric is calculated, i.e., KPI1_avg to KPI5_avg. Then, the expansion coefficient Bata is constructed, which is (KPI1 / KPI1_avg)×(KPI2 / KPI2_avg)×(KPI3 / KPI3_avg)×(KPI4 / KPI4_avg)×(KPI5 / KPI5_avg).

[0116] Step 106: Adjust the predicted runtime based on the performance parameters.

[0117] Specifically, in this embodiment, performance parameters can be used to inflate the predicted runtime. For example, the inflation coefficient in the performance parameters can be multiplied by the predicted runtime to obtain the adjusted predicted runtime, so that the predicted runtime can match the running state of the database.

[0118] It should be noted that the predicted runtime obtained in this embodiment can be used to issue timeout warnings for target nodes. For example, if the predicted runtime is greater than or equal to a duration threshold, an alarm message is output to indicate that the target node may be at risk of timeout.

[0119] refer to Figure 7 This is a schematic diagram of a data processing device provided in Embodiment 2 of this application. This device can be configured in an electronic device capable of data processing, such as a computer or server. The electronic device can be a device that executes the target workflow, or it can be a device that establishes a data connection with the device executing the target workflow. The technical solution in this embodiment is mainly used to improve the efficiency of predicting node runtime.

[0120] Specifically, the device in this embodiment may include the following units:

[0121] The first variable acquisition unit 701 is used to obtain the first variable based on the running time of the parent node in the current running process of the target workflow when the parent node corresponding to the target node in the target workflow has finished running.

[0122] The second variable acquisition unit 702 is used to obtain a second variable based on the runtime of the target node in the historical operation process of the target workflow;

[0123] Runtime prediction unit 703 is used to obtain the predicted runtime of the target node based at least on the first variable and the second variable.

[0124] As can be seen from the above technical solutions, in the data processing apparatus provided in Embodiment 2 of this application, the runtime of the target node is predicted based on the runtime of the parent node that has already completed its operation in the current process and the runtime of the target node in the historical process. Therefore, in this embodiment, it is not necessary to wait for all nodes in the target workflow to complete their operation; the runtime of the completed parent node can be used to predict the runtime of the nodes within it, thereby improving prediction efficiency.

[0125] In one implementation, the first variable obtaining unit 701 is specifically used to: perform principal component analysis on the runtime of the parent node in the current running process of the target workflow to obtain the first principal component corresponding to the parent node; obtain the principal component score of the parent node corresponding to the first principal component, and use the principal component score as the first variable.

[0126] Specifically, when obtaining the first principal component corresponding to the parent node, the first variable obtaining unit 701 is used to: obtain the principal component loading coefficient matrix corresponding to the parent node based on the runtime of the parent node in the current running process of the target workflow, wherein the matrix elements in the principal component loading coefficient matrix represent the degree of influence of the corresponding principal component on the runtime of the corresponding parent node; and obtain the first principal component based on the matrix elements in the principal component loading coefficient matrix, wherein the degree of influence of the first principal component on the runtime of the parent node satisfies the first screening condition.

[0127] Specifically, when obtaining the principal component score of the parent node corresponding to the first principal component, the first variable obtaining unit 701 is used to: obtain the first loading coefficient matrix corresponding to the first principal component, wherein the matrix elements in the first loading coefficient matrix represent the degree of influence of the first principal component on the runtime of the corresponding parent node; and process the runtime of the parent node according to the matrix elements in the first loading coefficient matrix to obtain the principal component score of the parent node corresponding to the first principal component.

[0128] In one implementation, the parent node is the parent node in the target workflow whose correlation coefficient with the target node regarding runtime satisfies the second screening condition.

[0129] In one implementation, the first variable obtaining unit 701 is specifically used to: obtain the partial autocorrelation coefficient between the runtime of the target node in the first running process and the runtime of the target node in at least one second running process, wherein the first running process is the most recent running process of the target workflow, and the second running process is the Nth running process before the first running process, where N is a positive integer greater than or equal to 1; and obtain the second variable based at least on the partial autocorrelation coefficient.

[0130] Specifically, when the first variable obtaining unit 701 obtains the second variable based at least on the partial autocorrelation coefficient, it is used to: obtain at least one target runtime based on the partial autocorrelation coefficient, wherein the target runtime is the runtime of the target node in the third running process, and the third running process is the running process in the second running process whose corresponding partial autocorrelation coefficient satisfies the third screening condition; and obtain the second variable based on the target runtime and its corresponding partial autocorrelation coefficient.

[0131] In one implementation, the apparatus in this embodiment may further include the following units, such as... Figure 8 As shown:

[0132] The runtime adjustment unit 704 is used to obtain the performance parameters of the running database corresponding to the target node after the runtime prediction unit 703 obtains the predicted runtime of the target node based on at least the first variable and the second variable; and adjusts the predicted runtime based on the performance parameters.

[0133] Specifically, when the runtime adjustment unit 704 obtains the performance parameters of the running database corresponding to the target node, it is used to: obtain the index value of the running database corresponding to the target node on at least one performance indicator; and obtain the expansion coefficient based on the index value, wherein the expansion coefficient represents the performance parameters of the running database corresponding to the target node.

[0134] It should be noted that the specific implementation of each unit in this embodiment can be referred to the corresponding content above, and will not be described in detail here.

[0135] refer to Figure 9 This is a schematic diagram of an electronic device provided in Embodiment 3 of this application. The electronic device can be a sub-device capable of data processing, such as a computer or server. The electronic device can be a device that executes a target workflow, or it can be a device that establishes a data connection with the device executing the target workflow. The technical solution in this embodiment is mainly used to improve the efficiency of predicting node runtime.

[0136] Specifically, the electronic device in this embodiment may include the following structure:

[0137] Memory 901 is used to store computer programs and data generated during the execution of computer programs;

[0138] The processor 902 is configured to execute a computer program to: when the parent node corresponding to the target node in the target workflow has completed its execution, obtain a first variable based on the runtime of the parent node in the current execution flow of the target workflow; obtain a second variable based on the runtime of the target node in the historical execution flow of the target workflow; and obtain a predicted runtime of the target node based on at least the first variable and the second variable.

[0139] As can be seen from the above technical solutions, in the electronic device provided in Embodiment 3 of this application, the runtime of the target node is predicted based on the runtime of the parent node that has already completed running in the current running process and the runtime of the target node in the historical running process. Therefore, in this embodiment, it is not necessary to wait for all nodes in the target workflow to complete running; the runtime of the completed parent node can be used to predict the runtime of the nodes, thereby improving prediction efficiency.

[0140] It should be noted that the specific implementation of the processor in this embodiment can be referred to the corresponding content above, and will not be described in detail here.

[0141] Taking the workflow of the JDA Planning system as an example, some nodes perform tasks based on the amount of business data that the current workflow needs to process. Their runtime (i.e., the runtime mentioned above) will vary depending on the amount and complexity of the data to be calculated. If we want to predict the runtime of such nodes and monitor whether they time out, in addition to considering historical factors, we also need to take the size and complexity of the data into account. However, these metric values ​​can only be known after the entire workflow and all nodes have completed their runs.

[0142] In addition, the idea that the runtime of a job node is only affected by the amount of business data is an ideal situation. In real-world scenarios, the performance of the device responsible for running the node program is also an important influencing factor. Most of the jobs in JDA Planning are also performing calculations and operations on the database. If the database performance deteriorates, the runtime of the currently running node may also increase.

[0143] Therefore, the first problem addressed by the technical solution of this application is to replace this factor with other observable indicators, and to predict the node's runtime by combining this influencing factor with historical factors. The second problem to be solved is to adjust the predicted node runtime in real time using key database performance indicators.

[0144] Specifically, in the technical solution of this application, the running time of the predicted node is used as the dependent variable. Principal component analysis is performed on the running times of the parent nodes that are correlated with the node, and the obtained first principal component score is used as the first independent variable (i.e., the first variable mentioned above). The partial correlation coefficient of the node's running time is calculated as the weighting coefficient, and the weighted variable constructed using nodes from the past N periods is used as the second independent variable (i.e., the second variable mentioned above). Based on this, a multiple linear regression model (i.e., the bivariate linear regression model mentioned above) is established. When it is necessary to predict the running time of the node, after the relevant parent nodes have completed running, the relevant variables are substituted into the model for calculation to obtain the predicted value of the node's running time, i.e., the predicted running time of the target node.

[0145] Furthermore, the technical solution of this application, based on the previously obtained predicted values ​​and combined with the performance indicators of the selected key database (i.e., the running database mentioned above), adjusts the predicted values ​​in real time, calculates the runtime of the current situation, and thus provides a timeout warning.

[0146] by Figure 2Taking the workflow shown as an example, it represents a complete workflow execution mechanism. Node_G is the target node, and the execution time is y(t), where t represents the current execution flow of the workflow. To predict the execution time of this target node and issue a timeout warning, the specific process is as follows: Figure 10 The process is shown in the diagram:

[0147] (1) Select the past 100 running records as sample data, which contain multiple historical records, and calculate the time consumption of each node.

[0148] (2) Perform Spearman correlation analysis on parent nodes Node_A, Node_C, Node_D, and Node_F, and select parent nodes with correlation coefficients > 0.2, such as Node_C, Node_D, and Node_F.

[0149] (3) Perform principal component analysis on the running time of Node_C, Node_D and Node_F, calculate the first principal component loading coefficient matrix and the first principal component score PC1 of each sample data as the first independent variable.

[0150] (4) Calculate the partial autocorrelation coefficient of y(t), select lag orders with partial autocorrelation coefficients greater than 0.2, such as p, q, w, with correlation coefficients of r(p), r(q), r(w), and construct new variables y(tp), y(tq), y(tw).

[0151] (5) Calculate the upper quartile Q3 and interquartile range IQR of y(t), and remove samples with possible missing values ​​where y(t), y(tp), y(tq), and y(tw) are greater than Q3 + 1.5IQR.

[0152] (6) Construct the second independent variable as z(t)=r(p)*y(tp)+r(q)*y(tq)+r(w)*y(tw).

[0153] (7) With y(t) as the dependent variable and PC1 and z(t) as independent variables, establish a multiple regression model M, i.e. y(t)=a*PC1+b*z(t)+c.

[0154] (8) After the new workflow is started and all the relevant parent nodes have finished running, calculate the score of the first principal component based on the first principal component load coefficient matrix; calculate z(t).

[0155] (9) Substitute the above two variables into model M to make a preliminary estimate of the node runtime y1.

[0156] (10) Calculate the following performance metrics for the database where the current node resides:

[0157] KPI1:[Oracle_Session]Instance sessions active;

[0158] KPI2:[Oracle_General]Instance user connections;

[0159] KPI3:[Oracle_Load]Load 1AAS;

[0160] KPI4:[Linux_CPU]Processor Idle Util(1Min Average);

[0161] KPI5:[Linux_DiskIO]IO%util.

[0162] And calculate the average values ​​of the above indicators for the selected node runtime period (2) from KPI1_avg to KPI5_avg, and construct the inflation coefficient: Beta=(KPI 1 / KPI 1_avg)*(KPI 2 / KPI 2_avg)*(KPI 3 / KPI 3_avg)*(KPI 4 / KPI 4_avg)*(KPI 5 / KPI 5_avg).

[0163] (11) Final node time prediction = Beta * y1.

[0164] Based on the above implementation scheme, the technical solution of this application is applicable to important work nodes whose runtime affects downstream business. For example, in production scheduling systems like JDA Planning, the workflow output provides material orders and production volume arrangements for various manufacturing plants, making it a crucial link in the manufacturing industry's supply chain. Factors such as work procedure anomalies, incorrect upstream system data, integration system anomalies, and database failures can all cause nodes to fail to complete on time. In such cases, timeout monitoring becomes a means of early detection of anomalies. However, for jobs with variable workloads, the runtime of each node does not fluctuate around a fixed level but rather oscillates significantly with changes in the amount of business data. In this case, incorporating metrics that measure data factors significantly improves the accuracy of node runtime prediction. Simultaneously, adjusting prediction values ​​based on changes in key infrastructure metrics such as database performance plays a significant role in helping users know when workflows will complete so they can schedule work and receive timeout warnings.

[0165] Figure 11To illustrate a specific workflow of the JDA Planning system, the empirical part of this application uses the prediction of the Engine Report node's runtime as an example.

[0166] The Engine report has four parent nodes: Data_Massage, SupplyPlan, and Shortage_Run1. The Spearman correlation coefficients are shown in Table 1. Data_Massage has a weak correlation and is not included in the calculation. The runtimes of the remaining four nodes are as follows: Figure 12 As shown, the nodes exhibit consistent direction of change. Principal component analysis is performed on these four nodes, and the first principal component, the loading coefficient, can be expressed as: PC1 = -0.013*Data_Download + 0.707*Shortage_Run1 + 0.707*SupplyPlan.

[0167] Partial autocorrelation analysis was performed on the time y(t) of the Engine report, and the results are as follows: Figure 13 As shown, the time consumption of lag 1, lag 9, and lag 13 is selected, and a new variable z(t) = 0.65*y(t-1) + 0.29*y(t-9) - 0.23*y(t-13) is constructed.

[0168] The modeling results based on PC1 and z(t) show that the explanatory variables are significant, and the predicted values ​​are comparable to the true values. Figure 14 As shown, the prediction results of the running time in the technical solution of this application are effective.

[0169] Table 1 Spearman correlation coefficients

[0170]

[0171]

[0172] In addition, the inclusion of database performance metrics also plays a corresponding role in adjusting the prediction results. Figure 15 Based on the changes in the duration of the above-mentioned work nodes over 6 days, the predicted Engine Report duration using the above scheme is 25.4 minutes, which is much smaller than the actual 112 minutes. Figure 16 The average fluctuation trend of the relevant performance indicators of the database running on the node every 30 minutes has increased significantly. Based on the calculated inflation coefficient of 4.3, the final calculated corrected prediction value of 111 minutes is closer to the actual value, which verifies the accuracy of the technical solution of this application.

[0173] In summary, the technical solution of this application can automatically raise the timeout threshold when business data increases in size or complexity, and automatically lower it when business data decreases, thereby improving prediction accuracy and alarm accuracy. Moreover, the technical solution of this application assigns different weights to the historical time consumption of nodes, making the time consumption of recent nodes have a greater impact on the prediction value, which can also improve prediction accuracy and alarm accuracy. In addition, the technical solution of this application dynamically adjusts the current prediction value based on the real-time changes in database performance, further reflecting the node runtime under real-world conditions.

[0174] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0175] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0176] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0177] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, comprising: When the parent node corresponding to the target node in the target workflow has completed its execution, a first variable is obtained based on the runtime of the parent node in the current execution flow of the target workflow. The first variable characterizes the degree of influence of the target factor on the runtime of the target node in the current execution flow, including: performing principal component analysis on the runtime of the parent node in the current execution flow of the target workflow to obtain the first principal component corresponding to the parent node, where the first principal component is the target factor; obtaining the principal component score of the parent node corresponding to the first principal component, where the principal component score is used as the first variable. Based on the runtime of the target node in the historical operation process of the target workflow, a second variable is obtained. The second variable can characterize the changing trend of the runtime of the target node in the historical operation process, including: obtaining the partial autocorrelation coefficient between the runtime of the target node in a first operation process and the runtime of the target node in at least one second operation process, wherein the first operation process is the most recent operation process of the target workflow, and the second operation process is the Nth operation process before the first operation process, where N is a positive integer greater than or equal to 1; the second variable is obtained at least based on the partial autocorrelation coefficient. Obtaining the predicted runtime of the target node based at least on the first variable and the second variable includes: substituting the first variable and the second variable into a multiple linear regression model, wherein the multiple linear regression model uses the second variable and the first variable as independent variables and the predicted runtime of the target node as the dependent variable to obtain the predicted runtime of the target node.

2. The method according to claim 1, wherein principal component analysis is performed on the runtime of the parent node in the current running process of the target workflow to obtain the first principal component corresponding to the parent node, comprising: Based on the runtime of the parent node in the current running process of the target workflow, the principal component loading coefficient matrix corresponding to the parent node is obtained. The matrix elements in the principal component loading coefficient matrix represent the degree of influence of the corresponding principal component on the runtime of the corresponding parent node. The first principal component is obtained based on the matrix elements in the principal component loading coefficient matrix, and the degree of influence of the first principal component on the runtime of the parent node satisfies the first screening condition.

3. The method according to claim 1 or 2, obtaining the principal component score of the parent node corresponding to the first principal component, includes: Obtain the first loading coefficient matrix corresponding to the first principal component, where the matrix elements in the first loading coefficient matrix characterize the degree of influence of the first principal component on the runtime of the corresponding parent node. The runtime of the parent node is processed based on the matrix elements in the first load coefficient matrix to obtain the principal component score of the parent node corresponding to the first principal component.

4. The method according to claim 1, wherein the parent node is the parent node in the target workflow whose correlation coefficient with the target node regarding runtime satisfies the second screening condition.

5. The method according to claim 1, wherein obtaining the second variable based at least on the partial autocorrelation coefficient includes: Based on the partial autocorrelation coefficient, at least one target runtime is obtained. The target runtime is the runtime of the target node in the third running process. The third running process is the running process in the second running process in which the corresponding partial autocorrelation coefficient satisfies the third screening condition. The second variable is obtained based on the target runtime and its corresponding partial autocorrelation coefficient.

6. The method according to claim 1, after obtaining the predicted runtime of the target node based at least on the first variable and the second variable, the method further includes: Obtain the performance parameters of the running database corresponding to the target node; The predicted runtime is adjusted based on the performance parameters.

7. The method according to claim 6, obtaining the performance parameters of the running database corresponding to the target node, includes: Obtain the index value of the running database corresponding to the target node on at least one performance metric; Based on the index value, an expansion coefficient is obtained, which represents the performance parameters of the running database corresponding to the target node.

8. A data processing apparatus, comprising: The first variable acquisition unit is used to obtain a first variable based on the runtime of the parent node in the current running process of the target workflow, when the parent node corresponding to the target node in the target workflow has completed its operation. The first variable characterizes the degree of influence of the target factor on the runtime of the target node in the current running process. The first variable includes: performing principal component analysis on the runtime of the parent node in the current running process of the target workflow to obtain the first principal component corresponding to the parent node, where the first principal component is the target factor; and obtaining the principal component score of the parent node corresponding to the first principal component, where the principal component score is used as the first variable. The second variable acquisition unit is used to obtain a second variable based on the runtime of the target node in the historical operation process of the target workflow. The second variable can characterize the changing trend of the runtime of the target node in the historical operation process. The acquisition unit includes: obtaining a partial autocorrelation coefficient between the runtime of the target node in a first operation process and the runtime of the target node in at least one second operation process, wherein the first operation process is the most recent operation process of the target workflow, and the second operation process is the Nth operation process before the first operation process, where N is a positive integer greater than or equal to 1; and obtaining the second variable based at least on the partial autocorrelation coefficient. A runtime prediction unit is used to obtain the predicted runtime of the target node based at least on the first variable and the second variable, including: substituting the first variable and the second variable into a multiple linear regression model, wherein the multiple linear regression model uses the second variable and the first variable as independent variables and the predicted runtime of the target node as the dependent variable to obtain the predicted runtime of the target node.

Images