Data processing method and device, equipment, storage medium and program product

Abstract

The application discloses a data processing method and device, equipment, storage medium and program product, and relates to the technical field of communication. The application is applied to a cluster service, the cluster service comprises a plurality of node servers, and the method comprises the following steps: acquiring running state information of each node server and inter-node network state information under a preset initial delay time; inputting the running state information of a first node server, the running state information of a second node server and the inter-node network state information of the first node server and the second node server into a preset training model, and outputting to obtain a data prediction transmission time of the first node server, wherein the training model is obtained by training the running state information of a plurality of source node servers, the running state information of each processing node server and the inter-node network state information; and adjusting the initial delay time of the first node server according to the data prediction transmission time, and determining a window delay time of the first node server.

Description

Technical Field

[0001] This application belongs to the field of communication technology, and in particular relates to a data processing method, apparatus, device, storage medium and program product. Background Technology

[0002] With the development and popularization of the internet industry, various data sources are constantly emerging. At the same time, massive amounts of data contain enormous value, urgently requiring enterprises to collect, analyze, and mine it. In the process of processing massive real-time data, streaming data is usually processed in chronological order. However, due to network latency, lack of computing resources, and other reasons, streaming data may experience delays and out-of-order processing. Existing technologies use a window + watermark approach to achieve data real-time performance and integrity. However, the watermark is determined by the window delay time; setting the window delay time too large will affect the real-time performance, while setting it too small will affect the integrity. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, storage medium, and program product that can dynamically adjust the window delay time, meet the transmission requirements of different streaming data, and ensure the real-time performance and integrity of data transmission.

[0004] In a first aspect, embodiments of this application provide a data processing method applied to a cluster service, the cluster service including multiple node servers, the method comprising:

[0005] Obtain the operating status information of each node server and the network status information between nodes under the preset initial delay time;

[0006] The running status information of the first node server, the running status information of the second node server, and the inter-node network status information between the first node server and the second node server are input into a preset training model, and the predicted data transmission time of the first node server is output. The first node server is a node server that sends data among the plurality of node servers, and the second node server is a node server that receives data sent by the first node server. The training model is trained from the running status information of the plurality of source node servers, the running status information of the corresponding processing node server of each source node server, and the inter-node network status information between each source node server and the corresponding processing node server.

[0007] Based on the data, the transmission time is predicted, the initial delay time of the first node server is adjusted, and the window delay time of the first node server is determined.

[0008] Secondly, embodiments of this application provide a data processing apparatus applied to a cluster service, the cluster service including multiple node servers, the apparatus comprising:

[0009] The first acquisition module is used to acquire the running status information of each node server and the network status information between nodes under a preset initial delay time.

[0010] The processing module is used to input the running status information of the first node server, the running status information of the second node server, and the inter-node network status information between the first node server and the second node server into a preset training model, and output the predicted data transmission time of the first node server. The first node server is a node server that sends data among the plurality of node servers, and the second node server is a node server that receives data sent by the first node server. The training model is trained from the running status information of the plurality of source node servers, the running status information of the corresponding processing node server of each source node server, and the inter-node network status information between each source node server and the corresponding processing node server.

[0011] An adjustment module is used to predict the transmission time based on the data, adjust the initial delay time of the first node server, and determine the window delay time of the first node server.

[0012] Thirdly, embodiments of this application provide an electronic device, the device including: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the data processing method described in any of the above.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the data processing method described in any of the above claims.

[0014] Fifthly, embodiments of this application provide a computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the data processing method described in any of the above claims.

[0015] The data processing method, apparatus, device, storage medium, and program product of this application embodiment can acquire the operating status information and inter-node network status information of each node server in a cluster service under a preset initial delay time; and input the operating status information of the first node server, the operating status information of the second node server, and the inter-node network status information of the first node server and the second node server into a preset training model, and output the predicted data transmission time of the first node server. The first node server is a node server that sends data among multiple node servers, and the second node server is a node server that receives data sent by the first node server. The training model is trained from the operating status information of multiple source node servers, the operating status information of the corresponding processing node server of each source node server, and the inter-node network status information between each source node server and the corresponding processing node server; finally, the initial delay time of the first node server is adjusted according to the predicted data transmission time to determine the window delay time of the first node server. In this embodiment, the predicted data transmission time of the first node server can be predicted based on the operating status information of the first node server, the operating status information of the second node server, and the network status information between the first node server and the second node server. The window delay time of the first node server can be dynamically adjusted in this way, so that the window delay time is not set too large or too small, which can meet the transmission requirements of different streams of data and ensure the real-time performance and integrity of data transmission. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a window and a water level line provided in an embodiment of this application;

[0018] Figure 2 This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of a real-time streaming data fault tolerance method provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0021] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0023] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0024] With the development and popularization of the internet industry, various data sources are constantly emerging. At the same time, massive amounts of data contain enormous value, urgently requiring enterprises to collect, analyze, and mine it. In the process of processing massive real-time data, streaming data is usually processed sequentially. However, due to network latency, lack of computing resources, and other reasons, streaming data may experience delays and out-of-order processing. Existing technologies use a window + watermark approach to achieve data real-time performance and integrity, such as... Figure 1 As shown, assuming the system's allowed window delay time is set to 3 minutes, when event D arrives at the window, since this event time is the current maximum event time entering the window, the waterline = 9:11 - 3 minutes = 9:08. The waterline is within the window, so window calculation is not triggered. When event C arrives, event C is not the current maximum event time, so window calculation is not triggered. When time B arrives, since time B is the current maximum time, the waterline = 9:15 - 3 minutes = 9:12. At this time, the waterline is outside the window, satisfying the window calculation triggering rule: waterline >= window end time. The window immediately triggers calculation, event A is discarded, and the window is destroyed. Through the above analysis, the waterline is determined by the window delay time. Setting the window delay time too large will affect the real-time performance of the data, while setting it too small will affect its integrity.

[0025] To address the problems of the prior art, embodiments of this application provide a data processing method, apparatus, device, storage medium, and program product. The data processing method provided in this application embodiment will be described first below.

[0026] Figure 2 A schematic flowchart of a data processing method provided in one embodiment of this application is shown. Figure 2 As shown, a data processing method is applied to a cluster service, which may include multiple node servers. The method may include the following steps S201 to S203:

[0027] S201. Obtain the running status information of each node server and the network status information between nodes under the preset initial delay time;

[0028] S202. Input the running status information of the first node server, the running status information of the second node server, and the inter-node network status information of the first node server and the second node server into the preset training model, and output the predicted data transmission time of the first node server. The first node server is a node server that sends data among multiple node servers, and the second node server is a node server that receives data sent by the first node server. The training model is trained from the running status information of multiple source node servers, the running status information of the corresponding processing node server of each source node server, and the inter-node network status information between each source node server and the corresponding processing node server.

[0029] S203. Based on the data prediction transmission time, adjust the initial delay time of the first node server and determine the window delay time of the first node server.

[0030] The data processing method of this application embodiment can obtain the operating status information and inter-node network status information of each node server in the cluster service under a preset initial delay time; and input the operating status information of the first node server, the operating status information of the second node server, and the inter-node network status information of the first node server and the second node server into a preset training model, and output the predicted data transmission time of the first node server. The first node server is a node server that sends data among multiple node servers, and the second node server is a node server that receives data sent by the first node server. The training model is trained from the operating status information of multiple source node servers, the operating status information of the corresponding processing node server of each source node server, and the inter-node network status information between each source node server and the corresponding processing node server; finally, the initial delay time of the first node server is adjusted according to the predicted data transmission time to determine the window delay time of the first node server. In this embodiment, the predicted data transmission time of the first node server can be predicted based on the operating status information of the first node server, the operating status information of the second node server, and the network status information between the first node server and the second node server. The window delay time of the first node server can be dynamically adjusted in this way, so that the window delay time is not set too large or too small, which can meet the transmission requirements of different streams of data and ensure the real-time performance and integrity of data transmission.

[0031] In S201, the aforementioned operating status information may include information such as CPU usage, memory usage, hard disk usage, and network bandwidth.

[0032] The above-mentioned network status information between nodes refers to the network status information between the source node server and the corresponding processing node server.

[0033] The above-mentioned acquisition of the operating status information of each node server and the network status information between nodes under the preset initial delay time can be exemplarily obtained through monitoring software such as Zabbix and Nagios.

[0034] In S202, the first node server mentioned above is one of the multiple node servers that sends data.

[0035] The second node server mentioned above is the node server that receives data sent by the first node server.

[0036] The above-mentioned data prediction transmission time is the data transmission time predicted by the preset training model based on the operating status information of the first node server, the operating status information of the second node server, and the network status information between the first node server and the second node server.

[0037] The above training model is trained using the running status information of multiple source node servers, the running status information of the corresponding processing node servers of each source node server, and the inter-node network status information between each source node server and its corresponding processing node server.

[0038] In S203, the above-mentioned adjustment of the initial delay time of the first node server based on the predicted data transmission time and the determination of the window delay time of the first node server can, for example, be based on the predicted data transmission time and the following formula, to adjust the initial delay time of the first node server and determine the window delay time of the first node server.

[0039] td1=td0+(tp0-td0)*υ

[0040] Where td0 is the initial delay time of the first node server, tp0 is the predicted data transmission time, υ is the preset movement step size, and td1 is the window delay time of the first node server.

[0041] As one implementation of this application, in order to accurately predict the data transmission time of the first node server, the method may further include the following before S202:

[0042] Obtain a data sample set, which includes multiple data samples. Each data sample includes the running status information of each source node server, the running status information of the corresponding processing node server of each source node server, the inter-node network status information between each source node server and the corresponding processing node server, and the data transmission time.

[0043] The running status information of each source node server, the running status information of each processing node server, and the network status information between each node in multiple data samples are used as independent variables of the preset model, and the data transmission time is used as the dependent variable of the preset model to train the preset model.

[0044] A training model is generated when the parameter error of the preset model meets the preset conditions.

[0045] The aforementioned data sample set includes multiple data samples. Each data sample includes the operational status information of each source node server, the operational status information of the corresponding processing node server, the inter-node network status information between each source node server and its corresponding processing node server, and the data transmission time. The source node server is the node server that sends data, and the processing node server is the node server that receives data sent by the source node server.

[0046] The aforementioned data sample set, for example, may be as shown below.

[0047]

[0048] The above uses the running status information of each source node server, each processing node server, and the network status information between nodes in multiple data samples as independent variables of the preset model, and the data transmission time as the dependent variable of the preset model, to train the preset model. For example, the data sample set can have n data samples, and the data sample set can be represented as X, where x nm Let λ represent the m-th independent variable of the n-th sample, and let λ be the parameter of the preset model.

[0049]

[0050] Based on the operational status information of the source node server, the operational status information of the processing node server, and the network status information between the two nodes, which show a strong correlation with data transmission time, an objective function is constructed between the independent and dependent variables. Assume the independent variable x = {x1, x2, ..., x...} m The predicted value of the dependent variable is} The objective function is as follows:

[0051]

[0052] Model predictions The calculation method is the product of the independent variable matrix X and λ, as follows:

[0053]

[0054] The parameter error of the aforementioned preset model satisfies a preset condition, for example, when the sample's true value y differs from the model's predicted value. The model is optimal when the time-varying error is minimized. Find the minimum value of J(λ).

[0055]

[0056] J(λ)=(yX·λ) T (yX·λ)

[0057] Taking the derivative of J(λ) such that Finally, λ is calculated using the following formula to form the training model.

[0058] λ=(X T X) -1 X T y............ (Formula 6)

[0059] In this embodiment, the running status information of the source node server, the running status information of the processing node server, and the network status information between the two nodes are strongly correlated with the data transmission time. The running status information of each source node server, the running status information of each processing node server, and the network status information between the nodes are used as independent variables of the preset model, and the data transmission time is used as the dependent variable of the preset model. The training model is then trained, which can accurately predict the data transmission time of the first node server.

[0060] As another implementation of this application, in order to prevent overfitting during model training, the aforementioned runtime status information may include CPU utilization, memory utilization, hard disk utilization, and network bandwidth. Before training the preset model by using the runtime status information of each source node server, each processing node server, and the network status information between nodes as independent variables of the preset model, and each data transmission time as the dependent variable of the preset model, the above method may further include:

[0061] Calculate the correlation coefficient between any two parameters from the first CPU utilization, first memory utilization, first hard disk utilization, and first network bandwidth of each source node server, the second CPU utilization, second memory utilization, second hard disk utilization, and second network bandwidth of each processing node server, and the network status information between each node.

[0062] If the correlation coefficient is greater than the preset first threshold, either of the two parameters corresponding to the correlation coefficient will be determined as the target parameter.

[0063] The above-mentioned method uses the operational status information of each source node server, each processing node server, and the network status information between nodes in multiple data samples as independent variables of a preset model, and the data transmission time as the dependent variable of the preset model, to train the preset model. Specifically, this may include:

[0064] The target parameters in the preset model are used as independent variables of the first CPU utilization, first memory utilization, first hard disk utilization, and first network bandwidth of each source node server, the second CPU utilization, second memory utilization, second hard disk utilization, and second network bandwidth of each processing node server, and the network status information between each node. The data transmission time is used as the dependent variable of the preset model. The preset model is then trained.

[0065] The correlation coefficient between any two parameters is calculated from the following: the first CPU utilization, first memory utilization, first hard disk utilization, and first network bandwidth of each source node server; the second CPU utilization, second memory utilization, second hard disk utilization, and second network bandwidth of each processing node server; and the network status information between nodes. For example, p and q are any two independent variables from the following: the first CPU utilization, first memory utilization, first hard disk utilization, and first network bandwidth of each source node server; the second CPU utilization, second memory utilization, second hard disk utilization, and second network bandwidth of each processing node server; and the network status information between nodes. The correlation coefficient ρ is calculated using the following formula. pq :

[0066]

[0067] Where D(p) is the variance of the p-th independent variable, D(q) is the variance of the q-th independent variable, and Cov(p,q) is the covariance matrix between the p-th and q-th parameters.

[0068] The aforementioned preset first threshold can be, for example, 0.9. Of course, the first threshold in this embodiment is not limited to this and can be set according to the user's actual needs. No specific limitation is made here.

[0069] In this embodiment, the correlation coefficient between any two parameters is calculated from the first CPU utilization, first memory utilization, first hard disk utilization, and first network bandwidth of each source node server, the second CPU utilization, second memory utilization, second hard disk utilization, and second network bandwidth of each processing node server, and the network status information between each node. When the correlation coefficient is greater than a preset first threshold, either parameter is determined as the target parameter. The larger the correlation coefficient, the stronger the correlation between the two parameters. Then, the model is trained on the preset model according to the target parameter to prevent overfitting during model training.

[0070] In some embodiments, the above-described S203 may specifically include:

[0071] Based on the predicted data transmission time and the following formula, adjust the initial latency of the first node server to determine its window latency.

[0072] td1=td0+(tp0-td0)*υ

[0073] Where td0 is the initial delay time of the first node server, tp0 is the predicted data transmission time, υ is the preset movement step size, and td1 is the window delay time of the first node server.

[0074] In this embodiment, when the predicted value tp0 is greater than the existing window delay time td0, it indicates that the current data source, data processing node, and network are in poor operating condition. Appropriately extending the window delay time td0 improves the integrity of the streaming data. When the predicted value tp0 is less than the existing window delay time td0, it indicates that the current data source, data processing node, and network are in good operating condition. Appropriately reducing the window delay time td0 improves data real-time performance while ensuring integrity.

[0075] As another implementation of this application, in order to improve fault tolerance and recovery efficiency, the above method may further include:

[0076] Get the average data transmission time of multiple first-node servers in the cluster service within the i-th window and the average transmission time of the associated k-1 historical data. The k-1 average data transmission time is the data of the first-node server in the k-1 windows before the i-th window, where i is a positive integer and k is a positive integer greater than 1.

[0077] Sort the average data transmission time of multiple first-node servers within the i-th window to obtain a sorted queue;

[0078] Based on the average data transmission time of each first node server in the i-th window and the average transmission time of k-1 historical data, the normal distribution function of each first node server is constructed. The normal distribution function is used to characterize the probability density of different average data transmission times taking the target value.

[0079] If the average data transmission time of the first node server in the i-th window is within the preset range of the sorting queue, or if the probability density of the average data transmission time in the i-th window of the normal distribution function taking the target value is less than the preset second threshold, the first node server is determined to be a potential risk node server.

[0080] Obtain the average transmission time of j data points for each potential risk node server within the (i+1)th to (i+j)th window, where j is a positive integer greater than or equal to 1;

[0081] If the average transmission time of j data points on a potential risk node server is within a preset range of the sorting queue, and the probability density of the average transmission time of each data point in the normal distribution function is less than a preset second threshold, then the potential risk node server is identified as a fault risk node server, and the data of the fault risk node server and the corresponding second node server are backed up.

[0082] The preset interval of the above sorting queue can be, for example, an interval α (0 < α < 1) after all data sources in the queuing sequence from smallest to largest. Here, α can be set according to the actual needs of the user and is not specifically limited here.

[0083] If the average data transmission time of the first node server within the i-th window is within the preset range of the sorting queue, or if the probability density of the average data transmission time within the i-th window in the normal distribution function being less than the preset second threshold, it means that the first node server has potential abnormal risks.

[0084] The j above is a positive integer greater than or equal to 1. For example, it can be 3 or 5, without any specific limitation.

[0085] The above-mentioned situation, where the average transmission time of j data points on the potential risk node server is within the preset range of the sorting queue, and the probability density of the average transmission time of each data point in the normal distribution function is less than the preset second threshold, means that the potential risk node server is likely to have a failure risk.

[0086] In this embodiment, potential anomaly risks are identified by the average data transmission time of the first node server in the i-th window and the average transmission time of the associated k-1 historical data. If the average data transmission time of the first node server in the i-th window is within a preset interval of the sorting queue, or the probability density of the average data transmission time in the i-th window of the normal distribution function taking the target value is less than a preset second threshold, the failure risks are further identified based on the average data transmission times of the j data of the potential risk node server in the i+1 to i+j windows. This enables micro-backup before a failure occurs, thereby improving fault tolerance and recovery efficiency.

[0087] As another implementation of this application, in order to improve fault tolerance and recovery efficiency, the above method may further include:

[0088] Obtain multiple trend coefficients for each second node server in the cluster service within multiple consecutive windows. Each trend coefficient is used to characterize the changing trend of the average data transmission time of each second node server within the corresponding window.

[0089] If multiple trend coefficients of each second node server are greater than a preset third threshold, the second node server is identified as a high-risk second node server, and the data of the high-risk second node server and the corresponding first node server are backed up.

[0090] The aforementioned preset third threshold can be 0, for example. However, in this embodiment, it is not limited to this and can also be set according to the user's actual needs. No specific limitation is made here.

[0091] The aforementioned trend coefficient can be used to characterize the changing trend of the average data transmission time of the second node server within the corresponding window. For example, if the trend coefficient is greater than 0, it indicates that the data transmission time of the current data processing node is increasing. If the trend coefficient is less than 0, it indicates that the data transmission time of the current data processing node is decreasing.

[0092] In this embodiment of the application, by obtaining multiple trend coefficients of each second node server in the cluster service within multiple consecutive windows, and determining that the multiple trend coefficients of the second node server are all greater than a preset third threshold, when the trend coefficients indicate that the average data transmission time of the second node server increases continuously multiple times, it indicates that the second node server has a high probability of failure, thus enabling early micro-backup and improving fault tolerance and recovery efficiency.

[0093] In some embodiments, obtaining multiple trend coefficients of each second node server in the cluster service within multiple consecutive windows may specifically include:

[0094] Obtain the average data transmission time of each second node server in the cluster service in the nth window and the average transmission time of m-1 historical data. The average transmission time of m-1 historical data is the average transmission time of historical data of the second node server in the m-1 windows before the nth window, where n is a positive integer and m is a positive integer greater than 1.

[0095] Based on the average data transmission time of each second-node server in the nth window and the average transmission time of m-1 historical data, the trend coefficient of each second-node server in the nth window is calculated according to the following formula.

[0096]

[0097] Where G is the trend coefficient. Let w represent the average data transmission time of the nth window and the average transmission time of m-1 historical data, and w represent the second node server.

[0098] In this embodiment, the trend coefficient of the second node server in the nth window can be accurately obtained by using the average data transmission time of the second node server in the nth window and the average transmission time of m-1 historical data, thereby improving the accuracy of identifying the fault risk of the second node server.

[0099] To facilitate understanding of the data processing method in the embodiments of this application, the actual application process of this data processing method is described as follows:

[0100] (I) A method for dynamically adjusting window delay time based on data source differences

[0101] This application addresses the shortcomings of lacking scientific evaluation settings for window delay time, which prevents adaptive adjustment based on the real-time operating status of the data source. It proposes a method for dynamically adjusting the window delay time based on data source differences. The specific method is as follows:

[0102] 1.1 Choosing independent and dependent variables

[0103] Using the running status of the data source, the data processing node, and the network status between them as independent variables, and the data transmission time (i.e., the event transmission time below) as the dependent variable, the model is trained separately for each data source and data processing node, and the window delay time of each data source is set differently.

[0104] (1) First, the node's operating status is obtained through monitoring software, as shown in Table 1:

[0105] Table 1: Training Data Samples

[0106]

[0107]

[0108] (2) To prevent overfitting during model training, a correlation analysis was performed on any two parameters in the data in Table 1, and the correlation coefficient of any parameter was calculated. Assuming p and q are any two independent variables, as shown in Formula 1:

[0109]

[0110] In the expression in Table 1, D(p) is the variance of the p-th independent variable, D(q) is the variance of the q-th independent variable, and Cov(p,q) is the covariance matrix between the p-th and q-th parameters.

[0111] The correlation between any two parameters is calculated using Formula 1, and the correlation coefficient ρ is... pq Randomly select one of the parameters greater than 0.9, ρ pq The larger the value, the stronger the correlation between the parameters. Select m independent variables from the 9 parameters (equivalent to the target parameters mentioned above).

[0112] 1.2 Training the Model

[0113] This application posits a strong correlation between the operational status of the data source, the operational status of the data processing nodes, and the network status between them, and the time transmission time. Based on this, an objective function is constructed between the independent and dependent variables, assuming the independent variable x = {x1, x2, ..., x...} m The predicted value of the dependent variable is} The objective function is shown in Formula 2:

[0114]

[0115] Suppose the model is trained with n data samples, and the set of data samples can be represented as X, where x nm Let λ represent the m-th independent variable of the n-th sample, and let λ be the model parameter.

[0116]

[0117] Model predictions The calculation method is the product of the independent variable matrix X and λ, as shown in Formula 3:

[0118]

[0119] The model is optimal when the error between the true sample value y and the model's predicted value is minimized; this can be transformed into finding the minimum value of J(λ).

[0120]

[0121] J(λ)=(yX·λ) T (yX·λ)........... (Formula 5)

[0122] Taking the derivative of J(λ) such that λ is calculated using Formula 6 to form a model.

[0123] λ=(X T X) -1 X T y............. (Formula 6)

[0124] 1.3 Predicted data arrival time

[0125] Substituting the running status of the current window's data source (equivalent to the first node server mentioned above), the running status of the data processing node (equivalent to the second node server mentioned above), and the network situation between the two into Formula 3, we can obtain the predicted data transmission time.

[0126] 1.4 Adaptive adjustment of window delay time

[0127] Assuming the initial window delay (equivalent to the initial delay time mentioned above) is td0, the predicted value based on the current state is tp0, and the step size is υ (which can be set by the user). When the predicted value is greater than the existing window delay time, it indicates that the current data source, data processing node, and network are in poor operating condition. The window delay time should be appropriately extended to improve the integrity of the streaming data. When the predicted value is less than the existing window delay time, it indicates that the current data source, data processing node, and network are in good operating condition. The window delay should be appropriately reduced to improve data real-time performance while ensuring integrity. The specific adaptive iterative algorithm is shown in Formula 7:

[0128] td1=td0+(tp0-td0)*υ......(Formula 7)

[0129] According to Formula 7, the window delay time is adaptively adjusted during the calculation of each window, thereby obtaining the optimal window delay time for the data source.

[0130] 1.5 Differentiated calculation window delay time

[0131] Repeat steps 1, 2, 3, and 4 above. Based on the situation between each data source and data processing node, maintain a separate window delay time for each data source. Different data sources have different window delay times. Customize the window calculation trigger conditions for different data sources to improve the real-time performance and completeness of real-time streaming data processing.

[0132] This application addresses the shortcomings of existing technologies, such as the lack of scientific evaluation in setting window latency and the inability to adaptively adjust window latency based on the real-time operating status of the data source. It proposes a method for dynamically adjusting window latency based on data source differences. First, for different data sources and data processing nodes, the operating status of the data source, data processing node, and the network state between them are used as independent variables, with data transmission time as the dependent variable to train the model. Second, the current status of the data source, data processing node, and the network state between them are substituted into the model to predict the window latency. Finally, based on the prediction results, the window latency for different data sources is dynamically and adaptively adjusted in a differentiated manner, overcoming the shortcomings of traditional methods that cannot adaptively adjust window latency. This ensures that when computational and network resources are sufficient, the window latency is reduced to improve data real-time performance; when computational and network resources are limited, the window latency is appropriately increased to ensure the integrity of the streaming data.

[0133] (II) A method for improving fault tolerance in stream processing using micro-snapshots

[0134] Another requirement for processing massive real-time data is strong fault tolerance, enabling rapid recovery in the event of a failure. Existing methods for fault tolerance in real-time streaming data involve periodically generating distributed snapshot checkpoint markers and inserting them into the data stream during processing. When a data processing node detects a marker, it immediately backs up the distributed snapshot. For example... Figure 3As shown, when the data processing node receives checkpoint n-1, it backs up and generates snapshots of data A, B, and C to the right of checkpoint n-1. When a failure occurs at point E, the data is restored to the state at checkpoint n-1, and incremental processing is performed based on this snapshot state, reprocessing D, E, and F. However, this method of periodically backing up distributed snapshots has limitations. If the backup cycle is too long, the fault tolerance and recovery efficiency is low; if the backup cycle is too short, it wastes computing power and storage resources. It lacks a method that analyzes potential failures based on the data source and the operating trends of the data processing nodes, and performs advance backups to improve fault tolerance efficiency.

[0135] Existing fault tolerance solutions based on periodic distributed snapshot backups suffer from drawbacks: excessively long backup cycles lead to low fault recovery efficiency, while excessively short cycles waste computing and storage resources. Furthermore, there's a lack of methods to analyze potential failures and proactively back them up based on data source and data processing node operating trends. This application proposes a fault tolerance method combining periodic backups and micro-snapshots to improve fault recovery efficiency. In stream processing, data transmission and processing times typically become abnormal before a task node fails. To address this, this application detects anomalies at both the data source and data processing node levels. When triggering conditions are met, a micro-backup is performed before the failure occurs, thereby improving fault recovery efficiency. The specific anomaly detection method is as follows:

[0136] 2.1 Data Source Anomaly Detection

[0137] In terms of data source anomaly detection, two dimensions are used for evaluation. First, comparing data transmission times across multiple data sources: when the average transmission time of a particular data source (equivalent to the first node server mentioned above) ranks after all data sources by α (0 < α < 1), it indicates a potential anomaly risk. Second, comparing data transmission times within a data source's time-series window: when a data transmission time within a certain window is compared to the previous k-1 windows and falls within the tail of the normal distribution by β (0 < β < 1), it also indicates a potential anomaly risk. Combining these two dimensions, if a data source ranks after α in the inter-data source comparison and simultaneously falls within the tail of the normal distribution by β in the previous k-1 time-series window comparison, and this occurs for three consecutive windows, it indicates a high probability of a fault risk, prompting an early micro-snapshot backup. The specific process is as follows:

[0138] (1) Assume the system has l data sources, and the average data transmission time in the i-th window is tm. i ={tm i1 ,tm i2 ,...,tm il When the data transmission time ranks after α, it carries a potential risk of anomalies.

[0139] (2) Assume a data source j, and the average data transmission time between time window i and the previous k-1 windows is expressed as: According to tr jConstruct a normal distribution function, when P(tr) j When > γ) < β, it indicates that the data source has potential anomaly risks.

[0140] In summary, when a data source is ranked last α in inter-data source comparisons and is located at the tail β of the normal distribution when compared with its own k-1 time series windows, this situation occurs continuously in three windows, indicating that the data source is likely to fail, and a micro-snapshot backup should be performed in advance.

[0141] 2.2 Data Processing Node Anomaly Detection

[0142] Regarding fault detection in data processing nodes, if the data transmission time increases continuously for η times within the past m-1 window, it indicates that the data processing node has a certain probability of failure. The specific detection method is as follows:

[0143] Assume that for a certain data source w, the average data transmission time for the m-1 windows prior to time window n is...

[0144]

[0145] Assumption Where n≥s>d>nm

[0146] calculate

[0147]

[0148] If G > 0, it indicates that the data transmission time trend of the current data processing node is increasing.

[0149] If G < 0, it indicates that the data transmission time of the current data processing node is decreasing.

[0150] When the data transmission time increases consecutively for η (adjustable) times, it indicates that the data processing node has a high probability of failure. Early micro-backup can improve fault tolerance and recovery efficiency.

[0151] In this embodiment, to address the shortcomings of existing real-time data fault tolerance methods, such as low efficiency and lack of a mechanism to analyze potential faults and back them up in advance based on the operating trends of data sources and data processing nodes, a fault tolerance method combining periodic snapshots and micro-snapshots is proposed. Regarding data source detection, on the one hand, a comparison is made within its own time-series window; if the data transmission time ranks late in the normal distribution (β), the data source is suspected of being abnormal. On the other hand, a horizontal comparison is made between data sources; if the data transmission time ranks late in the comparison between data sources (α), an anomaly is suspected. When both conditions are met simultaneously for three consecutive windows, a micro-backup is performed in advance. Regarding data processing node anomaly detection, if the data transmission time trend increases for η consecutive times within the past m-1 windows, it indicates a possible anomaly in the data processing node; when the trend increases for η consecutive times, a micro-backup is performed in advance. By combining periodic snapshot backups with fault prediction-based micro-backups, the fault tolerance and recovery efficiency can be significantly improved.

[0152] Based on the data processing method provided in the above embodiments, this application also provides specific implementations of a data processing apparatus. Please refer to the following embodiments.

[0153] like Figure 4 As shown, the data processing device 400 provided in this application embodiment is applied to a cluster service, which includes multiple node servers. The device 400 may include the following modules: a first acquisition module 401, a processing module 402, and an adjustment module 403.

[0154] The first acquisition module 401 is used to acquire the running status information of each node server and the network status information between nodes under the preset initial delay time.

[0155] The processing module 402 is used to input the running status information of the first node server, the running status information of the second node server, and the inter-node network status information of the first node server and the second node server into a preset training model, and output the predicted data transmission time of the first node server. The first node server is a node server that sends data among multiple node servers, and the second node server is a node server that receives data sent by the first node server. The training model is trained from the running status information of multiple source node servers, the running status information of the corresponding processing node server of each source node server, and the inter-node network status information between each source node server and the corresponding processing node server.

[0156] The adjustment module 403 is used to adjust the initial delay time of the first node server based on the data prediction transmission time and to determine the window delay time of the first node server.

[0157] The data processing apparatus of this application embodiment can acquire the operating status information and inter-node network status information of each node server in the cluster service under a preset initial delay time; and input the operating status information of the first node server, the operating status information of the second node server, and the inter-node network status information of the first node server and the second node server into a preset training model, and output the predicted data transmission time of the first node server. The first node server is a node server that sends data among multiple node servers, and the second node server is a node server that receives data sent by the first node server. The training model is trained from the operating status information of multiple source node servers, the operating status information of the corresponding processing node server of each source node server, and the inter-node network status information between each source node server and the corresponding processing node server; finally, the initial delay time of the first node server is adjusted according to the predicted data transmission time to determine the window delay time of the first node server. In this embodiment, the predicted data transmission time of the first node server can be predicted based on the operating status information of the first node server, the operating status information of the second node server, and the network status information between the first node server and the second node server. The window delay time of the first node server can be dynamically adjusted in this way, so that the window delay time is not set too large or too small, which can meet the transmission requirements of different streams of data and ensure the real-time performance and integrity of data transmission.

[0158] As one implementation of this application, in order to accurately predict the data prediction transmission time of the first node server, the above-mentioned device 400 may further include:

[0159] The second acquisition module is used to acquire a data sample set, which includes multiple data samples. Each data sample includes the running status information of each source node server, the running status information of the corresponding processing node server of each source node server, the inter-node network status information between each source node server and the corresponding processing node server, and the data transmission time.

[0160] The training module is used to train the preset model by taking the running status information of each source node server, the running status information of each processing node server, and the network status information between each node in multiple data samples as independent variables of the preset model, and the data transmission time as the dependent variable of the preset model.

[0161] The generation module is used to generate a training model when the parameter error of the preset model meets the preset conditions.

[0162] As another implementation of this application, in order to prevent overfitting during model training, the aforementioned operating status information may include CPU utilization, memory utilization, hard disk utilization, and network bandwidth. The aforementioned device 400 may also include:

[0163] The calculation module is used to calculate the correlation coefficient between any two parameters from the first CPU utilization, first memory utilization, first hard disk utilization and first network bandwidth of each source node server, the second CPU utilization, second memory utilization, second hard disk utilization and second network bandwidth of each processing node server, and the network status information between each node.

[0164] The first determining module is used to determine either of the two parameters corresponding to the correlation coefficient as the target parameter when the correlation coefficient is greater than a preset first threshold.

[0165] The aforementioned training module is also used to train the preset model by taking the first CPU utilization rate, first memory utilization rate, first hard disk utilization rate and first network bandwidth of each source node server, the second CPU utilization rate, second memory utilization rate, second hard disk utilization rate and second network bandwidth of each processing node server, and the target parameters in the network status information between each node as independent variables of the preset model, and each data transmission time as the dependent variable of the preset model.

[0166] In some embodiments, the adjustment module 403 is specifically used to adjust the initial delay time of the first node server according to the data prediction transmission time and the following formula, and to determine the window delay time of the first node server.

[0167] td1=td0+(tp0-td0)*υ

[0168] Where td0 is the initial delay time of the first node server, tp0 is the predicted data transmission time, υ is the preset movement step size, and td1 is the window delay time of the first node server.

[0169] As another implementation of this application, in order to improve fault tolerance and recovery efficiency, the above-mentioned device 400 may further include:

[0170] The third acquisition module is used to acquire the average data transmission time of multiple first node servers in the cluster service within the i-th window and the average transmission time of k-1 historical data. The average transmission time of k-1 historical data is the average data transmission time of the first node server in the k-1 windows before the i-th window, where i is a positive integer and k is a positive integer greater than 1.

[0171] The sorting module is used to sort the average data transmission time of multiple first-node servers within the i-th window to obtain a sorted queue;

[0172] The construction module is used to construct the normal distribution function of each first node server based on the average data transmission time of each first node server in the i-th window and the average transmission time of k-1 historical data. The normal distribution function is used to characterize the probability density of different average data transmission times taking the target value.

[0173] The second determining module is used to determine the first node server as a potential risk node server when the average data transmission time of the first node server in the i-th window is within a preset interval of the sorting queue, or when the probability density of the average data transmission time in the i-th window of the normal distribution function taking the target value is less than a preset second threshold.

[0174] The fourth acquisition module is used to acquire the average transmission time of j data points of each potential risk node server within the (i+1)th to (i+j)th window, where j is a positive integer greater than or equal to 1;

[0175] The first backup module is used to determine a potential risk node server as a fault risk node server and back up the data of the fault risk node server and the corresponding second node server when the average transmission time of j data of the potential risk node server is within the preset interval of the sorting queue and the probability density of the average transmission time of each data in the j data average transmission time in the normal distribution function is less than the preset second threshold.

[0176] As another implementation of this application, in order to improve fault tolerance and recovery efficiency, the above-mentioned device 400 may further include:

[0177] The fifth acquisition module is used to acquire multiple trend coefficients of each second node server in the cluster service within multiple consecutive windows. Each trend coefficient is used to characterize the changing trend of the average data transmission time of each second node server within the corresponding window.

[0178] The second backup module is used to determine a second node server as a high-risk second node server when multiple trend coefficients of each second node server are greater than a preset third threshold, and to back up the data of the high-risk second node server and the corresponding first node server.

[0179] In some embodiments, the fifth acquisition module described above may specifically include:

[0180] The acquisition unit is used to acquire the average data transmission time of each second node server in the cluster service in the nth window and the average transmission time of m-1 historical data. The average transmission time of m-1 historical data is the average data transmission time of the second node server in the m-1 windows before the nth window, where n is a positive integer and m is a positive integer greater than 1.

[0181] The calculation unit is used to calculate the trend coefficient of each second node server in the nth window based on the average data transmission time of each second node server in the nth window and the average transmission time of m-1 historical data, according to the following formula.

[0182]

[0183] Where G is the trend coefficient. Let w represent the average data transmission time of the nth window and the average transmission time of m-1 historical data, and w represent the second node server.

[0184] Figure 5 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0185] An electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0186] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0187] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0188] In certain embodiments, memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0189] The processor 501 implements any of the data processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0190] In one example, the electronic device may also include a communication interface 503 and a bus 510. Wherein, as... Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0191] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0192] Bus 510 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0193] The electronic device can execute the data processing method described in the embodiments of this application, thereby achieving the combination Figure 2 and Figure 4 The data processing methods and apparatus described.

[0194] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the data processing methods in the above embodiments.

[0195] In conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer program product, in which the instructions of the computer program product, when executed by the processor of an electronic device, cause the electronic device to perform any of the above data processing methods.

[0196] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0197] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0198] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0199] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0200] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data processing method, characterized in that, Applied to a cluster service, the cluster service comprising multiple node servers, the method includes: Obtain the operating status information of each node server and the network status information between nodes under the preset initial delay time; The running status information of the first node server, the running status information of the second node server, and the inter-node network status information between the first node server and the second node server are input into a preset training model, and the predicted data transmission time of the first node server is output. The first node server is a node server that sends data among the plurality of node servers, and the second node server is a node server that receives data sent by the first node server. The training model is trained from the running status information of the plurality of source node servers, the running status information of the corresponding processing node server of each source node server, and the inter-node network status information between each source node server and the corresponding processing node server. Based on the data, the transmission time is predicted, the initial delay time of the first node server is adjusted, and the window delay time of the first node server is determined. Different first node servers correspond to different window delay times. The method further includes: The average data transmission time of multiple first node servers in the cluster service within the i-th window and the average transmission time of k-1 historical data are obtained. The average transmission time of k-1 historical data is the average data transmission time of the first node servers in the k-1 windows before the i-th window, where i is a positive integer and k is a positive integer greater than 1. The average data transmission time of multiple first node servers within the i-th window is sorted to obtain a sorted queue; Based on the average data transmission time of each first node server within the i-th window and the average transmission time of the k-1 historical data, a normal distribution function is constructed for each first node server. The normal distribution function is used to characterize the probability density of different average data transmission times taking the target value. If the average data transmission time of the first node server in the i-th window is within the preset range of the sorting queue, or if the probability density of the average data transmission time in the i-th window of the normal distribution function taking the target value is less than the preset second threshold, the first node server is determined to be a potential risk node server. Obtain the average transmission time of j data points for each of the potential risk node servers within the (i+1)th to (i+j)th windows, where j is a positive integer greater than or equal to 1; If the average transmission time of the j data points of the potential risk node server is within the preset interval of the sorting queue, and the probability density of each of the j data average transmission times in the normal distribution function taking the target value is less than the preset second threshold, then the potential risk node server is determined to be a fault risk node server, and the data of the fault risk node server and the corresponding second node server are backed up.

2. The method according to claim 1, characterized in that, Before inputting the operating status information of the first node server, the operating status information of the second node server, and the inter-node network status information of the first node server and the second node server into a preset training model and outputting the predicted data transmission time of the first node server, the method further includes: Obtain a data sample set, which includes multiple data samples. Each data sample includes the running status information of each source node server, the running status information of the corresponding processing node server of each source node server, the inter-node network status information between each source node server and the corresponding processing node server, and the data transmission time. The operating status information of each source node server, the operating status information of each processing node server, and the network status information between each node in the multiple data samples are used as independent variables of the preset model, and the data transmission time is used as the dependent variable of the preset model to train the preset model. The training model is generated when the parameter error of the preset model meets the preset conditions.

3. The method according to claim 2, characterized in that, The operational status information includes CPU utilization, memory utilization, hard disk utilization, and network bandwidth. Before training the preset model by using the operational status information of each source node server, each processing node server, and the network status information between each node in the plurality of data samples as independent variables of the preset model, and the data transmission time as the dependent variable of the preset model, the method further includes: Calculate the correlation coefficient between any two parameters from the first CPU utilization, first memory utilization, first hard disk utilization, and first network bandwidth of each source node server, the second CPU utilization, second memory utilization, second hard disk utilization, and second network bandwidth of each processing node server, and the network status information between each node. If the correlation coefficient is greater than a preset first threshold, either of the two parameters corresponding to the correlation coefficient will be determined as the target parameter. The step of training the preset model by using the running status information of each source node server, the running status information of each processing node server, and the network status information between each node as independent variables of the preset model, and the data transmission time as the dependent variable of the preset model, includes: The target parameters in the first CPU utilization, first memory utilization, first hard disk utilization, and first network bandwidth of each source node server, the second CPU utilization, second memory utilization, second hard disk utilization, and second network bandwidth of each processing node server, and the network status information between each node are used as independent variables of the preset model, and the data transmission time is used as the dependent variable of the preset model. The preset model is then trained.

4. The method according to claim 1, characterized in that, The step of predicting the transmission time based on the data, adjusting the initial delay time of the first node server, and determining the window delay time of the first node server includes: Based on the predicted transmission time of the data and the following formula, the initial delay time of the first node server is adjusted to determine the window delay time of the first node server. in, The initial latency time for the first node server. Predict the transmission time for the data. The preset movement step size, This is the window delay time for the first node server.

5. The method according to claim 1, characterized in that, The method further includes: Obtain multiple trend coefficients for each of the second node servers in the cluster service within multiple consecutive windows. Each trend coefficient is used to characterize the changing trend of the average data transmission time of each of the second node servers within the corresponding window. If multiple trend coefficients of each of the second node servers are greater than a preset third threshold, the second node server is determined to be a fault risk second node server, and the data of the fault risk second node server and the corresponding first node server are backed up.

6. The method according to claim 5, characterized in that, The step of obtaining multiple trend coefficients for each of the second node servers in the cluster service within multiple consecutive windows includes: The average data transmission time of each second node server in the cluster service in the nth window and the average transmission time of m-1 historical data are obtained. The average transmission time of m-1 historical data is the average data transmission time of the second node server in the m-1 windows before the nth window. n is a positive integer and m is a positive integer greater than 1. Based on the average data transmission time of each second node server in the nth window and the average transmission time of the m-1 historical data, the trend coefficient of each second node server in the nth window is calculated according to the following formula. ， Wherein, G is the trend coefficient. The average data transmission time of the nth window and the average transmission time of the m-1 historical data are denoted by w, where w represents the second node server.

7. A data processing apparatus, characterized in that, Applied to a cluster service, the cluster service including multiple node servers, the device includes: The first acquisition module is used to acquire the running status information of each node server and the network status information between nodes under a preset initial delay time. The processing module is used to input the running status information of the first node server, the running status information of the second node server, and the inter-node network status information between the first node server and the second node server into a preset training model, and output the predicted data transmission time of the first node server. The first node server is a node server that sends data among the plurality of node servers, and the second node server is a node server that receives data sent by the first node server. The training model is trained from the running status information of the plurality of source node servers, the running status information of the corresponding processing node server of each source node server, and the inter-node network status information between each source node server and the corresponding processing node server. The adjustment module is used to predict the transmission time based on the data, adjust the initial delay time of the first node server, and determine the window delay time of the first node server. Different first node servers correspond to different window delay times. The device further includes: The third acquisition module is used to acquire the average data transmission time of multiple first node servers in the cluster service within the i-th window and the average transmission time of k-1 historical data. The average transmission time of k-1 historical data is the average data transmission time of the first node servers in the k-1 windows before the i-th window, where i is a positive integer and k is a positive integer greater than 1. The sorting module is used to sort the average data transmission time of multiple first node servers within the i-th window to obtain a sorted queue; The construction module is used to construct a normal distribution function for each of the first node servers based on the average data transmission time of each first node server in the i-th window and the average transmission time of the k-1 historical data. The normal distribution function is used to characterize the probability density of different average data transmission times taking the target value. The second determining module is used to determine the first node server as a potential risk node server when the average data transmission time of the first node server in the i-th window is within a preset interval of the sorting queue, or when the probability density of the average data transmission time in the i-th window of the normal distribution function taking the target value is less than a preset second threshold. The fourth acquisition module is used to acquire the average transmission time of j data points of each potential risk node server within the (i+1)th to (i+j)th window, where j is a positive integer greater than or equal to 1; The first backup module is used to determine the potential risk node server as a fault risk node server and back up the data of the fault risk node server and the corresponding second node server when the average transmission time of the j data of the potential risk node server is within a preset interval of the sorting queue and the probability density of each of the average transmission times of the j data in the normal distribution function taking the target value is less than the preset second threshold.

8. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the data processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the data processing method as described in any one of claims 1-6.

Images