Data processing method and device of server cluster, processor and electronic equipment

Abstract

The application discloses a data processing method and device of a server cluster, a processor and electronic equipment. It relates to the field of financial technology, and the method comprises the following steps: obtaining a plurality of jobs executed by a plurality of servers in the server cluster; clustering the plurality of jobs according to a similarity set of any two jobs in the plurality of jobs to obtain a plurality of clustering clusters; creating a linear regression model according to the historical running duration of a target job and the historical running duration of a plurality of similar jobs in the clustering cluster to which the target job belongs; and predicting the running duration of the target job of at least one server in the server cluster according to the linear regression model. Through the application, the linear regression model in the related art is created by using full data, the correlation between the full data is poor, and the linear regression model has the problem of low accuracy.

Description

Technical Field

[0001] This application relates to the field of financial technology, and more specifically, to a data processing method, apparatus, processor, and electronic device for a server cluster. Background Technology

[0002] A server cluster consists of multiple servers that work together to provide the same service, appearing to the client as a single server. Given the working principle of server clusters, load balancing is typically required. While load balancing algorithms exist, their calculations often rely solely on time-based assessments of server load, leading to inaccurate results and impacting load balancing effectiveness. For example, a linear regression model can be used to determine server load. Training this model typically utilizes the entire dataset of servers within the cluster; however, the poor correlation between these datasets results in insufficient accuracy for the trained linear regression model.

[0003] The linear regression model in related technologies is created using the full dataset. However, the poor correlation between the full datasets leads to low accuracy in the linear regression model, and no effective solution has yet been proposed. Summary of the Invention

[0004] The main objective of this application is to provide a data processing method, apparatus, processor, and electronic device for server clusters, in order to solve the problem that linear regression models in related technologies are created using full datasets, and the poor correlation between full datasets leads to low accuracy in linear regression models.

[0005] To achieve the above objectives, according to one aspect of this application, a data processing method for a server cluster is provided, comprising: acquiring multiple jobs executed by multiple servers in the server cluster; clustering the multiple jobs according to the similarity set of any two jobs to obtain multiple clusters; creating a linear regression model based on the historical runtime of a target job and the historical runtime of multiple similar jobs in the same cluster to which the target job belongs; and predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model.

[0006] Optionally, before clustering the multiple tasks based on the similarity set of any two tasks among the multiple tasks to obtain multiple clusters, the method further includes: determining the similarity value of the two tasks based on the scores of any two tasks in a preset time unit; normalizing the similarity value to obtain the intermediate similarity value corresponding to the similarity value within the preset time unit; determining the similarity of the two tasks based on the number of time units within the preset time range and the intermediate similarity value of multiple time units, wherein both tasks are run within the time unit; and determining the similarity set of any two tasks among the multiple tasks by traversing any two tasks among the multiple tasks.

[0007] Optionally, the two tasks include a first task and a second task. Determining the similarity value of the two tasks based on the scores of any two tasks in a preset time unit includes: determining a first score of the first task in the preset time unit and a second score of the second task in the preset time unit; determining the similarity Hamming distance between the first task and the second task based on the first score and the second score, wherein the similarity Hamming distance is the similarity value between the first task and the second task.

[0008] Optionally, determining the first score of the first task within the preset time unit and the second score of the second task within the preset time unit includes: using the ratio of the number of times the first task's running time reaches the standard running time within the time unit to the total number of times the first task is run within the time unit as the first score of the first task's time; and using the ratio of the number of times the second task's running time reaches the standard running time within the time unit to the total number of times the second task is run within the time unit as the second score of the second task's time.

[0009] Optionally, normalizing the similar values ​​to obtain intermediate similar values ​​within the preset time unit includes: traversing any two jobs among the plurality of jobs to obtain a set of similar values, wherein the set of similar values ​​includes the similar values ​​of any two jobs among the plurality of jobs; determining the maximum and minimum values ​​of the similar values ​​based on the set of similar values; and normalizing the similar values ​​based on the maximum, the minimum, and the similar values ​​of the two jobs to obtain intermediate similar values ​​corresponding to the similar values ​​of the two jobs.

[0010] Optionally, determining the similarity between the two tasks based on the number of time units within a preset time range and the intermediate similarity values ​​of the multiple time units includes: traversing any two tasks among the multiple tasks to obtain an intermediate similarity value set, wherein the intermediate similarity value set includes intermediate similarity values ​​corresponding to the similarity values ​​of the two tasks within multiple time units, and the multiple time units include the preset time units; determining intermediate similarity values ​​of multiple time units that satisfy the preset time range based on the intermediate similarity value set; summing the intermediate similarity values ​​of the multiple time units and calculating the ratio with the number of the multiple time units to obtain the similarity between the two tasks.

[0011] Optionally, before predicting the runtime of at least one server in the server cluster for the target job according to the linear regression model, the method further includes: receiving a job request for the target job; selecting a default server from the server cluster to process the job request; if the default server cannot execute the target job, performing the step of predicting the runtime of at least one server in the server cluster for the target job according to the linear regression model; after predicting the runtime of at least one server in the server cluster for the target job according to the linear regression model, the method further includes: determining the load rate of the corresponding server based on the server's runtime; selecting the target server with the lowest load rate among the at least one server and executing the job request.

[0012] To achieve the above objectives, according to another aspect of this application, a data processing apparatus for a server cluster is provided, comprising: an acquisition module for acquiring multiple jobs executed by multiple servers in the server cluster; a clustering module for clustering the multiple jobs according to the similarity set of any two jobs in the multiple jobs to obtain multiple clusters; a creation module for creating a linear regression model based on the historical runtime of a target job and the historical runtime of multiple similar jobs in the same cluster to which the target job belongs; and a prediction module for predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model.

[0013] To achieve the above objectives, according to another aspect of this application, a processor is provided for running a program, wherein the program executes the data processing method for a server cluster as described in any one of the above claims.

[0014] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data processing method of the server cluster described in any one of the above.

[0015] This application employs a method that involves obtaining multiple jobs executed by multiple servers in a server cluster; clustering these jobs based on the similarity set of any two jobs to obtain multiple clusters; creating a linear regression model based on the historical runtime of the target job and the historical runtimes of multiple similar jobs within the same cluster; and predicting the runtime of the target job on at least one server in the server cluster using the linear regression model. By clustering different jobs within the server cluster based on the similarity of the jobs, and creating a linear regression model based on the runtime of the target job and the runtimes of similar jobs within the same cluster, the resulting linear regression model can accurately predict the runtime of the target job. This solves the problem in related technologies where linear regression models are created using full datasets, resulting in poor correlation between the full datasets and thus low accuracy. Therefore, this method improves the accuracy of the linear regression model in predicting the runtime of the target job on different servers, and enhances the accuracy of determining server load. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a flowchart of a data processing method for a server cluster provided according to an embodiment of this application;

[0018] Figure 2 This is a flowchart illustrating an optional data processing method for a server cluster according to an embodiment of the present invention.

[0019] Figure 3 This is a comparison chart of predicted and actual values ​​of an optional linear regression model according to an embodiment of the present invention;

[0020] Figure 4 This is a performance test diagram of an optional server cluster data processing method according to an embodiment of the present invention;

[0021] Figure 5 This is another performance test diagram of an optional server cluster data processing method according to an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram of a data processing apparatus for a server cluster provided according to an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0028] According to an embodiment of the present invention, an embodiment of a load balancing method for a server cluster is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] Figure 1This is a flowchart of a server cluster data processing method provided according to an embodiment of this application, such as... Figure 1 As shown, this application provides a data processing method for a server cluster, which includes the following steps:

[0030] Step S102: Obtain multiple jobs that have been executed by multiple servers in the server cluster;

[0031] Step S104: Based on the similarity set of any two jobs among the multiple jobs, cluster the multiple jobs to obtain multiple clusters;

[0032] Step S106: Based on the historical runtime of the target job and the historical runtime of multiple similar jobs in the same cluster to which the target job belongs, create a linear regression model;

[0033] Step S108: Predict the runtime of the target job for at least one server in the server cluster based on the linear regression model.

[0034] The above steps involve obtaining multiple jobs executed by multiple servers in a server cluster; clustering these jobs based on the similarity set of any two jobs to obtain multiple clusters; creating a linear regression model based on the historical runtime of the target job and the historical runtimes of similar jobs within the same cluster; and predicting the runtime of the target job on at least one server in the server cluster using the linear regression model. By clustering different jobs in the server cluster based on the similarity of the jobs, and creating a linear regression model based on the runtime of the target job and the runtimes of similar jobs in the same cluster, the resulting linear regression model can accurately predict the runtime of the target job. This solves the problem of low accuracy in linear regression models created using full datasets, where poor correlation between datasets leads to low accuracy. Therefore, this improves the accuracy of the linear regression model in predicting the runtime of the target job on different servers and enhances the accuracy of determining server load.

[0035] The entity executing the above steps can be a load balancer, a third-party server, an application system, or other device with data processing capabilities that can provide services to a server cluster. This device can be used to create a linear regression model, and then determine the runtime of the target job on each server in the server cluster based on the linear regression model, thereby determining the server load rate. Based on the load rate, a suitable server is selected to process the target job. Optionally, in this embodiment, the entity executing the above steps can be a load balancer.

[0036] The aforementioned server cluster can be an ETL scheduling server cluster. ETL scheduling servers are used to import data into the data lake. This cluster includes multiple ETL scheduling servers, which can run different jobs as needed. The aforementioned retrieval of multiple jobs executed by multiple servers in the cluster can refer to jobs executed within a specific time frame. For example, jobs executed within the past year, which can be adjusted as needed.

[0037] Considering that existing technologies use the full dataset of a server cluster to create linear regression models, and randomly select a preset number of job data from the full dataset to create linear regression models, the correlation between job data in the full dataset is poor. This embodiment calculates the similarity of jobs and performs clustering, using the historical runtime data of multiple jobs in the same cluster as the target job to create a linear regression model. The resulting linear regression model is more targeted and the prediction of the target job is more accurate.

[0038] Based on the similarity set of any two jobs in a set of multiple jobs, the multiple jobs are clustered to obtain multiple clusters. The K-means clustering algorithm can be used to cluster the multiple jobs to obtain multiple clusters. The similarity of multiple jobs in each cluster is greater than a certain threshold, that is, their correlation is relatively strong.

[0039] The above-mentioned linear regression model is created based on the historical runtime of the target job and the historical runtimes of multiple similar jobs within the same cluster. This can be achieved by obtaining the historical runtime of the target job and the historical runtimes of multiple similar jobs within the same cluster from the load balancer. The historical runtime is then divided into multiple time units, and the arithmetic mean of these time units is calculated to obtain a first result. Finally, the linear regression model is constructed based on the actual runtime and the first result. The preset time range can be one month, one week, or determined according to the actual operating patterns of the job. Multiple time units can be divided according to the preset time range, with each time unit corresponding to one week, 10 days, or other time ranges. The above linear regression model can be a multiple linear regression model.

[0040] In this embodiment, the preset time range is one month, and the time unit is 10 days. The load balancer collects the actual runtime of 50 data lake loading jobs and the historical runtime of the most recent month corresponding to the actual runtime, and establishes a reasonable linear regression model based on the above data.

[0041] When determining the server load rate based on the server's predicted runtime, first determine the server's current load, and then determine the load rate based on the server's current load and inherent load.

[0042] When determining the current load of a server, the runtime of the server's historical jobs is used; based on the predicted runtime and the runtime, the time weight value of all jobs currently running on each server is determined; based on the time weight value, the time quantization result of all jobs currently running on each server is determined; based on the time quantization result, the current load of each server is determined.

[0043] When determining the load rate based on the server's current load and inherent load, the server's load factor is determined. The load factor represents the server's ability to withstand load after entering a critical state. The critical state represents the number of job requests on the server reaching a preset number. The server's load rate is determined based on the current load, inherent load, and load factor.

[0044] Optionally, before clustering multiple jobs based on the similarity set of any two jobs in multiple jobs to obtain multiple clusters, the method further includes: determining the similarity value of two jobs based on the scores of any two jobs in multiple jobs within a preset time unit; normalizing the similarity value to obtain the intermediate similarity value corresponding to the similarity value within the preset time unit; determining the similarity of two jobs based on the number of time units within the preset time range and the intermediate similarity value of multiple time units, wherein both jobs are run within the time unit; and determining the similarity set of any two jobs in multiple jobs by traversing any two jobs in multiple jobs.

[0045] The above similarity values ​​can be considered as the similarity Hamming distance, which is defined as follows:

[0046]

[0047] Where r ih r jh These represent the scores given by the ETL scheduling system to job i and job j in the h-th month of this year, with the preset time unit being the month.

[0048] Then, the similarity of all tasks is normalized according to the following formula to obtain the final similarity, which ranges from (0,1), and the larger the value, the higher the similarity between the two tasks:

[0049]

[0050] Where v represents the final similarity between assignment i and assignment j, which is the intermediate similarity value mentioned above; vi is the similarity value D(i,j,h); vmax is the maximum value of D(i,j,h), which is the maximum value of the similarity value; and vmin is the minimum value of D(i,j,h), which is the minimum value of the similarity value.

[0051] The similarity between two tasks is determined based on the number of time units within a preset time range and the median similarity value of multiple time units, calculated using the following formula:

[0052]

[0053] Where US(i,j) represents the similarity between two jobs, job i and job j, U1 and U2 are the sets of historical months in which the ETL scheduling system runs job i and job j, respectively, and N is the number of months in which the ETL scheduling system runs job i and job j simultaneously, which is also the number of time units within the aforementioned preset time range.

[0054] Optionally, the two assignments include a first assignment and a second assignment. Determining the similarity value of the two assignments based on the scores of any two assignments within a preset time unit includes: determining the first score of the first assignment within the preset time unit and the second score of the second assignment within the preset time unit; determining the similarity Hamming distance between the first assignment and the second assignment based on the first score and the second score, wherein the similarity Hamming distance is the similarity value between the first assignment and the second assignment.

[0055] The first task mentioned above can be task i, and the second task mentioned above can be task j. The ETL scheduling system Automation calculates the total number of times task i's running time meets the standard (i.e., the actual daily running time is less than or equal to t0) in the h-th month based on the standard running time t0 of task i in the total critical path of the batch. This yields a compliance rate rih = Ms / M, which is the first score of task i, i.e., the score of task i by the ETL scheduling system in the h-th month, where M is the total number of times task i runs in the h-th month. Similarly, the second score rjh of task j by the ETL scheduling system in the h-th month can be obtained.

[0056] Using the above formula for calculating the similarity Hamming distance D(i,j,h), and based on the first score rih and the second score rjh, the similarity values ​​of the first and second assignments are determined.

[0057] That is, determining the first score of the first task within a preset time unit and the second score of the second task within a preset time unit includes: using the ratio of the number of times the first task's running time reaches the standard running time within the time unit to the total number of times the first task is run within the time unit as the first score of the first task's time; and using the ratio of the number of times the second task's running time reaches the standard running time within the time unit to the total number of times the second task is run within the time unit as the second score of the second task's time.

[0058] Optionally, normalizing similar values ​​to obtain intermediate similar values ​​within a preset time unit includes: traversing any two jobs from multiple jobs to obtain a set of similar values, wherein the set of similar values ​​includes the similar values ​​of any two jobs from multiple jobs; determining the maximum and minimum values ​​of the similar values ​​based on the set of similar values; and normalizing the similar values ​​based on the maximum, minimum, and the similar values ​​of the two jobs to obtain intermediate similar values ​​corresponding to the similar values ​​of the two jobs.

[0059] In the process of determining the intermediate similarity value v based on the normalization formula, we first need to determine the similarity values ​​of all two jobs by traversing the data, and then determine the maximum value vmax and the minimum value vmin of the similarity values. Only then can we determine the intermediate similarity value between job i and job j based on the maximum value vmax and the minimum value vmin of the similarity values, as well as the aforementioned similarity values.

[0060] Optionally, determining the similarity between two tasks based on the number of time units within a preset time range and the intermediate similarity values ​​of multiple time units includes: traversing any two tasks from the multiple tasks to obtain a set of intermediate similarity values, wherein the set of intermediate similarity values ​​includes the intermediate similarity values ​​corresponding to the similarity values ​​of the two tasks within multiple time units, and the multiple time units include preset time units; determining the intermediate similarity values ​​of multiple time units that satisfy the preset time range based on the set of intermediate similarity values; summing the intermediate similarity values ​​of multiple time units and calculating the ratio with the number of multiple time units to obtain the similarity between the two tasks.

[0061] In the process of determining the similarity between job i and job j according to the similarity formula US(i,j), we first need to determine all the time units of job i and job j running at the same time by traversing through them. The intermediate similarity value includes the corresponding month h. Then we can determine the number of time units of job i and job j running at the same time. Only then can we sum up the intermediate similarity values ​​of multiple time units and calculate the ratio with the number of time units to obtain the similarity between the two jobs.

[0062] Optionally, before predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model, the method further includes: receiving a job request for the target job and selecting a default server from the server cluster to process the job request; if the default server cannot execute the target job, performing the step of predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model; after predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model, the method further includes: determining the load rate of the corresponding server based on the server's runtime; selecting the target server with the lowest load rate among the at least one server and executing the job request.

[0063] When a task request arrives, the load balancer first uses the Murmur hash algorithm to calculate the server node that will execute the task. The Murmur hash algorithm is a high-performance algorithm with a low collision rate.

[0064] The load balancer calculates the default server to be allocated using a consistent hash ring. During application, the load balancer hashes the server's IP address and port number, and maps the server to a node on the ring based on the calculation result. The field next to the server node (e.g., "11.1.121.1:8080") represents the server's IP address and port number. When a job request arrives, the load balancer maps the job request to the ring according to the specified hash key, and selects the nearest server node clockwise to process the job request. The hash key is an identifier associated with the job; in this embodiment, the hash key is the process ID (e.g., useid) during job execution. It should be noted that in practical applications, the number of data lake ETL scheduling servers can reach dozens or even hundreds. Therefore, using the Murmur hash algorithm in this application does not result in a large number of job requests falling onto the same server node.

[0065] It should be noted that by first selecting a default server from the server cluster to handle job requests, and then making a simple judgment on the load rate of the default server, and if the default server cannot execute the target job, the step of predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model is performed, instead of calculating the load rate of all servers in the server cluster first and then selecting a default server, the computational load of the load balancer can be effectively reduced and the allocation efficiency can be improved.

[0066] When the load rate of the default server is greater than the preset load rate, the load rate of at least one server is obtained. The load rate of at least one server is determined based on the runtime of the job running on at least one server in the server cluster. The runtime is predicted based on the linear regression model mentioned above.

[0067] Before determining the load rate of a server based on its runtime, the load balancer first determines the relationship between the default server and the preset load rate. The preset load rate can be a fixed value, linearly related to the average load rate of the server cluster, or correspond to other types of load rates in the server cluster (such as minimum load rate, maximum load rate). In this embodiment, the preset load rate is the average load rate of the server cluster. When the load balancer determines that the load rate of the first server exceeds the average load rate of the server cluster, the load balancer obtains the load rate of at least one server.

[0068] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0069] This embodiment also provides an optional implementation method, which will be described in detail below.

[0070] This embodiment addresses the shortcomings and deficiencies of existing technologies by providing an improved clustering-based ETL job runtime prediction method. Specifically, it includes the improved clustering-based load balancing scheduling algorithm KNN-TPLBSA (Time-Predicting based Load Balancing Scheduling Algorithm by KNN). In related technologies, the job dataset used for training the linear regression model is randomly sampled from all jobs, leading to noise and bias in the final training results. Therefore, this embodiment first clusters all datasets and then trains the model based on the clustered datasets, thereby achieving higher prediction performance.

[0071] To maximize the service quality and request response speed of the ETL server cluster, the load balancing algorithm must ensure a balanced distribution of user requests without compromising request speed. The KNN-TPLBSA algorithm in this implementation achieves "more work for the capable, balanced distribution" without affecting the load balancer's allocation efficiency.

[0072] Figure 2 This is a flowchart illustrating an optional server cluster load balancing method according to an embodiment of the present invention, as shown below. Figure 2 As shown, when a task request arrives, the server node assigned to the task is first calculated using a hash algorithm. Then, it is determined whether the load rate of the assigned server node exceeds the average load rate of the cluster. If it does not exceed the average load rate, the request is assigned to the server node. If it does exceed the average load rate, the load transfer module is executed to assign the request to the server node with the lowest load rate in the cluster. When the request returns, the load balancer calculates the load rate of the assigned server node and the average load rate of the cluster using a dynamic request load allocation algorithm. The entire algorithm process will be described in detail below.

[0073] 1. Hash Algorithm Implementation: The load balancer calculates the servers to be allocated using a consistent hash ring. During application, the load balancer hashes the server's IP address and port number, and maps the server to a node on the ring based on the calculation result. The field next to the server node (e.g., "11.1.121.1:8080") represents the server's IP address and port number. When a job request arrives, the load balancer maps the job request to the ring according to the specified hash key, and selects the nearest server node clockwise to process the job request. The hash key is an identifier associated with the job; in this embodiment, the hash key is the process ID (e.g., useid) during job execution. It should be noted that in practical applications, the number of data lake ETL scheduling servers can reach dozens or even hundreds. Therefore, using the Murmur hash algorithm in this implementation will not result in a large number of job requests falling onto the same server node.

[0074] It should be noted that by first selecting a default server from the server cluster to handle job requests, and then simply judging the load rate of the default server, instead of calculating the load rate of all servers in the server cluster first and then selecting the first server, the computational load of the load balancer can be effectively reduced and the allocation efficiency can be improved.

[0075] 2. Representation and measurement of dynamic load balancing server capacity:

[0076] (1) Determination of inherent load capacity;

[0077] The inherent load capacity of each server can be measured using a third-party server to at least determine the server's saturation state, critical state, critical response time, and saturation response time. In this embodiment, 1000 threads are set, with a time interval of 7 seconds, and these 1000 threads are continuously looped for 9 rounds to simulate 9K requests. The results are measured. The horizontal axis represents the number of job requests sent, and the vertical axis (solid line) represents the server's response time to the job requests. When the number of job requests reaches approximately 8000, the server's request success rate suddenly drops and persists for a period of time. This is a measure taken by the system software to prevent a crash, also known as a "false death" phenomenon. "False death" can severely impact server performance, so this phenomenon must be avoided. When a server appears to freeze, it's determined that the server is in a saturated state. The response time during this "freeze" is 4980ms, denoted as T(max). When the server's response time exceeds T(max), its available resources are zero. The server's minimum response time is 17ms, denoted as T(bas), which represents the basic load for the system to maintain operation. When the number of requests reaches a certain amount (3250 times), the server's response time will exhibit "jitter," with a critical time of 1900ms, denoted as T(crt). This indicates that the server has entered a critical state. Here, T(max) is the saturation response time, representing the response time when at least one server enters a saturated state, and T(crt) (hereinafter referred to as Tcrt) is the critical response time, representing the response time when at least one server enters a critical state.

[0078] (2) Representation and calculation of dynamic load:

[0079] Hamming distance is generally used in information theory to calculate the number of unequal characters in corresponding positions of two strings. For example, the Hamming distance between "123" and "124" is 1. This paper utilizes the relationship between the ETL scheduling system and jobs to calculate the similarity distance between two jobs, thereby determining their similarity. The similarity Hamming distance D(i,j,h) is defined as follows:

[0080]

[0081] Where rih and rjh represent the scores of job i and job j by the ETL scheduling system in the h-th month of the current year, respectively. The ETL scheduling system Automation calculates the total number of times job i's running time meets the standard (i.e., the actual daily running time is less than or equal to t0) in the h-th month based on the standard running time t0 of job i in the total critical path of the batch, thus obtaining a pass rate rih = Ms / M, which is the score of job i by the ETL scheduling system in the h-th month, where M is the total number of times job i runs in the h-th month; similarly, the score rjh of job j by the ETL scheduling system in the h-th month can be obtained. The value range of D(i,j,h) is [0,0.5]. The closer the value is to 0, the higher the similarity between the two jobs, and vice versa. Then, the similarity of all jobs is normalized according to the following formula to obtain the final similarity, which is the value range between (0,1) and the larger the value, the higher the similarity between the two jobs:

[0082]

[0083] Where v represents the final similarity between assignment i and assignment j, vi is the above D(i,j,h), vmax is the maximum value of D(i,j,h), and vmin is the minimum value of D(i,j,h).

[0084] The dataset contains many data points that have little or no interaction with each other. This implementation method uses the K-means clustering algorithm to reduce the size of the dataset, and then uses the reduced dataset to run the prediction model algorithm. The core part of the clustering process is the calculation of similarity distance. The method for calculating similarity Hamming distance has already been introduced. For example, the similarity Hamming distance between Job 1 and Job 2 can be calculated using the formula. The ETL scheduling system's scores for Job 1 and Job 2 in the first five months of this year are: D(1,2,1) = 0.5, D(1,2,2) = 0.5, D(1,2,3) = 0.1, D(1,2,4) = 0.5, D(1,2,5) = 0.5. Therefore, the similarity between Job 1 and Job 2 in the ETL scheduling system can be calculated using the formula US(i,j), which is:

[0085]

[0086] Where U1 and U2 are the sets of historical months in which the ETL scheduling system runs jobs i and j, respectively, and N is the number of months in which the ETL scheduling system runs jobs i and j simultaneously. The similarity between jobs 1 and 2 is calculated as US(1,2) = 0.42, indicating that the similarity between jobs 1 and 2 is not high. This method can obtain the relationship between jobs i and j, as well as the relationship between the ETL scheduling system in month h. Because it considers the relationship between jobs and the scheduling system, the reliability of the similarity between jobs is higher.

[0087] This implementation improves the distance calculation between core elements in the clustering algorithm by considering the relationship between the job, the job, and the scheduling system when clustering jobs. In other words, it adds the relationship between the job and the scheduling system when examining the similarity between jobs. This method increases the reliability of the similarity algorithm, resulting in a more realistic and convincing dataset. The improved algorithm has the following advantages:

[0088] As the amount of assignment scoring data increases, the scoring matrix becomes denser, making the similarity calculation between assignments more accurate.

[0089] Clustering data increases resource consumption, but processing datasets can be done offline, so it has little impact on user experience.

[0090] The system's scalability is enhanced. When adding new jobs and scheduling systems, calculations only need to be performed with the cluster center, without having to compare with all elements, thus reducing the amount of computation and making system updates faster.

[0091] Figure 3 This is a comparison chart of predicted and actual values ​​of an optional linear regression model according to an embodiment of the present invention, such as... Figure 3 The figure shows the experimental comparison results of the KNN-TPLBSA algorithm in this embodiment. Figure 3 The lighter-colored polylines in the middle square nodes represent the predicted job runtime values ​​from the KNN-TPLBSA algorithm. Figure 3 The darker colored lines in the middle diamond nodes represent the actual runtime of the job. By comparison, it can be seen that the KNN-TPLBSA algorithm model, which clusters the data before prediction, performs very well, with the predicted values ​​being very close to the actual values, almost overlapping.

[0092] The next step is to calculate the time quantification of the current load on each server based on the predicted runtime, and then rationally allocate the jobs in the pending queue to the server with the least load.

[0093] To test the algorithm's performance, a performance test was conducted on the implemented load balancing algorithm. The test environment consisted of 3 load balancers and 9 backend servers, all running Ubuntu 18 with 4GB of memory and 500GB of hard drive space.

[0094] The test environment was a high-speed LAN. The performance of the Random Selection Method (RSM), SLBT algorithm (Algorithm A), TPLBSA algorithm (Algorithm B), and the KNN-TPLBSA algorithm implemented in this paper was compared. The test request job was a Hadoop IMP import job. The average response time and success rate of the requests were analyzed. The test results are as follows: Figure 4 , Figure 5 As shown, Figure 4 This is a performance test diagram of an optional server cluster data processing method according to an embodiment of the present invention. Figure 5 This is another performance test diagram of an optional server cluster data processing method according to an embodiment of the present invention.

[0095] The bar charts in the two experimental results above, from left to right, represent the RSM algorithm, SLBT algorithm, TPLBSA algorithm, and KNN-TPLBSA algorithm; from... Figure 4 It can be seen that when the number of concurrent requests equals 5 * 5K = 25000, the average response time of the algorithm in this embodiment is less than 2 seconds, and as the number of concurrent requests increases, the algorithm in this embodiment shows a significant advantage over the other three existing algorithms. From Figure 5 It can be seen that as the number of concurrent requests increases, the success rate of the algorithm in this implementation is significantly higher than that of the previous three algorithms, with the RSM algorithm having the lowest success rate. Test results show that the algorithm in this implementation significantly improves upon the original algorithm in both request success rate and average response time.

[0096] The KNN-TPLBSA algorithm proposed in this implementation ensures that all requests are evenly distributed across each server. A dynamic request load balancing algorithm is implemented through job time quantization, utilizing in-memory computation of all job time quantization results and server load, significantly improving computational efficiency. This avoids analyzing the current server load and scheduling only during load balancer allocation, and proactively prevents servers from entering a "dead" state. Tests show that the proposed algorithm significantly improves load balancing performance and has broad application prospects in ETL scheduling server cluster systems loaded in data lakes.

[0097] Figure 6 This is a schematic diagram of a data processing apparatus for a server cluster provided according to an embodiment of this application, such as... Figure 6As shown, according to another aspect of this application, a data processing apparatus for a server cluster is provided, including: an acquisition module 82, a clustering module 84, a creation module 86, and a prediction module 88. The apparatus will be described in detail below.

[0098] The acquisition module 82 is used to acquire multiple jobs executed by multiple servers in the server cluster; the clustering module 84, connected to the acquisition module 82, is used to cluster multiple jobs based on the similarity set of any two jobs in the multiple jobs to obtain multiple clusters; the creation module 86, connected to the clustering module 84, is used to create a linear regression model based on the historical runtime of the target job and the historical runtime of multiple similar jobs in the same cluster to which the target job belongs; the prediction module 88, connected to the creation module 86, is used to predict the runtime of the target job for at least one server in the server cluster based on the linear regression model.

[0099] The aforementioned apparatus employs an acquisition module 82 to acquire multiple jobs executed by multiple servers in a server cluster; a clustering module 84 to cluster the multiple jobs based on the similarity set of any two jobs, resulting in multiple clusters; a creation module 86 to create a linear regression model based on the historical runtime of the target job and the historical runtimes of multiple similar jobs within the same cluster; and a prediction module 88 to predict the runtime of the target job on at least one server in the server cluster based on the linear regression model. By clustering different jobs in the server cluster based on the similarity of the jobs, and creating a linear regression model based on the runtime of the target job and the runtimes of similar jobs within the same cluster, the resulting linear regression model can accurately predict the runtime of the target job. This solves the problem in related technologies where linear regression models are created using full datasets, resulting in poor correlation between the full datasets and low accuracy. Therefore, it improves the accuracy of the linear regression model in predicting the runtime of the target job on different servers, and enhances the accuracy of determining server load rates.

[0100] Optionally, the device further includes: a scoring module for determining the similarity value of two jobs based on the scores of any two jobs in a preset time unit; a processing module for normalizing the similarity value to obtain the intermediate similarity value corresponding to the similarity value within the preset time unit; a calculation module for determining the similarity between two jobs based on the number of time units within the preset time range and the intermediate similarity value of multiple time units, wherein both jobs are run within the time unit; and a traversal module for determining the similarity set of any two jobs in the multiple jobs by traversing any two jobs.

[0101] Optionally, the two assignments include a first assignment and a second assignment. The scoring module includes: a scoring unit for determining a first score for the first assignment in a preset time unit and a second score for the second assignment in a preset time unit; and a first calculation unit for determining the similarity Hamming distance between the first assignment and the second assignment based on the first score and the second score, wherein the similarity Hamming distance is the similarity value between the first assignment and the second assignment.

[0102] Optionally, the scoring unit includes: a first scoring subunit, used to take the ratio of the number of times the running time of the first job reaches the standard running time within a time unit to the total number of times the first job is run within a time unit as the first score for the first job time; and a second scoring subunit, used to take the ratio of the number of times the running time of the second job reaches the standard running time within a time unit to the total number of times the second job is run within a time unit as the second score for the second job time.

[0103] Optionally, the processing module includes: a first traversal unit, used to traverse any two jobs among multiple jobs to obtain a set of similar values, wherein the set of similar values ​​includes the similar values ​​of any two jobs among multiple jobs; an extremum unit, used to determine the maximum and minimum values ​​of the similar values ​​based on the set of similar values; and a second calculation unit, used to normalize the similar values ​​based on the maximum, minimum and the similar values ​​of the two jobs to obtain the intermediate similar value corresponding to the similar values ​​of the two jobs.

[0104] Optionally, the calculation module includes: a second traversal unit, used to traverse any two jobs among multiple jobs to obtain an intermediate similarity value set, wherein the intermediate similarity value set includes intermediate similarity values ​​corresponding to the similarity values ​​of the two jobs within multiple time units, and the multiple time units include preset time units; a determination unit, used to determine the intermediate similarity values ​​of multiple time units that satisfy the preset time range based on the intermediate similarity value set; and a third calculation unit, used to sum the intermediate similarity values ​​of multiple time units and calculate the ratio with the number of multiple time units to obtain the similarity between the two jobs.

[0105] Optionally, the device further includes: a receiving module for receiving job requests from a target job and selecting a default server from the server cluster to process the job request; a load module for determining the load rate of the corresponding server based on the server's runtime; and a selection module for selecting a target server with the lowest load rate among at least one server to execute the job request.

[0106] To achieve the above objectives, according to another aspect of this application, a processor is provided for running a program, wherein the program executes the data processing method for a server cluster as described in any one of the above claims.

[0107] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data processing method of the server cluster described in any one of the above.

[0108] The data processing device of the server cluster includes a processor and a memory. The acquisition module 82, clustering module 84, creation module 86 and prediction module 88 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0109] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured to create a linear regression model based on the runtime of the target job and the runtime of similar jobs in a unified cluster, resulting in a linear regression model that can accurately predict the runtime of the target job.

[0110] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0111] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the data processing method of the server cluster.

[0112] This invention provides a processor for running a program, wherein the program executes a data processing method for the server cluster.

[0113] like Figure 7 As shown, an embodiment of the present invention provides an electronic device 50, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps:

[0114] Obtain multiple jobs executed by multiple servers in the server cluster; cluster the multiple jobs according to the similarity set of any two jobs in the multiple jobs to obtain multiple clusters; create a linear regression model based on the historical runtime of the target job and the historical runtime of multiple similar jobs in the same cluster to which the target job belongs; predict the runtime of the target job for at least one server in the server cluster based on the linear regression model.

[0115] Optionally, before clustering multiple jobs based on the similarity set of any two jobs in multiple jobs to obtain multiple clusters, the method further includes: determining the similarity value of two jobs based on the scores of any two jobs in multiple jobs within a preset time unit; normalizing the similarity value to obtain the intermediate similarity value corresponding to the similarity value within the preset time unit; determining the similarity of two jobs based on the number of time units within the preset time range and the intermediate similarity value of multiple time units, wherein both jobs are run within the time unit; and determining the similarity set of any two jobs in multiple jobs by traversing any two jobs in multiple jobs.

[0116] Optionally, the two assignments include a first assignment and a second assignment. Determining the similarity value of the two assignments based on the scores of any two assignments within a preset time unit includes: determining the first score of the first assignment within the preset time unit and the second score of the second assignment within the preset time unit; determining the similarity Hamming distance between the first assignment and the second assignment based on the first score and the second score, wherein the similarity Hamming distance is the similarity value between the first assignment and the second assignment.

[0117] Optionally, determining the first score of the first task within a preset time unit and the second score of the second task within a preset time unit includes: using the ratio of the number of times the first task's running time reaches the standard running time within the time unit to the total number of times the first task is run within the time unit as the first score of the first task's time; and using the ratio of the number of times the second task's running time reaches the standard running time within the time unit to the total number of times the second task is run within the time unit as the second score of the second task's time.

[0118] Optionally, normalizing similar values ​​to obtain intermediate similar values ​​within a preset time unit includes: traversing any two jobs from multiple jobs to obtain a set of similar values, wherein the set of similar values ​​includes the similar values ​​of any two jobs from multiple jobs; determining the maximum and minimum values ​​of the similar values ​​based on the set of similar values; and normalizing the similar values ​​based on the maximum, minimum, and the similar values ​​of the two jobs to obtain intermediate similar values ​​corresponding to the similar values ​​of the two jobs.

[0119] Optionally, determining the similarity between two tasks based on the number of time units within a preset time range and the intermediate similarity values ​​of multiple time units includes: traversing any two tasks from the multiple tasks to obtain a set of intermediate similarity values, wherein the set of intermediate similarity values ​​includes the intermediate similarity values ​​corresponding to the similarity values ​​of the two tasks within multiple time units, and the multiple time units include preset time units; determining the intermediate similarity values ​​of multiple time units that satisfy the preset time range based on the set of intermediate similarity values; summing the intermediate similarity values ​​of multiple time units and calculating the ratio with the number of multiple time units to obtain the similarity between the two tasks.

[0120] Optionally, before predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model, the method further includes: receiving a job request for the target job and selecting a default server from the server cluster to process the job request; after predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model, the method further includes: determining the load rate of the corresponding server based on the server's runtime; selecting the target server with the lowest load rate among the at least one server and executing the job request.

[0121] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0122] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: obtaining multiple jobs executed by multiple servers in a server cluster; clustering the multiple jobs according to the similarity set of any two jobs in the multiple jobs to obtain multiple clusters; creating a linear regression model based on the historical runtime of the target job and the historical runtime of multiple similar jobs in the same cluster to which the target job belongs; and predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model.

[0123] Optionally, before clustering multiple jobs based on the similarity set of any two jobs in multiple jobs to obtain multiple clusters, the method further includes: determining the similarity value of two jobs based on the scores of any two jobs in multiple jobs within a preset time unit; normalizing the similarity value to obtain the intermediate similarity value corresponding to the similarity value within the preset time unit; determining the similarity of two jobs based on the number of time units within the preset time range and the intermediate similarity value of multiple time units, wherein both jobs are run within the time unit; and determining the similarity set of any two jobs in multiple jobs by traversing any two jobs in multiple jobs.

[0124] Optionally, the two assignments include a first assignment and a second assignment. Determining the similarity value of the two assignments based on the scores of any two assignments within a preset time unit includes: determining the first score of the first assignment within the preset time unit and the second score of the second assignment within the preset time unit; determining the similarity Hamming distance between the first assignment and the second assignment based on the first score and the second score, wherein the similarity Hamming distance is the similarity value between the first assignment and the second assignment.

[0125] Optionally, determining the first score of the first task within a preset time unit and the second score of the second task within a preset time unit includes: using the ratio of the number of times the first task's running time reaches the standard running time within the time unit to the total number of times the first task is run within the time unit as the first score of the first task's time; and using the ratio of the number of times the second task's running time reaches the standard running time within the time unit to the total number of times the second task is run within the time unit as the second score of the second task's time.

[0126] Optionally, normalizing similar values ​​to obtain intermediate similar values ​​within a preset time unit includes: traversing any two jobs from multiple jobs to obtain a set of similar values, wherein the set of similar values ​​includes the similar values ​​of any two jobs from multiple jobs; determining the maximum and minimum values ​​of the similar values ​​based on the set of similar values; and normalizing the similar values ​​based on the maximum, minimum, and the similar values ​​of the two jobs to obtain intermediate similar values ​​corresponding to the similar values ​​of the two jobs.

[0127] Optionally, determining the similarity between two tasks based on the number of time units within a preset time range and the intermediate similarity values ​​of multiple time units includes: traversing any two tasks from the multiple tasks to obtain a set of intermediate similarity values, wherein the set of intermediate similarity values ​​includes the intermediate similarity values ​​corresponding to the similarity values ​​of the two tasks within multiple time units, and the multiple time units include preset time units; determining the intermediate similarity values ​​of multiple time units that satisfy the preset time range based on the set of intermediate similarity values; summing the intermediate similarity values ​​of multiple time units and calculating the ratio with the number of multiple time units to obtain the similarity between the two tasks.

[0128] Optionally, before predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model, the method further includes: receiving a job request for the target job and selecting a default server from the server cluster to process the job request; after predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model, the method further includes: determining the load rate of the corresponding server based on the server's runtime; selecting the target server with the lowest load rate among the at least one server and executing the job request.

[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0135] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0137] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data processing method for a server cluster, characterized in that, include: Retrieve multiple jobs that have been executed by multiple servers in a server cluster; Based on the similarity set of any two jobs among the multiple jobs, the multiple jobs are clustered to obtain multiple clusters; A linear regression model is created based on the historical runtime of the target job and the historical runtime of multiple similar jobs in the same cluster to which the target job belongs. Predict the runtime of the target job for at least one server in the server cluster based on the linear regression model. Before predicting the runtime of the target job for at least one server in the server cluster according to the linear regression model, the method further includes: receiving a job request for the target job, selecting a default server from the server cluster to process the job request; and if the default server cannot execute the target job, performing the step of predicting the runtime of the target job for at least one server in the server cluster according to the linear regression model. The method further includes, after predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model, the method further includes: determining the load rate of the corresponding server based on the server runtime; selecting the target server with the lowest load rate among the at least one server and executing the job request. The process of determining the load rate of a server based on its runtime includes: determining the time weight value of all currently running jobs on each server based on the historical runtime of all jobs executed by each server and the runtime of the target job predicted by the linear regression model; determining the time quantization result of all currently running jobs on each server based on the time weight value; determining the current load of each server based on the time quantization result; and determining the server load rate based on the current load, inherent load, and a boosting factor, wherein the boosting factor is used to characterize the server's ability to withstand load after entering a critical state, and the critical state is used to characterize the number of job requests on the server reaching a preset number.

2. The method according to claim 1, characterized in that, Before clustering the multiple tasks based on the similarity set of any two tasks to obtain multiple clusters, the method further includes: Based on the scores of any two tasks within a preset time unit, determine the similarity value of the two tasks; The similar values ​​are normalized to obtain the intermediate similar values ​​corresponding to the similar values ​​within the preset time unit; The similarity between the two jobs is determined based on the number of time units within a preset time range and the intermediate similarity value of the multiple time units, wherein both jobs are run within the time unit; By traversing any two of the multiple tasks, the similarity set of any two tasks among the multiple tasks is determined.

3. The method according to claim 2, characterized in that, The two tasks include a first task and a second task. The similarity score between the two tasks is determined based on the scores of any two tasks within a preset time unit, including: Determine a first score for the first task within the preset time unit, and a second score for the second task within the preset time unit; The similarity Hamming distance between the first task and the second task is determined based on the first score and the second score, wherein the similarity Hamming distance is the similarity value between the first task and the second task.

4. The method according to claim 3, characterized in that, Determining a first score for the first task within the preset time unit, and a second score for the second task within the preset time unit, includes: The ratio of the number of times the first job's running time reaches the standard running time within the preset time unit to the total number of times the first job is run within the preset time unit is used as the first score of the first job. The ratio of the number of times the second task's running time reaches the standard running time within the preset time unit to the total number of times the second task is run within the preset time unit is used as the second score for the second task.

5. The method according to claim 2, characterized in that, Normalizing the similar values ​​yields intermediate similar values ​​within the preset time unit, including: Traverse any two jobs among the plurality of jobs to obtain a set of similar values, wherein the set of similar values ​​includes the similar values ​​of any two jobs among the plurality of jobs; The maximum and minimum values ​​of the similar values ​​are determined based on the set of similar values. Based on the maximum value, the minimum value, and the similarity value of the two jobs, the similarity value is normalized to obtain the intermediate similarity value corresponding to the similarity value of the two jobs.

6. The method according to claim 2, characterized in that, Determining the similarity between the two jobs based on the number of time units within a preset time range and the intermediate similarity values ​​of multiple time units includes: Traverse any two jobs among the plurality of jobs to obtain an intermediate similarity value set, wherein the intermediate similarity value set includes intermediate similarity values ​​corresponding to the similarity values ​​of the two jobs within a plurality of time units, and the plurality of time units includes the preset time unit; Based on the set of intermediate similarity values, determine intermediate similarity values ​​for multiple time units that satisfy the preset time range; The similarity between the two tasks is obtained by summing the intermediate similarity values ​​of the multiple time units and then comparing the sum with the number of the multiple time units.

7. A data processing device for a server cluster, characterized in that, include: The acquisition module is used to acquire multiple jobs that have been executed by multiple servers in the server cluster; The clustering module is used to cluster the multiple jobs based on the similarity set of any two jobs, thereby obtaining multiple clusters. A module is created to generate a linear regression model based on the historical runtime of the target job and the historical runtime of multiple similar jobs in the same cluster to which the target job belongs. The prediction module is used to predict the runtime of the target job for at least one server in the server cluster based on the linear regression model. The data processing device of the server cluster further includes: a receiving module, used to receive a job request for a target job, select a default server from the server cluster to process the job request; and, if the default server cannot execute the target job, perform a step of predicting the runtime of at least one server in the server cluster for the target job based on the linear regression model. The data processing device of the server cluster further includes: a load module, used to determine the load rate of the corresponding server based on the server's runtime; and a selection module, used to select the target server with the lowest load rate among the at least one server and execute the job request. The load module is further configured to: determine the time weight value of all currently running jobs on each server based on the historical runtime of all jobs executed by each server and the runtime of the target job predicted by the linear regression model; determine the time quantization result of all currently running jobs on each server based on the time weight value; determine the current load of each server based on the time quantization result; and determine the server load rate based on the current load, inherent load, and support factor, wherein the support factor is used to characterize the server's ability to withstand load after entering a critical state, and the critical state is used to characterize the number of job requests on the server reaching a preset number.

8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the data processing method of the server cluster according to any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data processing method of the server cluster according to any one of claims 1 to 6.

Images