A method for estimating running time of a computing cluster job

Abstract

This invention provides a method for estimating the runtime of computing cluster jobs, comprising: extracting multiple features of the job to be predicted to form a multi-dimensional feature vector and performing normalization processing; calculating adaptive weights for each dimension of features to generate a weighted feature representation, and generating spatiotemporal embedded features through a gating fusion mechanism; decoupling the spatiotemporal embedded features to generate a decoupled feature representation; aligning the decoupled feature representations of the current job with those of historical jobs and calculating their similarity, and compensating for environmental disturbances in the similarity; filtering a set of candidate historical jobs based on the comprehensive similarity, and generating a final runtime estimate through weighted estimation and multi-level correction. This invention achieves accurate representation of job information and high-quality matching of candidate historical jobs through the synergistic effect of multi-dimensional feature fusion and adaptive weight allocation, feature decoupling and alignment, and similarity environmental disturbance compensation, thus improving the overall accuracy and robustness of the prediction.

Description

Technical Field

[0001] This invention relates to the field of computing cluster job scheduling technology, and in particular to a method for estimating the running time of computing cluster jobs. Background Technology

[0002] In existing technologies, job runtime estimation mainly relies on users manually entering the estimated duration. Due to the selfish tendency of users to significantly overestimate the duration to prevent forced job termination, the overall estimation accuracy is generally below 60%, severely restricting scheduling quality. Some studies have attempted to introduce historical job similarity matching methods, but these still have several shortcomings: traditional job fingerprints only use static and simple features, failing to perceive dynamic environmental information such as real-time load, network congestion, and queue status; static features and dynamic environmental features are highly coupled in the same vector space, making similarity matching susceptible to perturbations; existing methods lack the ability to align with the non-Euclidean manifold trajectory of the job execution process, resulting in insufficient accuracy in similar job matching; simultaneously, the lack of environmental negative entropy gain compensation mechanisms and gated adaptive spatiotemporal embedding capabilities prevents the model from dynamically perceiving system entropy increases and context changes, causing the estimation results to continuously drift over runtime; furthermore, existing models generally lack online iterative optimization capabilities, leading to a continuous decline in long-term accuracy. These shortcomings indicate that existing technologies cannot yet meet the accuracy and robustness requirements of high-performance computing clusters for job runtime estimation. Summary of the Invention

[0003] The purpose of this invention is to solve the problems of low accuracy and poor robustness in the estimation of operation time in the prior art.

[0004] This invention provides a method for estimating the runtime of computing cluster jobs, comprising: Features are extracted from the task to be predicted to form a multidimensional feature vector, and the multidimensional feature vector is then normalized. Calculate the adaptive weights of each feature dimension of the normalized multidimensional feature vector to generate a weighted feature representation; A gated vector is constructed and fused with the weighted feature representation to generate a spatiotemporal embedding feature; the spatiotemporal embedding feature is then decoupled to generate a decoupled feature representation. Align the decoupled feature representations of the current job and historical jobs, calculate their similarity, and compensate for the decoupled feature representations or the similarity. Calculate the comprehensive similarity and generate a candidate historical job set. Based on the candidate historical job set, generate basic prediction results. Perform multi-level corrections on the basic prediction results to generate the final runtime estimate.

[0005] Furthermore, the multidimensional feature vector includes extracting user features, job semantic features, resource request features, and dynamic environment features, and normalizing the feature values ​​of each dimension to generate normalized feature values: in, These are the original eigenvalues. and These are the minimum and maximum values ​​of this feature obtained from historical statistics, respectively. Based on the statistical variance, prior importance, and discriminative contribution of historical data, a multidimensional adaptive weight calculation model is constructed to calculate the adaptive weights: ; in, Features Historical variance; Features The importance weight; λ is the weight adjustment coefficient.

[0006] Furthermore, a gating vector is constructed based on system load, queue length, network congestion, and time decay factor, and then weighted and fused with the weighted feature representation to generate spatiotemporal embedding features: ; in, This represents the activation function. For the gated weight matrix, For bias terms, ⊙ indicates element-wise multiplication. This is the input feature vector.

[0007] Furthermore, the spatiotemporal embedding features are divided into a static semantic feature group and a dynamic environment feature group, and the two groups of features are decoupled to generate decoupled static semantic features and dynamic environment features: ; in, Indicates the rotation angle. and These represent the static semantic features and dynamic environmental features after decoupling, respectively.

[0008] Furthermore, the decoupled feature representations of the current and historical tasks are mapped to the manifold space, the state trajectories are aligned, and the similarity between the static semantic trajectory and the dynamic environment trajectory is calculated: Where T represents the current work trajectory, This represents the historical operation trajectory, where σ is the scale parameter.

[0009] Furthermore, based on system load fluctuations, network disturbances, and queue changes, an environmental negative entropy gain compensation mechanism is introduced to compensate for the static semantic trajectory similarity and the dynamic environmental trajectory similarity, respectively. The compensated decoupled feature representation or the similarity is as follows: ; Where η is the compensation coefficient, and ΔH represents the change in system environmental entropy.

[0010] Furthermore, based on static semantic similarity and dynamic environment similarity, a comprehensive similarity is calculated: in, Represents static similarity. This represents the dynamic similarity, where α is the fusion weight coefficient.

[0011] Furthermore, a two-order similarity model is constructed based on the compensated static semantic similarity and dynamic environment similarity, and a set of candidate historical assignments is generated by filtering based on credibility thresholds. The candidate job set is weighted to estimate the running time to generate a basic prediction result, including: using the fusion similarity of each candidate historical job as a weight, the actual running time of the candidate historical jobs is weighted and averaged, with the historical jobs with higher similarity contributing a greater weight.

[0012] Furthermore, the basic prediction results are corrected in multiple stages through load calibration, congestion calibration, and quota calibration to generate the final runtime estimate: in, Based on the predicted value, For cluster load calibration coefficient, For network congestion calibration coefficient, Adjust the user quota calibration coefficient.

[0013] Furthermore, after the task is completed, the model parameters are iteratively updated using gradient descent, with the prediction error as the loss function. in, For the updated model parameters, For learning rate, For the loss function with respect to gradient, The estimated value output by the model. This refers to the actual running time of the task.

[0014] Compared to existing technologies, this invention offers at least the following advantages: By constructing a multi-dimensional feature vector encompassing user behavior, job semantics, resource requirements, and dynamic environment, and combining adaptive weights and gating fusion strategies, it achieves accurate capture of multi-dimensional information and dynamic perception of spatiotemporal context; by decoupling spatiotemporal embedded features into static semantic groups and dynamic environment groups and introducing a manifold space trajectory alignment mechanism, it effectively avoids the problems of mutual interference between features and similarity distortion in Euclidean space; the negative entropy gain compensation mechanism eliminates the impact of system load fluctuations and network disturbances, and the dual-order similarity fusion and credibility screening further ensure the quality of candidate samples; on this basis, the multi-level correction strategy combined with the online feedback iterative update mechanism enables the model to continuously adapt to the dynamic changes in the cluster operating environment, thereby improving the overall accuracy and robustness of job running time prediction. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained as provided without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating the steps of a method for estimating the runtime of a computing cluster job in one embodiment of the present invention. Detailed Implementation

[0017] The present invention will now be described in more detail with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. It should be understood that those skilled in the art can modify the invention described herein while still achieving its advantageous effects. Therefore, the following description should be understood as being broadly known to those skilled in the art and is not intended to limit the invention.

[0018] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0019] The invention is described more specifically by way of example in the following paragraphs with reference to the accompanying drawings. The advantages and features of the invention will become clearer as explained below. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the invention.

[0020] This embodiment provides a method for estimating the runtime of computing cluster jobs. Please refer to [link / reference]. Figure 1 ,include: Features of the task to be predicted are extracted to form a multidimensional feature vector, and the multidimensional feature vector is then normalized. Calculate the adaptive weights of each feature dimension of the normalized multidimensional feature vector to generate a weighted feature representation; A gated vector is constructed and fused with the weighted feature representation to generate a spatiotemporal embedding feature; the spatiotemporal embedding feature is then decoupled to generate a decoupled feature representation. Align the decoupled feature representations of the current job and historical jobs, calculate their similarity, and compensate for the decoupled feature representations or the similarity. Calculate the comprehensive similarity and generate a candidate historical job set. Based on the candidate historical job set, generate basic prediction results. Perform multi-level corrections on the basic prediction results to generate the final runtime estimate.

[0021] In one embodiment, the method for predicting the runtime of high-performance computing cluster jobs is executed as follows: First, multiple features are extracted from the job to be predicted to form a multi-dimensional feature vector and normalized. Then, adaptive weights are calculated based on the statistical characteristics of each feature in historical samples to generate a weighted feature representation. Next, a gated vector is constructed and fused with the weighted feature representation to generate a spatiotemporal embedded feature. Finally, the spatiotemporal embedded feature is decoupled to eliminate mutual interference between features and generate a decoupled feature representation.

[0022] Based on this, the decoupled feature representations of the current job and the historical jobs are mapped to non-Euclidean manifold spaces respectively, and the state trajectory alignment is performed independently and the corresponding similarity is calculated. The above similarity is compensated, and a two-order similarity model is constructed based on the compensated similarity. The comprehensive similarity is calculated and a candidate historical job set is generated to generate basic prediction results. The basic prediction results are then corrected at multiple levels to generate the final running time estimate.

[0023] In this embodiment, the multidimensional feature vector includes extracted user features, job semantic features, resource request features, and dynamic environment features. The feature values ​​of each dimension are then normalized to generate normalized feature values. in, These are the original eigenvalues. and These are the minimum and maximum values ​​of this feature obtained from historical statistics, respectively. Based on the statistical variance, prior importance, and discriminative contribution of historical data, a multidimensional adaptive weight calculation model is constructed to calculate the adaptive weights: ; in, Features Historical variance; Features The importance weight; λ is the weight adjustment coefficient.

[0024] Specifically, normalization, also known as min-max normalization, aims to normalize the original features. Scaling is performed to a preset range, typically [0, 1]. Its core function is to eliminate dimensional and numerical range differences between different feature dimensions, ensuring comparability of all features in subsequent processing and preventing features with large numerical ranges from having a dominant influence on the model. This mapping is achieved through the formula described above, where... and These are the minimum and maximum values ​​of this feature obtained from historical statistics. In practice, it is necessary to perform statistical analysis on a large amount of historical operation data in advance to determine the minimum and maximum values ​​of each feature. and When normalizing the features of the current task to be predicted, the original features are... Substituting into this formula, we can obtain the normalized features. Through the above normalization process, features with different dimensions and numerical ranges are uniformly mapped to the interval [0, 1], eliminating the interference caused by the difference in dimensions of each feature to subsequent calculations, and ensuring the comparability and numerical stability of the feature vectors.

[0025] After normalization, a multidimensional adaptive weight calculation model is constructed based on the statistical variance, prior importance, and discriminative contribution of each feature in historical data. Features The historical variance reflects the degree of fluctuation of this feature; the smaller the variance, the higher the stability and the greater the weight contribution. Features The importance weights are determined by domain knowledge or historical prediction contributions; λ is the weight adjustment coefficient. This mechanism enables the model to dynamically allocate weights based on the actual statistical characteristics of each feature dimension, effectively improving the adaptability and prediction accuracy of the weighted feature representation.

[0026] Furthermore, a gating vector is constructed based on system load, queue length, network congestion, and time decay factor, and then weighted and fused with the weighted feature representation to generate spatiotemporal embedding features: ; in, This represents the activation function. For the gated weight matrix, For bias terms, ⊙ indicates element-wise multiplication. This is the input feature vector.

[0027] In one embodiment, to enable the feature representation to dynamically perceive the cluster's operating status, a gating vector is constructed based on system load, queue length, network congestion level, and time decay factor. This gating vector is then weighted and fused with the weighted feature representation to generate spatiotemporal embedded features. Through activation function Dynamic environmental information such as system load, queue length, network congestion, and time decay is mapped to gating coefficients within the range [0, 1]. Each dimension of the input feature vector is adaptively scaled using element-wise multiplication, allowing for timely suppression or enhancement of the weights of relevant feature dimensions under adverse environmental conditions such as high system load and severe network congestion. Compared to fixed-weight feature fusion methods, this gating adaptive mechanism can dynamically adjust the contribution ratio of temporal and contextual information in feature representation, improving the ability of spatiotemporally embedded features to perceive changes in the dynamic environment of the cluster.

[0028] Furthermore, the spatiotemporal embedding features are divided into a static semantic feature group and a dynamic environment feature group, and the two groups of features are decoupled to generate decoupled static semantic features and dynamic environment features: ; Specifically, to eliminate the coupling interference between static semantic features and dynamic environmental features in the same vector space, orthogonal rotation transformations are applied to the two sets of features in the spatiotemporal embedding features to generate decoupled feature representations. The decoupled static semantic features... With dynamic environmental characteristics The results are calculated using the formulas described above, where, Indicates the rotation angle. and Let represent the decoupled static semantic features and dynamic environment features, respectively. The orthogonality of the rotation matrix maps the original coupled feature space to two mutually orthogonal subspaces, making the static semantic feature subspace and the dynamic environment feature subspace geometrically perpendicular and independent. Rotation angle. The projection directions of the two sets of features in the new coordinate system are determined by selecting appropriate... The value can maximize the separation degree of the two types of features, so that the subsequent manifold trajectory alignment and similarity calculation performed independently in each subspace are not affected by the different types of features, effectively improving the accuracy of similar job matching.

[0029] Furthermore, the decoupled feature representations of the current and historical tasks are mapped to the manifold space, the state trajectories are aligned, and the similarity between the static semantic trajectory and the dynamic environment trajectory is calculated: Where T represents the current work trajectory, This represents the historical operation trajectory, where σ is the scale parameter.

[0030] Specifically, after mapping the decoupled feature representations of the current task and historical tasks to a non-Euclidean manifold space and performing state trajectory alignment, a similarity calculation formula based on the Gaussian kernel function is used to compare the current task trajectory T with the historical task trajectory. The degree of similarity between them is measured and calculated using the formula mentioned above, where... σ is the squared distance between the current job trajectory and the historical job trajectory after alignment in the manifold space, and σ is a scale parameter that controls the similarity decay rate.

[0031] By calculating similarity, the trajectory distance in manifold space can be transformed into an intuitive similarity score in a non-linear manner: when the distance between two job trajectories approaches zero, the similarity approaches 1, and the similarity decreases smoothly at an exponential rate as the distance increases. The scale parameter σ determines the sensitivity of the similarity to changes in distance. The smaller σ is, the faster the similarity decreases with increasing distance, thus more strictly distinguishing between similar and dissimilar jobs; the larger σ is, the more gradual the similarity distribution, which is suitable for scenarios where job features are more dispersed. By performing the above similarity calculation in manifold space, the similarity distortion problem caused by the non-linearity of job execution paths in Euclidean space is effectively avoided, improving the accuracy of similar historical job retrieval.

[0032] Furthermore, based on system load fluctuations, network disturbances, and queue changes, an environmental negative entropy gain compensation mechanism is introduced to compensate for the static semantic trajectory similarity and the dynamic environmental trajectory similarity, respectively. The compensated decoupled feature representation or the similarity is as follows: ; Where η is the compensation coefficient, and ΔH represents the change in system environmental entropy.

[0033] In one embodiment, to eliminate the impact of dynamic environmental disturbances such as system load fluctuations, network congestion, and queue state changes on feature representation and similarity calculation, a negative entropy gain compensation mechanism is introduced to adaptively compensate for decoupled feature representations or similarities. This represents the decoupling features or similarity values ​​before compensation. This represents the change in system environmental entropy at the current moment, reflecting the degree of uncertainty in the cluster's operating environment. This is the compensation coefficient, used to control the intensity of the influence of entropy change on the compensation result.

[0034] When the cluster is in a chaotic state such as high load and high congestion, the system environment entropy continues to increase. The further away from zero, the greater the compensation factor. The greater the deviation from 1, the more significant the positive gain compensation for the original feature representation or similarity, thus offsetting the weakening of feature reliability by environmental disturbances; conversely, the smaller the deviation, the more stable the cluster operation tends to be. The closer the factor is to zero, the closer it is to 1, and the smaller the compensation magnitude, thus avoiding excessive intervention in the prediction results under stable conditions. By introducing this mechanism, the model can perceive the entropy increase state of the cluster operating environment in real time and adaptively compensate for the prediction process, effectively suppressing the prediction drift caused by environmental uncertainty and enhancing the robustness of the method in complex dynamic environments.

[0035] Furthermore, based on static semantic similarity and dynamic environment similarity, a comprehensive similarity is calculated: in, Represents static similarity. This represents the dynamic similarity, where α is the fusion weight coefficient.

[0036] After negative entropy gain compensation, the corresponding static semantic similarities are obtained in the static semantic feature subspace and the dynamic environment feature subspace, respectively. Similarity to dynamic environment Then, the comprehensive similarity between the current job and historical jobs is calculated by linear weighted fusion. , where α is the fusion weight coefficient, used to balance the contribution ratio of static semantic similarity and dynamic environment similarity in the comprehensive similarity.

[0037] Static semantic similarity and dynamic environment similarity characterize the degree of similarity between jobs from different dimensions: static semantic similarity reflects the consistency of jobs in essential attributes such as user behavior, computing patterns, and resource requests, exhibiting strong stability; dynamic environment similarity reflects the similarity of the cluster environment in which jobs are executed, and is more sensitive to real-time load and network status. By linearly combining the two with a weight coefficient α, the weight ratio of the two types of similarity can be flexibly adjusted according to the actual scenario. While taking into account both the matching of essential job attributes and environmental conditions, it effectively avoids matching bias caused by the limitation of information dimensions of a single similarity index, thereby improving the comprehensiveness and accuracy of candidate historical job retrieval.

[0038] Furthermore, a two-order similarity model is constructed based on the compensated static semantic similarity and dynamic environment similarity, and a set of candidate historical assignments is generated by filtering based on credibility thresholds. The candidate job set is weighted to estimate the running time to generate a basic prediction result, including: using the fusion similarity of each candidate historical job as a weight, the actual running time of the candidate historical jobs is weighted and averaged, with the historical jobs with higher similarity contributing a greater weight.

[0039] In one embodiment, after the candidate historical job set is filtered, the fusion similarity between each candidate historical job and the current job is used as the criterion. As a weight, the actual running times of candidate historical jobs are weighted and averaged to obtain the basic prediction result for the current job. Specifically, historical jobs with higher fusion similarity contribute more weight to the weighted estimation and have a more significant impact on the final prediction result; while historical jobs with lower similarity have their weight reduced accordingly to minimize their interference with the prediction result. Compared to simply taking the average running time of candidate jobs or selecting only the most similar single historical job, the weighted average based on fusion similarity can make full use of the running time information of multiple historical jobs in the candidate set. While retaining the strong guidance of high-similarity historical jobs, it effectively suppresses the biased impact of individual abnormal historical jobs on the prediction result by leveraging the comprehensive constraints of diverse candidate samples, thus achieving a good balance between accuracy and robustness and providing reliable basic prediction results for subsequent multi-level corrections.

[0040] Furthermore, the basic prediction results are corrected in multiple stages through load calibration, congestion calibration, and quota calibration to generate the final runtime estimate: in, Based on the predicted value, For cluster load calibration coefficient, For network congestion calibration coefficient, Adjust the user quota calibration coefficient.

[0041] To further eliminate the impact of the cluster's dynamic environment on the basic prediction results, this embodiment applies three levels of correction—load calibration, congestion calibration, and quota calibration—to the basic prediction values ​​in sequence, generating the final runtime estimate. This is a cluster load calibration factor used to correct the impact of the current overall cluster load level on job execution speed. This is a network congestion calibration factor used to correct the impact of inter-node communication congestion on job execution efficiency. This is a user quota calibration factor used to correct the impact of user resource quota restrictions on actual available computing resources.

[0042] The multi-level correction mechanism decomposes key environmental factors affecting job runtime into independent calibration coefficients, which are then applied to the base prediction value in a product-like manner. This decoupling of the calibration dimensions facilitates independent modeling and updating for different environmental factors. When the cluster load exceeds the preset threshold... Use a value greater than 1 to adjust the estimated time; when network congestion is severe. The corresponding increase reflects the additional latency due to communication overhead; when user quota limitations result in actual resource allocation being lower than the requested amount, The corresponding estimates are then adjusted. Through the synergistic effect of the three-level calibration, the final estimate can comprehensively reflect the combined impact of the cluster's real-time operating status on job execution time, effectively improving the accuracy of the estimates in complex dynamic environments.

[0043] Furthermore, after the task is completed, the model parameters are iteratively updated using gradient descent, with the prediction error as the loss function. in, For the updated model parameters, For learning rate, For the loss function with respect to gradient, The estimated value output by the model. This refers to the actual running time of the task.

[0044] In one embodiment, after the job is completed, the estimated value output by the model is used. Compared with the actual running time of the operation The prediction error between the two is used to construct a loss function, and gradient descent is used to iteratively update the model parameters online. These are the current model parameters. For learning rate, For the loss function with respect to the parameters The gradient reflects the direction and magnitude of the change in the loss function under the current parameter configuration.

[0045] Each time a job completes and its actual runtime is obtained, a parameter update is triggered using the prediction error of that sample as a supervision signal. This drives the model parameters to adjust in the negative direction of the loss function gradient, gradually reducing the systematic bias in subsequent predictions. Compared to offline batch training, this online update strategy enables the model to track the dynamic changes in cluster workload distribution in real time, adapting promptly to the evolution of user behavior patterns and the long-term drift of the cluster environment. This maintains high prediction accuracy during continuous system operation and effectively solves the problem of performance degradation in static models due to outdated training data.

[0046] The following specific embodiments illustrate this solution: Taking a large-scale parallel molecular dynamics simulation job as an example (such as the VASP task), the resource scale is: 8 nodes, a total of 512 cores (64 cores / node), and 128GB of memory. The real-time system status is as follows: the current cluster is under high load (CPU load 70%), there is I / O contention in the parallel file system (latency 50ms), and there is local micro-burst congestion in the network (30%).

[0047] First, four-dimensional feature extraction and normalization are performed. The original feature settings (derived from system monitoring and historical statistics) are: static semantic features. = [80, 40, 50, 30]; Dynamic environment characteristics: = [70, 50, 60, 40]. Historical statistical range (used for normalization): = [0, 0, 0, 0]; = [100, 100, 100, 100]. According to the formula: ; The normalized result is obtained: = [0.80, 0.40, 0.50, 0.30]; = [0.70, 0.50, 0.60, 0.40]; Perform multidimensional adaptive weight calculation and set historical variance: = [0.20, 0.25, 0.33, 0.50]; Prior importance: = [0.9, 0.8, 0.7, 0.6]; Let λ = 0.5.

[0048] According to the formula: ; calculate: = [5.00, 4.00, 3.03, 2.00].

[0049] = 0.5*[5, 4, 3.03, 2] + 0.5*[0.9, 0.8, 0.7, 0.6]= [2.95, 2.40, 1.865, 1.30].

[0050] Gated adaptive spatiotemporal embedding, the feature enhancement coefficients are calculated through gating units: The gated output vector is obtained as: G = [0.76, 0.69, 0.62, 0.55]. Simultaneously, based on the previously obtained adaptive weight vector... The gating results are weighted to obtain the enhanced feature representation: = [2.26, 1.66, 1.16, 0.72].

[0051] Orthogonal spatial rotation decoupling: Let θ = 45°; cosθ = sinθ = 0.707. According to the formula: = cosθ·f_s + sinθ·f_d; = -sinθ·f_s + cosθ·f_d; calculate: = 0.707*(Fs + Fd)= 0.707*[1.5, 0.9, 1.1, 0.7]= [1.06, 0.64, 0.78, 0.49]; = 0.707*(Fd - Fs)= 0.707*[-0.10, 0.10, 0.10, 0.10]= [-0.07, 0.07, 0.07, 0.07].

[0052] Non-Euclidean manifold trajectory alignment: Calculating Euclidean distance (local approximation of the manifold): = (1.06-1.00) 2 + (0.64-0.60) 2 + (0.78-0.75) 2 + (0.49-0.45) 2 = 0.0036 + 0.0016 + 0.0009 + 0.0016 = 0.0077 = (1.06-1.10) 2 + (0.64-0.70) 2 + (0.78-0.80) 2 + (0.49-0.50) 2 = 0.0016 + 0.0036 + 0.0004 + 0.0001 = 0.0057 = (1.06-0.95) 2 + (0.64-0.55)2 + (0.78-0.70) 2 + (0.49-0.40) 2 = 0.0121 + 0.0081 + 0.0064 + 0.0081 = 0.0347 Let σ = 0.1, Sim = exp(-d 2 / (2σ 2 Since 2σ 2 = 2×0.1 2 = 0.02, resulting in: Sim1 = exp(-0.0077 / 0.02) = exp(-0.385) ≈ 0.680. Sim2 = exp(-0.0057 / 0.02) = exp(-0.285) ≈ 0.752. Sim3 = exp(-0.0347 / 0.02) = exp(-1.735) ≈ 0.176.

[0053] Environmental negative entropy gain compensation: Assume the system entropy change is: = 0.05η = 0.8, compensation: Sim' = Sim × (1 + ηΔH) = Sim × (1+0.04).

[0054] get: Sim1' = 0.680 × 1.04 ≈ 0.707. Sim2' = 0.752 × 1.04 ≈ 0.782. Sim3' = 0.176 × 1.04 ≈ 0.183.

[0055] Two-order similarity fusion and filtering: Let α = 0.6, and the final similarity (assuming static and dynamic similarity are consistent): Sim_mix = Sim'. The filtering threshold is 0.5. Since Sim1'=0.707 and Sim2'=0.782 are greater than the threshold of 0.5, while Sim3'=0.183 is less than the threshold of 0.5, T1 and T2 are retained as candidate samples.

[0056] Level 3 calibration and final prediction: Historical run time: T1=10 hours, T2=11 hours.

[0057] Weighted average: = (0.707×10 + 0.782×11) / (0.707+0.782) = (7.07 + 8.602) / 1.489 ≈ 10.53 hours Calibration parameters: = 1.05; = 1.02; = 0.98.

[0058] Final result: = 10.53 × 1.05 × 1.02 × 0.98 ≈ 11.05 hours Online feedback: Actual running time 11.2 hours, relative error = |11.20 - 11.05| / 11.20 ≈ 1.34%.

[0059] In summary, firstly, multiple features are extracted from the task to be predicted to form a multidimensional feature vector and normalized. Then, adaptive weights are calculated based on the statistical characteristics of each feature in historical samples to generate a weighted feature representation. Next, a gating vector is constructed and fused with the weighted feature representation to generate a spatiotemporal embedded feature. Finally, the spatiotemporal embedded feature is decoupled to eliminate mutual interference between features and generate a decoupled feature representation.

[0060] Based on this, the decoupled feature representations of the current job and historical jobs are mapped to non-Euclidean manifold spaces respectively. State trajectory alignment is performed independently in the static semantic feature subspace and the dynamic environmental feature subspace, and the corresponding similarity is calculated. A negative entropy gain compensation mechanism is introduced to compensate for system load fluctuations and network disturbances, and the above similarity is adaptively compensated. Then, the two-way compensated similarities are fused to construct a two-order similarity model, and a candidate set of historical jobs is generated based on the confidence threshold. Finally, a weighted running time estimation is performed based on the candidate set. After multiple levels of correction such as load calibration, congestion calibration, and quota calibration, the final prediction value is generated. After the job is executed, the actual running time and prediction error are used as feedback signals to update the model parameters online, so as to continuously optimize the prediction accuracy.

[0061] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A method for estimating the runtime of computing cluster jobs, characterized in that, include: Multiple features of the task to be predicted are extracted to form a multidimensional feature vector, which is then normalized. Calculate the adaptive weights of each feature dimension of the normalized multidimensional feature vector to generate a weighted feature representation; A gated vector is constructed and fused with the weighted feature representation to generate a spatiotemporal embedding feature; The spatiotemporal embedding features are decoupled to generate decoupled feature representations; Align the decoupled feature representations of the current job and historical jobs, calculate their similarity, and compensate for the decoupled feature representations or the similarity. Calculate the comprehensive similarity and generate a candidate historical job set. Based on the candidate historical job set, generate basic prediction results. Perform multi-level corrections on the basic prediction results to generate the final runtime estimate.

2. The method for estimating the runtime of computing cluster jobs as described in claim 1, characterized in that, The multidimensional feature vector includes extracted user features, job semantic features, resource request features, and dynamic environment features. The feature values ​​of each dimension are then normalized to generate normalized feature values. in, These are the original eigenvalues. and These are the minimum and maximum values ​​of this feature obtained from historical statistics, respectively. Based on the statistical variance, prior importance, and discriminative contribution of historical data, a multidimensional adaptive weight calculation model is constructed to calculate the adaptive weights: ； wherein is a feature of the historical variance; is a feature of the importance weight; λ is a weight adjustment coefficient.

3. The method for estimating the runtime of a computing cluster job of claim 1, wherein, A gating vector is constructed based on system load, queue length, network congestion, and time decay factor, and then fused with the weighted feature representation to generate spatiotemporal embedding features: ； wherein, represents an activation function, is a gating weight matrix, is a bias term, and is an input feature vector.

4. The method for estimating the runtime of computing cluster jobs as described in claim 1, characterized in that, The spatiotemporal embedding features are divided into a static semantic feature group and a dynamic environment feature group, and the two groups of features are decoupled to generate decoupled static semantic features and dynamic environment features: ； ； in, Indicates the rotation angle. and These represent the static semantic features and dynamic environmental features after decoupling, respectively.

5. The method for estimating the runtime of computing cluster jobs as described in claim 4, characterized in that, The decoupled feature representations of the current and historical tasks are mapped to the manifold space, the state trajectories are aligned, and the similarity between the static semantic trajectory and the dynamic environment trajectory is calculated: ； Where T represents the current work trajectory, This represents the historical operation trajectory, where σ is the scale parameter.

6. The method for estimating the runtime of computing cluster jobs as described in claim 5, characterized in that, An environmental negative entropy gain compensation mechanism is introduced based on system load fluctuations, network disturbances, and queue changes to compensate for the static semantic trajectory similarity and the dynamic environmental trajectory similarity, respectively. The compensated decoupled feature representation or similarity is as follows: ； Where η is the compensation coefficient, and ΔH represents the change in system environmental entropy.

7. The method for estimating the runtime of computing cluster jobs as described in claim 1, characterized in that, Calculate the comprehensive similarity based on static semantic similarity and dynamic environment similarity: ； in, Represents static similarity. This represents the dynamic similarity, where α is the fusion weight coefficient.

8. The method for estimating the runtime of computing cluster jobs as described in claim 1, characterized in that, A two-order similarity model is constructed based on the compensated static semantic similarity and dynamic environment similarity, and a candidate historical assignment set is generated by filtering based on the credibility threshold. The candidate job set is weighted to estimate the running time to generate a basic prediction result, including: using the fusion similarity of each candidate historical job as a weight, the actual running time of the candidate historical jobs is weighted and averaged, with the historical jobs with higher similarity contributing a greater weight.

9. The method for estimating the runtime of computing cluster jobs as described in claim 1, characterized in that, The basic prediction results are corrected in multiple stages through load calibration, congestion calibration, and quota calibration to generate the final runtime estimate. ； in, Based on the predicted value, For cluster load calibration coefficient, For network congestion calibration coefficient, Adjust the user quota calibration coefficient.

10. The method for estimating the runtime of computing cluster jobs as described in claim 1, characterized in that, After the task is completed, the model parameters are iteratively updated using gradient descent, with the prediction error as the loss function. ； in, For the updated model parameters, For learning rate, For the loss function with respect to gradient, The estimated value output by the model. This refers to the actual running time of the task.

Images