Method and system for automatic orchestration of internet operation tasks based on reinforcement learning

Abstract

The application relates to the field of controlled distributed computing, and discloses an Internet operation task automatic arrangement method and system based on reinforcement learning, which comprises the following steps: constructing a directed acyclic graph and a topological adjacency matrix according to the dependency relationship of a to-be-processed task node; collecting controlled system resource variables and response time delays, and coding node attributes into a multi-dimensional state vector input policy network; outputting a scheduling probability distribution by using the policy network, and determining a scheduling strategy based on a Markov decision process; then, the topological adjacency matrix is analyzed to determine a subtask node set, a subset value estimate is calculated as a credit compensation, and the policy network weight parameters are corrected in combination with a physical reward; the application internalizes the topological adjacency relationship into a gradient descent guiding force, eliminates the dimensional disconnection between time sequence exploration and logical topology, realizes endogenous perception of future topology dividends, improves system convergence stability, and avoids key link blockage without increasing additional computing power consumption.

Description

Technical Field

[0001] This invention relates to the field of controlled distributed computing technology, specifically to a method and system for automatically orchestrating Internet operation and maintenance tasks based on reinforcement learning. Background Technology

[0002] Currently, in the process of automated management of large-scale Internet infrastructure, the scheduling of operation and maintenance tasks usually adopts reinforcement learning models. Such solutions use Markov decision processes to describe the interaction between task sequences and computing resources, and use the cumulative feedback in continuous decision steps to learn the sorting strategy, thereby dealing with the resource allocation imbalance caused by business load fluctuations. The inspection, change and backup tasks in Internet operation and maintenance systems have rigid dependencies. When dealing with such task flows with deep topological constraints, the value evaluation mechanism of existing reinforcement learning methods is limited by the temporal dimension. When dealing with task nodes at bottleneck positions, the immediate feedback often only reflects the instantaneous resource occupation, while the real value of the node lies in unlocking the set of subsequent subtasks. Under this mechanism, the agent needs to undergo large-scale random trial and error in order to perceive the long-term gains brought by the topological structure through the backpropagation of feedback signals.

[0003] To address these issues, common improvements include adding offline training loops or introducing graph neural networks to fit spatial relationships. However, these methods incur significant computational overhead and do not fundamentally resolve the perceptual deficiencies of the update equations in the topological dimension. Another common approach is to use action masks at the output to filter actions that do not meet dependency conditions. However, this only addresses execution-level constraints and cannot internalize the logical topological weights between tasks into the gradient guidance of the policy network, leading to the model's susceptibility to local optima in dynamic environments. In large-scale distributed environments, increasing sampling density is insufficient to avoid the inherent contradiction between topological logic and temporal learning. The software-level credit allocation logic faces challenges, for example… Chinese invention patent application CN120762855A discloses a real-time scheduling method and system for cloud-edge collaborative workflow based on deep reinforcement learning. It uses LSTM to predict server status and combines it with clustering algorithms to divide tasks. It belongs to the prediction matching discrete decision-making method. In the preprocessing stage, task dependencies are clustered and dimensionality is reduced. The reinforcement learning loop cannot directly perceive the dynamic evolution of the task topology. Under conditions of limited resources or drastic bandwidth fluctuations, the prediction model becomes inaccurate due to environmental noise. It lacks explicit modeling of topology benefits. The algorithm has difficulty penetrating the long-link credit lag barrier, which leads to the marginalization of critical path nodes in resource allocation, causing task blockage or cost overruns.

[0004] Therefore, how to construct a policy update logic that incorporates explicit injection of topological credit, so that the policy network can perform weight iteration based on the topological gain of the task dependency graph, thereby improving the convergence stability of the model in dynamic environments, is the technical problem to be solved by this invention. Summary of the Invention

[0005] To address the issue that when reinforcement learning is used to orchestrate operational tasks with rigid dependencies, the mismatch between the credit allocation mechanism and the logical topology leads to poor convergence stability and a tendency to get trapped in local optima.

[0006] This invention proposes a method and system for automatic orchestration of internet operation and maintenance tasks based on reinforcement learning, comprising the following steps: Step S1: Obtain the attribute feature set of the task nodes to be processed in the controlled distributed computing system, and construct a directional acyclic graph representing the node dependency structure based on the control flow and data flow dependency relationship between each task node to be processed, and convert it into a two-dimensional topological adjacency matrix. Step S2: Real-time acquisition of resource state variables and response latency of the controlled distributed computing system, and encoding them together with the priority and estimated time requirements of each task node to be processed into a multi-dimensional state vector, which is then input into the policy network. Step S3: Utilize the policy network to output the scheduling probability distribution of each task node to be processed within the legal candidate set, and perform action sampling on the scheduling probability distribution based on the Markov decision process to determine the scheduling strategy for the current decision cycle. Step S4: Obtain the physical immediate reward feedback, and parse the two-dimensional topological adjacency matrix to determine the set of subtask nodes that have been decoupled from their previous dependencies due to the scheduling policy. Calculate the value estimate of the subtask node set using the policy network, and use it as a compensation for the credit allocation lag problem. This value estimate, along with the physical immediate reward, participates in the update of the policy network's weight parameters.

[0007] Preferably, step S2 specifically includes: performing one-hot encoding on the task nodes to be processed for different business types, and concatenating them with the normalized task waiting time and historical resource consumption rate using tensors; using a self-attention mechanism to calculate the weight allocation coefficients of each dimension feature in the multidimensional state vector, and by increasing the feature expression weights of the critical path nodes, constructing a mapping relationship between the multidimensional state vector and the system load fluctuation and long-link business dependency features.

[0008] Preferably, step S3 specifically includes: step S31, using the real-time constructed node dependency graph to perform legality filtering on the scheduling probability distribution to remove illegal nodes that violate the logical constraints of the directed acyclic graph; step S32, combining resource stability, timeliness and dependency conflict penalty factor to construct a scalar weighted feedback function, and under the environment of satisfying the rigid constraints of control flow, determining the scheduling priority according to the out-degree weight of each task node in the directed acyclic graph.

[0009] Preferably, step S4, which uses a policy network to calculate the value estimate of the set of subtask nodes, includes: parsing the two-dimensional topological adjacency matrix, extracting the number of out-degrees and the depth of subsequent paths for each task node in the directed acyclic graph; calculating the topological gain factor based on the number of out-degrees and the depth of subsequent paths, and using the topological gain factor to compensate for the value of the physical instant reward, thereby obtaining the corrected action value evaluation parameters.

[0010] Preferably, the weight parameter update in step S4 includes: constructing a value loss function based on topology look-ahead capability, the value loss function being used to measure the deviation between the predicted value under the current strategy and the target value incorporating topology gain factors; iteratively updating the parameters of the strategy network using the gradient descent algorithm, so that the scheduling strategy matches the real-time load changes of the controlled distributed computing system within a preset convergence threshold range.

[0011] Preferably, the attribute feature set of each task node to be processed includes node type, estimated time consumption, memory requirement, CPU utilization, and disk I / O intensity.

[0012] Preferably, after generating the scheduling strategy, the method further includes: real-time monitoring of the state deviation of the scheduling strategy during execution; when the fluctuation amplitude of the multi-dimensional state vector exceeds a preset threshold, online correction of the strategy network is performed; the scheduling probability distribution is perturbed and superimposed based on the exploration noise of the current controlled distributed computing system, and the scheduling path that avoids the resource fragmentation area is searched again based on the superposition result.

[0013] Preferably, the controlled distributed computing system includes a cloud computing resource layer, a data center logical cluster, and edge processing nodes.

[0014] Preferably, the construction of the directed acyclic graph in step S1 includes: extracting the triggering logic and data exchange interface between each task node by traversing the metadata repository of the controlled computing environment, and using a depth-first search algorithm to identify and eliminate dependency loops to ensure the logical unidirectionality of the directed acyclic graph.

[0015] An automatic orchestration system for internet operations and maintenance tasks based on reinforcement learning, which implements an automatic orchestration method for internet operations and maintenance tasks based on reinforcement learning, includes: The topology modeling module is used to obtain the attribute feature set of the task nodes to be processed in the controlled distributed computing system, and construct a directional acyclic graph representing the node dependency structure and the corresponding two-dimensional topological adjacency matrix based on the control flow and data flow dependencies between the task nodes to be processed. The environmental status observation module is used to collect resource status variables and response latency of the controlled distributed computing system in real time, and encode the priority, estimated time consumption requirements and collected data of each task node into a multi-dimensional state vector. The orchestration decision module, connected to the environmental state observation module, is used to input multi-dimensional state vectors into the policy network, so that the policy network can output the scheduling policy of each task node to be processed. The value network correction module is connected to the topology modeling module and the orchestration decision module, respectively. It is used to obtain the physical immediate reward of the feedback, parse the two-dimensional topological adjacency matrix to determine the set of sub-task nodes, and then use the policy network to calculate the compensation amount corresponding to the set of sub-task nodes. Based on the compensation amount and the physical immediate reward, the weight parameters of the policy network are corrected.

[0016] The beneficial effects of this invention are: 1. In the automatic orchestration method for Internet operation and maintenance tasks, a Bellman objective explicit dimensionality reduction injection mechanism based on the topological adjacency matrix is ​​constructed to solve the problem of the separation between temporal exploration and logical topological structure dimension in reinforcement learning. During the policy network update process, the set of subtasks that are released from pre-dependencies due to the completion of the current task is determined by reading the two-dimensional topological adjacency matrix in memory. The future value prediction of the subtask set is directly accumulated to the current temporal differential update target in the form of topological credit compensation. This processing method makes the update of policy network weights no longer blindly rely on occasional feedback on the time axis, but directly absorbs the future topological value unlocked in a single iteration. From a physical mechanism perspective, the credit backhaul path of critical path tasks is shortened, and the convergence speed of the system in complex operation and maintenance dependency scenarios is improved.

[0017] 2. By dynamically coupling a directed acyclic dependency graph with a multi-objective composite reward function, the system achieves coordinated optimization of task orchestration security and resource utilization. Within each decision step, the system uses the real-time constructed task dependency graph to filter and generate a set of legal candidates. Combined with scalarized weighted feedback that includes resource stability, task timeliness, and dependency conflict penalties, the system guides the agent to find the best within the legal action space. This mechanism ensures that the orchestration sequence automatically identifies and prioritizes bottleneck nodes while satisfying the rigid constraints of data flow and control flow, thereby eliminating deadlocks or cascading failures caused by dependency conflicts and ensuring the operational continuity of large-scale Internet infrastructure under high-concurrency task conditions.

[0018] 3. By leveraging multidimensional feature encoding in the state space and online iterative learning of the policy network, the robustness of the operation and maintenance orchestration scheme to environmental fluctuations is enhanced. By vectorizing features such as task type, estimated duration, resource requirements, and waiting time, the agent can perceive real-time changes in system load and dynamically adjust task priorities based on long-term cumulative rewards. This approach avoids resource fragmentation or blockage of important tasks caused by traditional static rules under sudden business traffic. It enables the system to have global perception capabilities that penetrate complex task chains, achieving continuous improvement in operation and maintenance orchestration quality as the environment evolves without increasing additional graph computing hardware power. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the decision-making process for topological credit compensation and strategy updates in this invention. Figure 2 This is a diagram showing the architecture of the topology-aware operation and maintenance task orchestration system of this invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] An automatic orchestration method for internet operations and maintenance tasks based on reinforcement learning includes the following steps: Step S1: Obtain the attribute feature set of the task nodes to be processed in the controlled distributed computing system, and construct a directional acyclic graph representing the node dependency structure based on the control flow and data flow dependency relationship between each task node to be processed, and convert it into a two-dimensional topological adjacency matrix. Step S2: Real-time acquisition of resource state variables and response latency of the controlled distributed computing system, and encoding them together with the priority and estimated time requirements of each task node to be processed into a multi-dimensional state vector, which is then input into the policy network. Step S3: Utilize the policy network to output the scheduling probability distribution of each task node to be processed within the legal candidate set, and perform action sampling on the scheduling probability distribution based on the Markov decision process to determine the scheduling strategy for the current decision cycle. Step S4: Obtain the physical immediate reward feedback, and parse the two-dimensional topological adjacency matrix to determine the set of subtask nodes that have been decoupled from their previous dependencies due to the scheduling policy. Calculate the value estimate of the subtask node set using the policy network, and use it as a compensation for the credit allocation lag problem. This value estimate, along with the physical immediate reward, participates in the update of the policy network's weight parameters.

[0023] Preferably, step S2 specifically includes: performing one-hot encoding on the task nodes to be processed for different business types, and concatenating them with the normalized task waiting time and historical resource consumption rate using tensors; using a self-attention mechanism to calculate the weight allocation coefficients of each dimension feature in the multidimensional state vector, and by increasing the feature expression weights of the critical path nodes, constructing a mapping relationship between the multidimensional state vector and the system load fluctuation and long-link business dependency features.

[0024] Preferably, step S3 specifically includes: step S31, using the real-time constructed node dependency graph to perform legality filtering on the scheduling probability distribution to remove illegal nodes that violate the logical constraints of the directed acyclic graph; step S32, combining resource stability, timeliness and dependency conflict penalty factor to construct a scalar weighted feedback function, and under the environment of satisfying the rigid constraints of control flow, determining the scheduling priority according to the out-degree weight of each task node in the directed acyclic graph.

[0025] Preferably, step S4, which uses a policy network to calculate the value estimate of the set of subtask nodes, includes: parsing the two-dimensional topological adjacency matrix, extracting the number of out-degrees and the depth of subsequent paths for each task node in the directed acyclic graph; calculating the topological gain factor based on the number of out-degrees and the depth of subsequent paths, and using the topological gain factor to compensate for the value of the physical instant reward, thereby obtaining the corrected action value evaluation parameters.

[0026] Preferably, the weight parameter update in step S4 includes: constructing a value loss function based on topology look-ahead capability, the value loss function being used to measure the deviation between the predicted value under the current strategy and the target value incorporating topology gain factors; iteratively updating the parameters of the strategy network using the gradient descent algorithm, so that the scheduling strategy matches the real-time load changes of the controlled distributed computing system within a preset convergence threshold range.

[0027] Preferably, the attribute feature set of each task node to be processed includes node type, estimated time consumption, memory requirement, CPU utilization, and disk I / O intensity.

[0028] Preferably, after generating the scheduling strategy, the method further includes: real-time monitoring of the state deviation of the scheduling strategy during execution; when the fluctuation amplitude of the multi-dimensional state vector exceeds a preset threshold, online correction of the strategy network is performed; the scheduling probability distribution is perturbed and superimposed based on the exploration noise of the current controlled distributed computing system, and the scheduling path that avoids the resource fragmentation area is searched again based on the superposition result.

[0029] Preferably, the controlled distributed computing system includes a cloud computing resource layer, a data center logical cluster, and edge processing nodes.

[0030] Preferably, the construction of the directed acyclic graph in step S1 includes: extracting the triggering logic and data exchange interface between each task node by traversing the metadata repository of the controlled computing environment, and using a depth-first search algorithm to identify and eliminate dependency loops to ensure the logical unidirectionality of the directed acyclic graph.

[0031] An automatic scheduling system for internet operations and maintenance tasks based on reinforcement learning, comprising: The topology modeling module is used to obtain the attribute feature set of the task nodes to be processed in the controlled distributed computing system, and construct a directional acyclic graph representing the node dependency structure and the corresponding two-dimensional topological adjacency matrix based on the control flow and data flow dependencies between the task nodes to be processed. The environmental status observation module is used to collect resource status variables and response latency of the controlled distributed computing system in real time, and encode the priority, estimated time consumption requirements and collected data of each task node into a multi-dimensional state vector. The orchestration decision module, connected to the environmental state observation module, is used to input multi-dimensional state vectors into the policy network, so that the policy network can output the scheduling policy of each task node to be processed. The value network correction module is connected to the topology modeling module and the orchestration decision module, respectively. It is used to obtain the physical immediate reward of the feedback, parse the two-dimensional topological adjacency matrix to determine the set of sub-task nodes, and then use the policy network to calculate the compensation amount corresponding to the set of sub-task nodes. Based on the compensation amount and the physical immediate reward, the weight parameters of the policy network are corrected.

[0032] Example 1: In dealing with long-link rigid constraints, when a system faces a massive flow of internet operation and maintenance tasks with deep rigid dependencies continuously generated by a large-scale cloud computing data center under the impact of sudden high-concurrency business, the traditional reinforcement learning scheduling framework, due to the lag in credit allocation on the time axis of the Markov decision process, causes the agent to retreat to a state of prioritizing short-term independent but unrelated small tasks when faced with tasks such as the expansion of large-scale database clusters that are at the choke point of the critical link and have long processing time. This leads to the fragmentation of computing resources and cascading blockage of critical links; the controlled distributed computing system obtains the attribute feature set of the task nodes to be processed, traverses and proposes... The system extracts the control flow and data flow triggering logic between each task node to construct a directed acyclic graph (AAG) representing the node dependency structure. This AAG is then converted into a two-dimensional topological adjacency matrix stored in memory. The system collects the resource state variables and response latency of the controlled distributed computing system in real time. These variables, along with the priority and estimated time requirements of each task node, are encoded into a multi-dimensional state vector and input into the policy network. The AAG is used to filter and generate a set of legitimate candidates. The policy network outputs the scheduling probability distribution of each task node within this set of legitimate candidates. Based on a Markov decision process, the scheduling probability distribution is sampled to determine the scheduling strategy for the current decision cycle.

[0033] When the scheduling policy instructs the execution of time-consuming task nodes acting as choke points in a controlled computing environment, the system obtains the immediate physical reward, cuts off the value feedback path dependent on the next temporal state evaluation, directly parses the two-dimensional topological adjacency matrix, extracts the out-degree number and subsequent path depth of each task node in the directed acyclic graph to determine the set of sub-task nodes that have been freed from pre-dependencies due to the scheduling policy. The system calls the policy network to calculate the value prediction of this set of sub-task nodes, using it as compensation for the credit allocation lag problem. This value, along with the immediate physical reward, participates in the update of the policy network's weight parameters. Based on the temporal difference update principle in reinforcement learning, the network prediction bias is quantified. Specifically, during this calculation process, the policy network activates its separately configured deterministic evaluation layer, receives the attribute tensor sequence of the aforementioned set of sub-task nodes as the input feature dimension, and then uses the feedforward neuron matrix to perform linear weighted summation and nonlinear activation scaling on the topological gain features of each node. Finally, a scalar prediction value corresponding to the input tensor is generated at the forward propagation output of the network model. Let the current task node scheduling correspond to the system state as... The physical reward for collecting is The system state for the next decision cycle is: The estimated value of the set of subtask nodes output by the strategy network computation is: According to the formula Calculate and update the metrics, where, The representative characterization strategy evaluates the time-series difference error. This represents a forward discount factor with values ​​between 0 and 1, and the accumulation of forward returns over a convergent time series. and The representative policy network outputs a state value scalar benchmark under corresponding state feature inputs, obtains the calculated temporal difference error, constructs a minimum mean squared error loss function, and uses an adaptive moment estimation algorithm for backpropagation and iterative update of the policy network tensor weight values. To ensure the absolute consistency of the physical dimensions of each variable in the temporal difference error update formula, before actually executing the algebraic addition, the system pre-extracts the average task response latency of the current controlled computing environment as a benchmark reference equivalent using a built-in normalization algorithm. The spatial geometric scalar obtained from matrix analysis is proportionally converted into a time benefit equivalent scale with the same physical instantaneous reward scale, thereby eliminating the underlying computational logic conflicts caused by the direct superposition of features of different dimensions. The spatial logic is limited by the two-dimensional topological adjacency matrix. The relationship provides a priori target set for value assessment that is free from random trial and error. The value prediction of this priori target set eliminates the time delay constraint of state value update. In a single iteration, the system reconstructs the future topology unlocking gain into gradient descent guidance. Within the single loss function update architecture, it reconciles the credit allocation conflict between the short-term high resource penalty of time-consuming bottleneck tasks and the long-term high topology unlocking gain. The above-mentioned flow mechanism reconstructs the discrete control flow and data flow dependency structure into compensation parameters in the network parameter update formula, so that the orchestration decision transforms the global topology constraint into a compensation force driving the iteration of policy network parameters. The controlled distributed computing system exhibits convergence stability for deep concurrent task chains in dynamic load environments, while avoiding system deadlock caused by prior dependency conflicts.

[0034] Example 2: A distributed computing cluster simulation platform is configured with 500 virtual computing nodes to generate concurrent business task flows containing directed acyclic graph control flow. A Gaussian distributed time delay with a mean of 120ms and a variance of 25ms is superimposed on the node resource status reporting link to simulate measurement disturbances. When the concurrent task arrival rate is in a sudden fluctuation state of 10000QPS, the decision sampling period is set to 50ms to balance the real-time performance of scheduling and the computing power load of policy network inference. A control group using a standard near-end policy optimization algorithm with physical instant reward based on latency and an experimental group deploying a topology gain compensation mechanism are set. The maximum dependency depth of the directed acyclic graph task flow is selected as the gradient variable and set to 5, 15, and 30, respectively. In the tier with a depth of 15, due to the existence of communication disturbances, the variance of the state value estimation fluctuation recorded by the control group is 4.62. The experimental group extracts the target database expansion task node with an out-degree of 8 and subsequent paths by analyzing the two-dimensional topology adjacency matrix. With a depth of 12, the calculated topology gain factor is 2.45. This topology gain factor corrects the physical instant reward of 0.5 to an action value assessment parameter of 1.72, and the variance of the state value estimation in the experimental group converges to 0.85. When the task dependency depth spans the above three gradients, the average completion time of the critical path in the control group is recorded as 15.4s, 84.7s, and 342.1s, while the average completion time of the critical path in the experimental group is recorded as 14.8s, 31.2s, and 56.5s. When the dependency depth is 30, the control group experiences cascading blocking, with a resource fragmentation rate of 78.4%. The experimental group guides action sampling through a value compensation mechanism, with a resource fragmentation rate of 18.2%. When the dependency depth exceeds 45, the parsing time of the two-dimensional topology adjacency matrix increases, the single action decision latency exceeds 50ms, and the system resource scheduling efficiency shows a downward trend. The data establishes that a path depth of no more than 40 is the preferred working range for this orchestration method.

[0035] Example 3: This example combines Figures 1 to 2 This document describes a method and system for automatically orchestrating internet operations and maintenance tasks based on reinforcement learning. Figure 1As shown, steps S1 to S4 are as follows: Step S1: Obtain the attribute feature set of the task nodes to be processed in the controlled distributed computing system, the control flow and data flow dependencies between the task nodes, construct a directional acyclic graph representing the node dependency structure, and convert it into a 2D topological adjacency matrix. Step S2: Collect the resource state variables and response latency of the controlled distributed computing system in real time, and encode them together with the priority and estimated time requirement of each task node into a multi-dimensional state vector, which is then input into the policy network. Step S3: Utilize the policy network to output the scheduling probability distribution of each task node in the legal candidate set. The Markov decision process samples the scheduling probability distribution to determine the scheduling policy for the current decision cycle. Step S4: Obtain the feedback physical immediate reward, parse the 2D topological adjacency matrix to determine the set of sub-task nodes that have been released from prior dependencies due to the scheduling policy, and use the policy network to calculate the value estimate of the sub-task node set as a credit compensation amount, which, together with the physical immediate reward, participates in the update of the policy network's weight parameters.

[0036] like Figure 2 As shown, the system is divided into a topology logic management domain, an infrastructure execution domain, and a core intelligent orchestration node. The topology logic management domain includes a metadata repository, a directed acyclic graph model, and a 2D topology adjacency matrix. The directed acyclic graph model maps node dependency structures to the orchestration decision module and policy network within the core intelligent orchestration node. The infrastructure execution domain includes a cloud computing resource layer, a data center logical cluster, and edge processing nodes. The cloud computing resource layer transmits resource state variables and response latency to the environment state observation module of the core intelligent orchestration node. The environment state observation module points to the orchestration decision module and policy network. The orchestration decision module and policy network transmit scheduling policies and action sampling to the data center logical cluster. The edge processing node transmits physical instant rewards to the value network correction module. The value network correction module updates weight parameters and performs topology credit compensation to the orchestration decision module and policy network.

[0037] Example 4: When scheduling internet operation and maintenance tasks with multi-dimensional temporal constraints in a distributed computing cluster, the heterogeneity of task node feature dimensions and the lack of a topology weight evaluation benchmark lead to numerical instability in the policy network during credit compensation calculation. The controlled distributed computing system obtains the business type attributes of the task nodes to be processed, generates discrete feature vectors using a one-hot encoding algorithm, and synchronously collects the waiting time of each task node in the task queue and the average processor utilization rate within a preset historical time window. The statistical step size of the average processor utilization rate is set to 10 seconds. The system uses a maximum-minimum normalization algorithm to convert the waiting time and average processor utilization rate into continuous feature scalars distributed in the interval between 0 and 1. The discrete feature vectors and continuous feature scalars are then concatenated into tensors to construct a multi-dimensional state vector, which is then used in conjunction with a neural network. The feature space mapping principle extracts and concatenates the multidimensional state vector, and uses parameterized settings of the fully connected feature mapping layer to uniformly project it into a continuous latent space with fixed dimensions, generating a node representation sequence composed of isomorphic feature elements, eliminating the difference in dimensionality and scale between discrete encoding and continuous scalars. The above multidimensional state vector is input into the self-attention mechanism module built into the policy network. The self-attention mechanism module calculates the interaction weights between features of different dimensions in the multidimensional state vector. The system calculates the dot product of the query vector and the key vector obtained by linear transformation of the multidimensional state vector. After dividing the result of the dot product by the scaling factor of the square root value of the feature dimension, the weight allocation coefficient is determined by the normalized exponential function. The system performs weighted summation on the value vector according to the weight allocation coefficient, generating a reconstructed state vector representing the long-link business dependency features and system load fluctuations.

[0038] After determining the set of subtask nodes whose pre-dependencies have been removed due to the scheduling strategy, the system parses the two-dimensional topological adjacency matrix in memory. The system accumulates the number of non-zero elements in the corresponding row dimension of the target task node to determine its out-degree. It then traverses along directed edges to find the longest connected subgraph from the target task node to a node without successors, determining the depth of subsequent paths. To determine the topological gain factor, preset first and second benchmark coefficients are retrieved. The first benchmark coefficient is set to 0.6, and the second benchmark coefficient is set to 0.4. These benchmark coefficients are based on the asymmetric consumption of computational resources by breadth-first and depth-first traversals in graph theory. The calibration process is fixed, continuously injecting a set of standard tasks with gradient-distributed concurrency dependency depth into an independent, isolated test cluster. The cluster's memory page-turning frequency and CPU bus latency are recorded. For the frequencies and latency indicators with absolute physical dimensions such as Hertz and microseconds obtained from the aforementioned records, the system calls a built-in inverse proportional normalization mapping function for numerical preprocessing to eliminate their original dimensional attributes. After unifying and reducing their dimensionality to a dimensionless relative coefficient reference value between 0 and 1, used to characterize the micro-level congestion of the underlying system, the average response time of the task's out-degree at the memory read / write blocking critical point is used as the benchmark test quantity. The benchmark test quantity and the link integrity are then calculated. The system calculates the ratio of the total execution delay, extracts the ratio as the basis for the first benchmark coefficient, and selects the difference between the natural number constant 1 and the ratio as the second benchmark coefficient. This establishes a quantitative mapping rule for the weighting of system resource allocation based on the topology unlocking breadth and dependency depth. The system multiplies the first benchmark coefficient by the number of out-degrees to obtain the first product, multiplies the second benchmark coefficient by the depth of subsequent paths to obtain the second product, and adds the first and second products to output the topology gain factor. This multiplication-addition algebraic operation step is configured as the mapping operator of the policy network and integrated into the computational model within its computational architecture. By fixing the aforementioned first and second benchmark coefficients... The bias calculation weights of the corresponding neurons in the model ensure that the operation of these topological parameters is entirely within the forward inference process of the network architecture, guaranteeing a closed loop at the computational level. The system converts the topological gain factor into a topological value compensation amount and adds it to the feedback physical instant reward to obtain the corrected action value evaluation parameters. The above process, through a defined normalization algorithm and weight calculation path, converts physical queuing latency and resource consumption indicators into data that the policy network can parse. Based on the topological space dimension attribute of the directed acyclic graph, a quantification rule is constructed to establish a defined numerical correspondence between the action value evaluation parameters and the topological dependency depth of the controlled distributed computing system.

[0039] Example 5: When the system faces the initial deployment of a distributed computing cluster or business architecture restructuring, the system reads batch processing records within a preset time window from the historical operation and maintenance log database. It parses the process call stack sequence and data input / output timestamps contained in the batch processing records, counts task node pairs with sequential call relationships, and filters task node pairs whose frequency exceeds a frequency threshold to construct control flow and data flow triggering logic. Based on the control flow and data flow triggering logic, it constructs an initial oriented acyclic graph and converts it into an initial two-dimensional topological adjacency matrix. It then processes and converts discrete historical operation and maintenance parameters into a topological state benchmark for policy network evaluation. In the controlled distributed computing system... Before the system connects to real-time concurrent business flows, it retrieves the pre-calibration procedure for the topology gain factor and injects a standard probe task flow containing the gradient out-degree number and subsequent path depth into the virtual computing node. It iterates the numerical combination of the first and second benchmark coefficients in the real number range of 0 to 1 according to a preset step size. It monitors the queuing waiting time of critical path nodes in the standard probe task flow and the processor bus bandwidth utilization of the computing cluster under each numerical combination. It extracts specific numerical combinations that minimize the queuing waiting time and keep the processor bus bandwidth utilization in a safe working range, and solidifies them as the operating parameters of the first and second benchmark coefficients in the actual orchestration decision.

[0040] Example 6: In a multi-tenant resource isolation environment of a distributed computing cluster, the system acquires operation and maintenance task flow samples and calculates the mutual information entropy of the state vector feature dimension. In an offline environment, the scaling factor in the policy network self-attention mechanism is then applied. The initial value is set as the square root of the input vector dimension. Simultaneously, impulse noise of 1% to 5% is superimposed on the state vector. The offset of the weight allocation coefficient output by the normalized exponential function is monitored, and the minimum scaling factor value that makes the offset of the weight allocation coefficient less than the preset variance benchmark is extracted as the deployment parameter. For the determination of the values ​​of the first benchmark coefficient and the second benchmark coefficient, the system constructs a coordinate grid with an equal interval of 0.05 in the closed interval of 0 to 1. For each grid coordinate point, the weight parameters of the strategy network are iteratively calculated, and the corresponding Bellman residual descent slope is statistically analyzed. The coordinate point that makes the Bellman residual descent slope reach the maximum value and the mean square error is less than 0.02 is identified. Before the controlled distributed computing system is connected to the real-time production environment, the system injects a directional acyclic graph control flow containing different dependency depths. When it is detected that the scheduling delay of the task node is positively linearly correlated with the topology unlocking speed and the processor bus bandwidth fluctuation rate is in the fluctuation range of 10%, the value corresponding to the coordinate point is locked as the online running parameter.

[0041] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatically orchestrating internet operation and maintenance tasks based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Obtain the attribute feature set of the task nodes to be processed in the controlled distributed computing system, and construct a directional acyclic graph representing the node dependency structure based on the control flow and data flow dependency relationship between each task node to be processed, and convert it into a two-dimensional topological adjacency matrix. Step S2: Real-time acquisition of resource state variables and response latency of the controlled distributed computing system, and encoding them together with the priority and estimated time requirements of each task node to be processed into a multi-dimensional state vector, which is then input into the policy network. Step S3: Utilize the policy network to output the scheduling probability distribution of each task node to be processed within the legal candidate set, and perform action sampling on the scheduling probability distribution based on the Markov decision process to determine the scheduling strategy for the current decision cycle. Step S4: Obtain the physical immediate reward feedback, and parse the two-dimensional topological adjacency matrix to determine the set of subtask nodes that have been decoupled from their previous dependencies due to the scheduling policy. Calculate the value estimate of the subtask node set using the policy network, and use it as a compensation for the credit allocation lag problem. This value estimate, along with the physical immediate reward, participates in the update of the policy network's weight parameters.

2. The method for automatic orchestration of Internet operation and maintenance tasks based on reinforcement learning according to claim 1, characterized in that, Step S2 specifically includes: performing one-hot encoding on the task nodes to be processed for different business types, and concatenating them with the normalized task waiting time and historical resource consumption rate as tensors; using the self-attention mechanism to calculate the weight allocation coefficients of each dimension feature in the multidimensional state vector, and by increasing the feature expression weights of the critical path nodes, constructing the mapping relationship between the multidimensional state vector and the system load fluctuation and long-link business dependency features.

3. The method for automatic orchestration of Internet operation and maintenance tasks based on reinforcement learning according to claim 1, characterized in that, Step S3 specifically includes: Step S31, using the real-time constructed node dependency graph to perform legality filtering on the scheduling probability distribution to remove illegal nodes that violate the logical constraints of the directed acyclic graph; Step S32, combining resource stability, timeliness, and dependency conflict penalty factor to construct a scalar weighted feedback function, and under the environment of satisfying the rigid constraints of control flow, determining the scheduling priority according to the out-degree weight of each task node in the directed acyclic graph.

4. The method for automatic orchestration of Internet operation and maintenance tasks based on reinforcement learning according to claim 1, characterized in that, Step S4 involves calculating the value estimate of the subtask node set using the policy network, which includes: parsing the two-dimensional topological adjacency matrix, extracting the number of out-degrees and subsequent path depths of each task node in the oriented acyclic graph; calculating the topological gain factor based on the number of out-degrees and subsequent path depths, and using the topological gain factor to compensate for the value of the physical instant reward, thereby obtaining the corrected action value evaluation parameters.

5. The method for automatic orchestration of Internet operation and maintenance tasks based on reinforcement learning according to claim 1, characterized in that, Step S4, which updates the weight parameters, includes: constructing a value loss function based on topology look-ahead capability, which measures the deviation between the predicted value under the current policy and the target value incorporating topology gain factors; and iteratively updating the parameters of the policy network using the gradient descent algorithm, so that the scheduling policy matches the real-time load changes of the controlled distributed computing system within a preset convergence threshold range.

6. The method for automatic orchestration of Internet operation and maintenance tasks based on reinforcement learning according to claim 1, characterized in that, The set of attribute features for each task node to be processed includes node type, estimated time, memory requirements, CPU utilization, and disk I / O intensity.

7. The method for automatic orchestration of Internet operation and maintenance tasks based on reinforcement learning according to claim 1, characterized in that, After generating the scheduling strategy, the process also includes: real-time monitoring of the state deviation of the scheduling strategy during execution; when the fluctuation amplitude of the multi-dimensional state vector exceeds the preset threshold, the strategy network is corrected online; the scheduling probability distribution is perturbed and superimposed based on the exploration noise of the current controlled distributed computing system, and the scheduling path that avoids the resource fragmentation area is searched again based on the superposition result.

8. The method for automatic orchestration of Internet operation and maintenance tasks based on reinforcement learning according to claim 1, characterized in that, A controlled distributed computing system includes a cloud computing resource layer, a data center logical cluster, and edge processing nodes.

9. The method for automatic orchestration of Internet operation and maintenance tasks based on reinforcement learning according to claim 1, characterized in that, The construction of the directed acyclic graph in step S1 includes: extracting the triggering logic and data exchange interface between each task node by traversing the metadata repository of the controlled computing environment, and using the depth-first search algorithm to identify and eliminate dependency loops to ensure the logical unidirectionality of the directed acyclic graph.

10. An automatic orchestration system for internet operation and maintenance tasks based on reinforcement learning, used to implement the automatic orchestration method for internet operation and maintenance tasks based on reinforcement learning as described in claim 1, characterized in that, include: The topology modeling module is used to obtain the attribute feature set of the task nodes to be processed in the controlled distributed computing system, and construct a directional acyclic graph representing the node dependency structure and the corresponding two-dimensional topological adjacency matrix based on the control flow and data flow dependencies between the task nodes to be processed. The environmental status observation module is used to collect resource status variables and response latency of the controlled distributed computing system in real time, and encode the priority, estimated time consumption requirements and collected data of each task node into a multi-dimensional state vector. The orchestration decision module, connected to the environmental state observation module, is used to input multi-dimensional state vectors into the policy network, so that the policy network can output the scheduling policy of each task node to be processed. The value network correction module is connected to the topology modeling module and the orchestration decision module, respectively. It is used to obtain the physical immediate reward of the feedback, parse the two-dimensional topological adjacency matrix to determine the set of sub-task nodes, and then use the policy network to calculate the compensation amount corresponding to the set of sub-task nodes. Based on the compensation amount and the physical immediate reward, the weight parameters of the policy network are corrected.

Images