A server exception automatic detection and recovery method and system

Abstract

The application relates to the technical field of server operation and maintenance, and discloses a server abnormality automatic detection and recovery method and system, the method comprising collecting real-time logs and monitoring data, extracting abnormal load characteristics to construct a system state vector; using a deep reinforcement learning model to generate a candidate recovery action, and simulating and predicting stability benefits in a digital twin environment; combining high-load resource estimation to generate an action priority sequence through Monte Carlo tree search; optimizing a recovery path according to topological dependence encapsulation, verifying overload risk based on real-time resource snapshots, and automatically deploying and executing after confirming feasibility. The method can solve the problem of low server automatic recovery success rate in the prior art.

Description

Technical Field

[0001] This invention relates to the field of server operation and maintenance technology, and in particular to a method and system for automatic detection and recovery of server anomalies. Background Technology

[0002] Currently, in the field of server operation and maintenance technology, server clusters bear massive concurrent requests and core business logic, and their operational stability directly determines the overall availability of the system. In current industrial scenarios, there is an urgent need to use fault prediction and health management technologies to perceive the cluster status in real time and quickly generate accurate recovery strategies when anomalies occur, in order to cope with increasingly complex system load changes and dynamic resource constraints.

[0003] In one existing technology, server anomaly recovery primarily relies on a static rule engine based on fixed thresholds or human experience. The system triggers a single recovery action only according to preset indicator boundaries, such as forcibly restarting the service process upon detecting a connection timeout. However, this approach ignores the strong dynamic dependency between the server's current state and the recovery action, failing to quantify and assess in real time the potential benefits and risks of a particular action to the overall stability of the cluster. For example, when cluster resources are already heavily utilized, mechanically executing a restart or traffic migration operation can easily trigger resource overload boundaries, causing fault propagation and escalating local problems into global paralysis. Furthermore, there is a lack of a mechanism to simulate and predict resource consumption after the action is executed. Because existing technologies lack the ability to dynamically predict the subsequent impact of recovery actions, it is difficult to balance the effectiveness of recovery with system security when facing complex anomaly scenarios.

[0004] Therefore, existing technologies suffer from a low success rate in automatic server recovery. Summary of the Invention

[0005] This invention provides a method and system for automatic detection and recovery of server anomalies, in order to solve the problem of low success rate of automatic server recovery in the prior art.

[0006] Firstly, in order to solve the above-mentioned technical problems, the present invention provides a method for automatic detection and recovery of server anomalies, comprising: Acquire real-time log streams and cluster monitoring data, perform abnormal fluctuation detection, locate abnormal server segments, extract feature indicators of the abnormal server segments, and perform feature dimension mapping to obtain the system state vector; The system state vector is input into a pre-trained action generation network to generate an initial recovery action sequence, and a value evaluation network is used to score and filter the results to obtain candidate recovery actions. The candidate recovery action is simulated in a pre-set digital twin environment to generate a predicted cluster load distribution matrix. If the predicted cluster load distribution matrix indicates that there are no overloaded nodes, the fault propagation link is identified according to the system state vector, and the potential risk value is calculated. The stability is determined by combining the predicted cluster load distribution matrix to obtain a stability benefit score. By integrating the stability gain score with the high load resource consumption estimate corresponding to the predicted cluster load distribution matrix, a pre-set Monte Carlo search tree is initialized, and multi-step simulation is performed to generate a sorted action priority sequence. The corresponding execution parameters, concurrent execution constraints, and consistency verification logic are determined based on the first action in the action priority sequence, and an optimized recovery path is encapsulated. The estimated resource consumption vector of the optimized recovery path is quantified, and a resource overload risk verification is performed in combination with the real-time snapshot of cluster resources collected in real time, generating an execution feasibility confirmation signal. In response to the execution feasibility confirmation signal, the optimized recovery path is mapped to atomic operation code, and dynamic configuration parameters are injected to drive automated execution until the real-time calculated fit is within the preset convergence range, at which point the recovery is determined to be complete.

[0007] Secondly, the present invention provides an automatic server anomaly detection and recovery system, comprising: The state awareness module is used to acquire real-time log streams and cluster monitoring data, perform abnormal fluctuation detection, lock abnormal server segments, extract feature indicators of the abnormal server segments, and perform feature dimension mapping to obtain the system state vector. The strategy generation module is used to input the system state vector into a pre-trained action generation network to generate an initial recovery action sequence, and to use a value evaluation network to score and filter candidates for recovery actions. The simulation and deduction module is used to simulate the execution of the candidate recovery action in a preset digital twin environment, generate a predicted cluster load distribution matrix, and if the predicted cluster load distribution matrix indicates that there are no overloaded nodes, then the fault propagation link is identified according to the system state vector, and the potential risk value is calculated. The stability is determined by combining the predicted cluster load distribution matrix to obtain a stability benefit score. The global decision module is used to integrate the stability benefit score with the high load resource consumption estimate corresponding to the predicted cluster load distribution matrix, initialize the preset Monte Carlo search tree, and perform multi-step simulation to generate a sorted action priority sequence. The path optimization module is used to determine the corresponding execution parameters, concurrent execution constraints and consistency verification logic based on the first action in the action priority sequence, and encapsulate them to obtain an optimized recovery path; The risk verification module is used to quantify the estimated resource consumption vector of the optimized recovery path, and perform resource overload risk verification in combination with the real-time snapshot of cluster resources collected in real time, and generate an execution feasibility confirmation signal. The execution control module is used to respond to the execution feasibility confirmation signal, map the optimized recovery path into atomic operation code, and inject dynamic configuration parameters to drive automated execution until the real-time calculated fitting degree is in the preset convergence range, and then determine that the recovery is complete.

[0008] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention performs multi-source fusion and feature extraction operations on real-time log streams and cluster monitoring data. All original operation and maintenance data are processed by time-series alignment, abnormal fluctuation detection and key indicator mapping to generate a system state vector that represents the real-time operation of the server. Through this standardization process, the system can more comprehensively and accurately depict the load characteristics and anomaly types in complex environments, providing a standard and high-quality data foundation for subsequent intelligent decision-making.

[0009] (2) Based on the system state vector, the present invention uses a deep reinforcement learning model to generate candidate recovery actions and performs a pre-play in a digital twin environment. It can output a stability gain score that includes the expected load distribution. Through this mechanism, the system breaks through the limitation that traditional static rules cannot cope with dynamic dependencies. It can predict the real impact of the action on the cluster state before the action is executed, thereby significantly reducing the probability of fault propagation or misjudgment due to improper recovery strategy.

[0010] (3) The present invention applies the Monte Carlo tree search algorithm to perform a global stability assessment operation on the recovery action, and comprehensively considers the stability benefit score and the high load resource occupation estimate to deduce the potential impact of the action on the cluster as a whole. This process makes the recovery decision no longer limited to the short-term repair from a local perspective, but seeks the optimal solution of the global long-term steady state, effectively avoiding the risk of local recovery causing a global chain reaction and improving the overall robustness of the cluster.

[0011] (4) The present invention performs resource overload risk verification operation based on the optimized recovery path, and uses real-time resource snapshots and estimated resource consumption vectors to superimpose and compare them to generate a feasibility confirmation signal. This strategy builds a strict safety boundary before automated execution to prevent new resource bottlenecks from being triggered due to blindly executing recovery actions in scenarios with high resource occupancy, thus ensuring the safety and reliability of the recovery process. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a server anomaly automatic detection and recovery method provided in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an automatic server anomaly detection and recovery system provided in the second embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Reference Figure 1 The first embodiment of the present invention provides a method for automatic detection and recovery of server anomalies, including the following steps: S11, acquire real-time log stream and cluster monitoring data, perform abnormal fluctuation detection, lock the abnormal server segment, extract the feature indicators of the abnormal server segment, and perform feature dimension mapping to obtain the system state vector. S12, the system state vector is input into the pre-trained action generation network to generate an initial recovery action sequence, and the value evaluation network is used to score and filter to obtain candidate recovery actions; S13, simulate the execution of the candidate recovery action in a preset digital twin environment to generate a predicted cluster load distribution matrix. If the predicted cluster load distribution matrix indicates that there are no overloaded nodes, identify the fault propagation link according to the system state vector and calculate the potential risk value. Combine the predicted cluster load distribution matrix to make a stability judgment and obtain a stability benefit score. S14, integrate the stability gain score with the high load resource consumption estimate corresponding to the predicted cluster load distribution matrix, initialize the preset Monte Carlo search tree, and perform multi-step simulation to generate a sorted action priority sequence; S15, determine the corresponding execution parameters, concurrent execution constraints and consistency verification logic based on the first action in the action priority sequence, and encapsulate them to obtain an optimized recovery path; S16, quantify the estimated resource consumption vector of the optimized recovery path, and perform resource overload risk verification in combination with the real-time snapshot of cluster resources collected in real time, and generate an execution feasibility confirmation signal; S17, in response to the execution feasibility confirmation signal, the optimized recovery path is mapped to atomic operation code, and dynamic configuration parameters are injected to drive automated execution until the real-time calculated fitting degree is in the preset convergence range, and the recovery is determined to be complete.

[0015] In step S11, real-time log streams and cluster monitoring data are acquired, and abnormal fluctuation detection is performed to identify abnormal server segments. Feature indicators of these abnormal server segments are extracted, and feature dimension mapping is performed to obtain a system state vector, including: Obtain the real-time log stream and the cluster monitoring data; For the real-time log stream and the cluster monitoring data, a time-series aligned dataset is generated based on the timestamp index, and fluctuation detection is performed on the time-series aligned dataset to lock the abnormal server segments. Extract the error stack information from the server anomaly fragment to determine the anomaly type indicator; Construct a resource utilization waveform for the abnormal time window corresponding to the abnormal server segment, and calculate the current load characteristics; The system state vector is generated by mapping the anomaly type index and the current load characteristics into a pre-constructed state vector space and then mapping the feature dimensions.

[0016] It's worth noting that acquiring real-time log streams and cluster monitoring data is fundamental to data collection and integration. For real-time log streams, lightweight log collection agents (such as Filebeat or Fluentd) are deployed on each node of the target server cluster. These agents monitor the standard output and error log files of application services, middleware, and the operating system in real time, aggregating the generated unstructured text data into a centralized message queue (such as Kafka) via streaming. Real-time log streams include timestamps, log levels, source IP addresses, and the actual text content. For cluster monitoring data, a monitoring system (such as Prometheus) is used to pull key performance indicators from the exporters of each node at a preset sampling frequency. These indicators include CPU utilization, memory usage, disk I / O throughput, and network bandwidth utilization. The preset sampling frequency is typically set to once every 5 seconds. This frequency is determined based on Shannon's sampling theorem and industrial operation and maintenance practices, ensuring that instantaneous load spikes are captured while avoiding excessive system resource consumption due to overly frequent collection.

[0017] Next, a time-series aligned dataset is generated based on timestamp indexes for the real-time log stream and cluster monitoring data. Since log data is event-driven discrete text, while monitoring data is periodic continuous numerical data, there is a natural misalignment between the two on the timeline. Specifically, a preset time tolerance window (e.g., 1 second before and after) is set based on the sampling time point of the cluster monitoring data. For each monitoring sampling point, all log entries falling within this time tolerance window are retrieved from the real-time log stream, aggregated into a log feature vector (e.g., by counting the number of entries at each log level), and concatenated with the monitoring metric value at that moment to form a time-series data sample containing multi-dimensional attributes. The preset time tolerance window is determined considering network transmission latency and system clock synchronization errors; typically, twice the average round-trip time (RTT) is chosen as the window size to ensure the timeliness consistency of the associated data.

[0018] Then, fluctuation detection is performed on the time-aligned dataset to identify abnormal server segments. Specifically, a sliding window-based statistical process control (SPC) method is used, with a preset sliding window length, such as 30 sample points corresponding to a 150-second duration. Within the current window, the moving average and standard deviation of key monitoring metrics (such as CPU utilization) are calculated. Simultaneously, the cumulative frequency of error / critical logs within the window is counted. If a monitoring metric exceeds the moving average plus three times the standard deviation (3-Sigma criterion) at a given moment, or if the error log frequency exceeds a preset frequency threshold, an abnormal fluctuation is identified at that moment. The preset frequency threshold is determined based on statistical data from historical normal operating cycles. The 99th percentile of the number of error logs per unit window within a historical normal period is calculated and set as the threshold (e.g., 5 logs per window) to mask occasional non-critical errors. Once an anomaly is detected, data segments extending from the current window and before and after it by a preset length (e.g., 5 minutes) are extracted, marked as abnormal server segments, and this period is defined as the abnormal time window.

[0019] Extract error stack traces from server exception fragments to determine exception type indicators. Specifically, for each log entry containing an error stack trace in a server exception fragment, use regular expressions to extract the exception class name (ExceptionClass) and key error message (Message). Using a pre-trained text classification model (such as TF-IDF combined with a Naive Bayes classifier) ​​or a rule base based on keyword matching, map the extracted text information to predefined exception category codes. For example, stack traces containing the keywords "ConnectionRefused" or "Timeout" are classified as "Network Connection Exception" and encoded as 01; those containing "OutOfMemory" or "GC overhead limit exceeded" are classified as "Memory Overflow Exception" and encoded as 02. Statistically analyze the frequency of each category code within the exception fragment to construct a normalized frequency vector, which is the exception type indicator.

[0020] Resource utilization waveforms are constructed for the abnormal time windows corresponding to server anomalies, and current load characteristics are calculated. Time-series numerical sequences of resource indicators such as CPU, memory, and I / O are extracted within the abnormal time window to form a multi-dimensional resource utilization waveform. For each dimension of the waveform, its statistical characteristics are calculated, including mean, peak value, variance, and trend slope. For example, the first-order linear regression slope of the CPU utilization sequence is calculated. If the slope is positive and the absolute value is greater than a preset trend threshold (e.g., 0.5% / second), it indicates that the load is in a rapidly increasing state. The preset trend threshold is determined by analyzing the load curve characteristics before historical crashes, selecting the minimum growth rate that leads to system crashes as the critical value. The statistical characteristics of each resource are concatenated into a high-dimensional numerical vector as the current load characteristic, which can quantitatively describe the system pressure state (e.g., compute-intensive overload or I / O throughput bottleneck) when the anomaly occurs.

[0021] Finally, the anomaly type index and the current load features are input into a pre-constructed state vector space, and the system state vector is generated through feature dimension mapping. The state vector space is a predefined standardized numerical space designed to eliminate dimensional differences between different features. A min-max normalization operation is performed to map the values ​​in the current load features to the [0,1] interval. Simultaneously, one-hot encoding or embedding techniques are used to process the anomaly type index. The processed load feature vector and the anomaly type vector are concatenated to obtain a comprehensive feature vector. To reduce dimensionality redundancy, Principal Component Analysis (PCA) can be used to reduce the dimensionality of the concatenated vector, retaining principal components with a cumulative contribution rate exceeding 95%. The resulting low-dimensional dense vector is the system state vector. The 95% cumulative contribution rate is an empirical value that significantly reduces the computational complexity of subsequent reinforcement learning models while minimizing data loss.

[0022] In step S12, the system state vector is input into a pre-trained action generation network to generate an initial recovery action sequence, and a value evaluation network is used for scoring and filtering to obtain candidate recovery actions, including: The system state vector is input into a pre-trained action generation network, which outputs an action probability distribution matrix. Atomic action units are extracted based on the action probability distribution matrix and combined to generate the initial recovery action sequence; The initial recovery action sequence is scored using a pre-trained value evaluation network combined with the system state vector to obtain a sequence value score; Sequences whose sequence value scores exceed a preset score threshold are selected as candidate recovery actions.

[0023] First, the system state vector is input into a pre-trained action generation network, which outputs an action probability distribution matrix. Specifically, the action generation network is designed as a sequence generation model based on the Transformer architecture, aiming to handle long-distance temporal dependencies between recovery actions. This network mainly consists of an encoder and a decoder. The encoder receives the system state vector output from step S11, first mapping it to a high-dimensional hidden layer vector through a fully connected layer, and adding positional encoding to preserve the temporal features of the state. Subsequently, the data flows through a multi-layered stacked self-attention module, each containing multiple attention heads and a feedforward neural network, used to extract high-order correlations within the state features. The decoder receives the action embedding vector generated at each time step, processes it through a masked multi-head attention layer, and finally outputs a probability distribution vector with a dimension equal to the predefined atomic action space size through a linear projection layer and a Softmax activation function. The training data comes from fault repair logs in the historical operations and maintenance knowledge base. A large number of successful fault recovery records are cleaned from the historical database, and the fault time monitoring data in each record is converted into a system state vector, and the corresponding operations and maintenance operation step sequence is converted into a standardized action number sequence. The training method adopts the behavior cloning strategy in supervised learning, taking the system state vector as input and the standardized action number sequence as target label. The cross-entropy loss function is used to calculate the difference between the probability distribution predicted by the model and the real action sequence, and the AdamW optimizer is used to iteratively update the parameters until the accuracy on the validation set no longer improves.

[0024] Next, atomic action units are extracted based on the action probability distribution matrix and combined to generate an initial recovery action sequence. To avoid local optima caused by a single greedy strategy, one implementation uses a beam search algorithm for sequence decoding. A preset beam width, such as 5, is set, and at each time step, the five atomic actions with the highest probabilities are selected as candidate branches, and the five complete paths with the highest cumulative probabilities are retained as the initial recovery action sequence. For example, for a cascading failure caused by a database connection timeout, the generated sequence may include combinations with different temporal logic, such as "rate limiting front-end requests → expanding the database connection pool → restarting the transaction module" and "restarting the transaction module → clearing temporary files → rate limiting front-end requests".

[0025] Then, a pre-trained value evaluation network is used in conjunction with the system state vector to score the initial recovery action sequence, resulting in a sequence value score. Specifically, the value evaluation network is designed as a deep multilayer perceptron to quantify the effectiveness of the recovery strategy. The input layer of this network consists of two concatenated parts: the current system state vector and the embedded representation of the initial recovery action sequence to be evaluated. The concatenated feature vector is then passed through three fully connected hidden layers, each followed by a ReLU activation function and a Dropout layer to prevent overfitting. The output layer contains a Sigmoid activation function, outputting a scalar between 0 and 1, which is the sequence value score. To train this network, a triplet containing state, action, and reward needs to be constructed. The reward value is calculated based on the recovery performance of historical data, and a comprehensive performance function is defined, which is obtained by weighted summation of normalized fault recovery time and normalized resource release amount. The calculated comprehensive performance function value is normalized and mapped to the 0-1 interval, serving as the target label for training. The training method uses state and action as input, target label as supervision signal, and mean squared error as the loss function. The network is trained using the stochastic gradient descent algorithm, enabling it to fit a nonlinear mapping relationship from the fault state and recovery plan to the expected recovery effect.

[0026] Finally, sequences with value scores exceeding a preset threshold are selected as candidate recovery actions. This step aims to filter out inefficient or high-risk randomly generated sequences. The preset threshold is determined based on statistical analysis of the score distribution in the historical validation set. Specifically, the value scores of all successful recovery cases in the historical validation set are statistically analyzed, and the 25th quartile (i.e., the lower quartile, for example, 0.75) of its score distribution is calculated. This value is then set as the preset threshold. This means that only recovery sequences with expected results superior to 75% of historical successful cases are retained, thus ensuring that candidate actions entering subsequent simulation phases have high credibility and effectiveness.

[0027] In step S13, the candidate recovery action is simulated in a preset digital twin environment to generate a predicted cluster load distribution matrix. If the predicted cluster load distribution matrix indicates that there are no overloaded nodes, the fault propagation link is identified based on the system state vector, and the potential risk value is calculated. A stability assessment is then performed based on the predicted cluster load distribution matrix to obtain a stability gain score, including: The candidate recovery actions are rehearsed in a pre-set digital twin environment, node resource usage change information is recorded, and the virtual cluster state after the rehearsal is generated; Extract the resource usage indicators of each node in the virtual cluster state, and construct the predicted cluster load distribution matrix according to the dimensional mapping relationship between node identifier and resource type; If the predicted cluster load distribution matrix indicates that there are no overloaded nodes, then the fault propagation link is identified based on the system state vector; Calculate the potential risk value of the candidate recovery action for the fault propagation link; Based on the predicted cluster load distribution matrix, the resource balance is calculated, and a weighted calculation is performed on the resource balance and the potential risk value to obtain the stability benefit score.

[0028] First, candidate recovery actions are rehearsed in a pre-defined digital twin environment, recording changes in node resource usage and generating the virtual cluster state after the rehearsal. Specifically, this digital twin environment is a discrete event simulation system built upon the cluster's physical topology (server hardware configuration, network bandwidth limitations) and logical topology (microservice call chains, database dependencies). To achieve high-fidelity action simulation, the environment incorporates various "action-resource" impact models. For example, for the "rate limiting" action, the model defines a linear regression coefficient for the decrease in CPU utilization as throughput decreases by a certain value; for the "restart" action, the model defines a time-resource curve for the process startup phase, obtained by fitting historical startup log data, accurately describing the instantaneous peak characteristics of CPU and memory from process initialization to service readiness. Real load data at the current moment is injected into the simulation engine as the initial state, simulating the time evolution process after the action is executed, recording the numerical changes in CPU, memory, and I / O throughput of each node within each time step (e.g., 1 second). Finally, a resource snapshot at the moment the evolution tends towards a steady state is selected as the virtual cluster state after the rehearsal.

[0029] In one implementation, resource usage metrics for each node in the virtual cluster state are extracted, and a predictive cluster load distribution matrix is ​​constructed based on the dimensional mapping relationship between node identifiers and resource types. All node objects in the virtual cluster state are traversed, and normalized values ​​for three key dimensions—CPU utilization, memory usage, and network bandwidth utilization—are extracted. A two-dimensional matrix is ​​constructed, with the number of rows representing the total number of cluster nodes and the number of columns representing the number of resource types. Elements in the matrix Characterizing the first The node at the th This matrix represents the predicted resource occupancy levels. To ensure the validity of the matrix data, if certain nodes do not have specific services deployed, their corresponding resource values ​​are filled with a baseline noise level (e.g., 0.01). This matrix visually reflects the reallocation of resources within the cluster after a recovery operation.

[0030] Next, if the predicted cluster load distribution matrix indicates that there are no overloaded nodes, the fault propagation path is identified based on the system state vector. First, the resource overload determination logic is set, and the matrix is ​​traversed. All elements in, if any element Exceeding the preset security threshold If an overloaded node is detected, the evaluation process for the current action is immediately terminated, and its stability gain score is set to 0. This preset safety threshold... Based on the queuing theory model, the threshold value is typically set to 0.85. According to the M / M / 1 queuing model, when the server utilization exceeds this value, the request response time will increase exponentially, indicating that the system has entered an unstable region. If there is no overload, the system state vector output from step S11 is analyzed to extract the anomaly type indicators and load characteristics. Using a rule-based topology search algorithm, starting from the node where the anomaly occurred in the cluster dependency graph, the fault propagation path is defined as the directed acyclic graph formed by these nodes and their connecting edges, tracing backward along the call relationships to the source node, and forward tracing the affected downstream nodes.

[0031] It is worth noting that the calculation of the potential risk value of candidate recovery actions for the fault propagation path is also important. This risk value aims to quantify the likelihood that an action is merely a temporary fix or will trigger a cascading failure. Its calculation formula encompasses two dimensions: link blocking capability and side effect intensity. For link blocking capability, a pre-built operations and maintenance knowledge graph is queried to determine whether the current action type matches the anomaly source type, thus obtaining a normalized blocking coefficient. (For example, a perfect match is 1, and mitigation is 0.5). To assess the intensity of side effects, evaluate the impact of the action on non-faulty nodes on the link, calculate the sum of the weights of the affected nodes, and obtain the normalized side effect coefficient. Potential risk value The calculation formula is as follows: In the formula, and To adjust the weights, they are typically set based on the business's sensitivity to stability, for example, 0.7 and 0.3 respectively. This formula indicates that the stronger the blocking capability, the lower the risk; the greater the side effects, the higher the risk.

[0032] Finally, resource balance is calculated based on the predicted cluster load distribution matrix. A weighted calculation is then performed on the resource balance and potential risk values ​​to obtain a stability benefit score. The measure is the inverse of the combined variance of all column vectors in the matrix. Specifically, the matrix is ​​calculated... The variance of each column (i.e., each resource) is used to calculate the average variance of all resource dimensions, thus obtaining the overall variance. Resource balance Defined as the negative exponential function value of the overall variance, i.e.: The closer this value is to 1, the more evenly the cluster load is distributed. Stability Gain Score The calculation uses a linear weighted formula: In the formula, The balance coefficient, determined using the analytic hierarchy process (AHP), is typically set to 0.6. This formula reflects the principle of prioritizing the optimal distribution of resource utilization while ensuring low risk. The final result... As a quantitative indicator, it is used for node value evaluation in subsequent Monte Carlo tree search.

[0033] In step S14, the stability gain score is fused with the estimated high-load resource consumption value corresponding to the predicted cluster load distribution matrix to initialize a preset Monte Carlo search tree, and a multi-step simulation is performed to generate a sorted sequence of action priorities, including: Extract the peak resource usage data from the predicted cluster load distribution matrix to determine the estimated value of high load resource usage; The stability gain score and the estimated high load resource consumption are weighted and the result is assigned to the initial state value of the root node of the preset Monte Carlo search tree. Based on the initial state value of the root node, an action search path is generated, and a multi-step simulation is performed on the action search path to obtain the overall stability impact value of the cluster. The overall stable impact value is backpropagated to the nodes along the Monte Carlo search tree, and the cumulative reward value and access count of the nodes along the tree are updated using the incremental averaging algorithm. The actions are sorted according to the accumulated reward value and the access count to generate the action priority sequence.

[0034] First, extract peak resource usage data from the predicted cluster load distribution matrix to determine the estimated high-load resource usage. Specifically, iterate through the predicted cluster load distribution matrix constructed in step S13, extracting the maximum value from each column of resource vectors (such as CPU, memory, and network bandwidth) to form a peak feature set. The estimated high-load resource usage is obtained by calculating the arithmetic mean of this peak feature set. The numerical physical meaning of this value lies in characterizing the local extreme stress state that the cluster experiences after performing a certain action. Next, a weighted calculation is performed on the stability gain score and the estimated high-load resource consumption value, and the result is assigned as the initial state value of the root node of a pre-defined Monte Carlo search tree. Setting the root node state value... The calculation formula is: In the formula, The stability gain score is the output of step S13. This is a preset empirical balance coefficient. Balance coefficient The setting is derived from a grid search of historical fault simulation data, typically ranging from 0.5 to 0.7, to ensure that while pursuing global resource balance, the decision-making process severely punishes single-point resource overload. The calculated... It is directly used as the initial value assessment benchmark for the root node of the Monte Carlo search tree.

[0035] Next, an action search path is generated based on the initial state value of the root node. Multi-step simulations are then performed along this path to obtain the overall stability impact value of the cluster. The action search path expansion uses an upper confidence interval algorithm; for each child action node of the current node, its upper confidence interval is calculated. : In the formula, This represents the average current cumulative reward of the child nodes. This is the visit count for the parent node. This is the visit count for this child node. To explore constants. Usually set to (Approximately 1.414), used to balance the exploitation of known high-yield actions with the exploration of unknown actions. Along... The largest node continues to expand downwards until it encounters a leaf node that is not fully expanded. Then, multi-step simulations are initiated from this leaf node, employing lightweight stochastic strategies or heuristic rules to continuously simulate and execute the simulations to a preset depth. The random recovery action of the step. Depth of inference. Based on the average number of steps required for historical fault diagnosis, typically set to 5 steps. After the simulation, the health status of the final virtual cluster is evaluated, and a value between 0 and 1 is output as the overall stability impact value. .

[0036] Then, the overall stable impact value is backpropagated to the nodes along the Monte Carlo search tree, and the cumulative reward value and visit count of the nodes are updated using the incremental averaging algorithm. After a single simulation is completed, the search path is backtracked layer by layer from the leaf nodes to the root node. For each node along the path, its visit count is first calculated. Updated to To avoid storing all historical earnings values, an incremental averaging algorithm is used to dynamically update the cumulative reward value of each node. Its update formula is: This formula can smooth the node value in real time with low computational complexity during each backpropagation, so that it gradually converges to the true expected return of the action in the long-term evolution.

[0037] Finally, the actions are sorted according to the cumulative reward value and access count to generate an action priority sequence. The search process stops after the Monte Carlo search tree completes a preset number of iterations (e.g., 2000) or reaches a preset computation time threshold (e.g., 500 milliseconds). All first-level child nodes under the root node (i.e., the direct recovery actions selectable in the current state) are traversed, and the cumulative reward value of each child node is extracted. With access count Calculate ranking metrics, prioritizing by access count. Sort in descending order; if visit counts are the same, sort by cumulative reward value. Sort in descending order. A higher access count indicates that the action is considered the most promising path direction by the Monte Carlo search tree after sufficient exploration and utilization trade-offs. The sorted set of actions is output, generating an action priority sequence, which is then used by the subsequent execution scheduling module to extract and deploy the first action.

[0038] In step S15, the corresponding execution parameters, concurrent execution constraints, and consistency verification logic are determined based on the first action in the action priority sequence, and an optimized recovery path is encapsulated, including: Extract the node connection relationships involved in the first action in the action priority sequence, determine the node topology, and analyze the data flow and call frequency of the nodes involved in the first action to construct a dependency matrix; The expansion ratio or rate limiting threshold is set according to the node topology and used as the execution parameter; Based on the dependency matrix, single-node operation or mutual exclusion operation logic is set as the concurrent execution constraint; The consistency verification logic is generated by matching the preset verification rules with the type of the first action. The optimized recovery path is generated by integrating the execution parameters, the concurrent execution constraints, and the consistency verification logic.

[0039] First, the node connections involved in the first action in the action priority sequence are extracted to determine the node topology. The data flow and call frequency of the nodes involved in the first action are then analyzed to construct a dependency matrix. Specifically, by calling the application programming interface (API) of a distributed tracing system (such as SkyWalking or Zipkin), real-time call chain data between microservices within the current time window is obtained. Based on this call chain data, the target node directly related to the first action and its adjacent nodes within one hop of its upstream and downstream nodes are extracted. These nodes are used as vertices of the graph, and the network connections between nodes are used as edges, constructing a directed graph as the node topology. Next, the network traffic characteristics on the graph are analyzed, extracting the number of requests per unit time and the direction of data packet transmission between adjacent nodes. The node data flow direction is defined as the sign of the matrix elements, with positive values ​​for forward calls and negative values ​​for reverse callbacks. The normalized value of the call frequency is defined as the absolute value of the matrix elements. This constructs a matrix with dimension 1. Dependency matrix ,in This represents the total number of nodes in the node topology. This matrix precisely quantifies the tightness of coupling between microservices within the first and second action execution boundaries.

[0040] In one implementation, the scaling ratio or rate limiting threshold is set based on the node topology as an execution parameter. Each target node in the node topology is traversed, and its hardware configuration baseline and historical load test capacity limit are read. If the first action type is node scaling, the current node's real-time CPU load rate is used as the basis for the calculation. With target safe load rate Calculate the expansion ratio Among them, the target safe load rate This is a preset empirical constant, typically set to 0.70. Its physical meaning is to maintain the node at 70% CPU utilization, reserving a 30% resource buffer to handle sudden traffic spikes during system recovery. If the first action type is service rate limiting, the rate limiting threshold is calculated using the token bucket algorithm model, combined with the upstream call frequency in the dependency matrix. Rate limiting threshold The rate limit is set as the product of the downstream node's maximum processing capacity and a preset rate limit tolerance. The preset rate limit tolerance is derived by back-calculating historical Service Level Agreement (SLA) metrics and typically ranges from 0.8 to 0.9. The calculated rate limit... or It is then encapsulated into standardized numerical variables and used as execution parameters.

[0041] It is worth noting that the preset rate limiting tolerance is determined by collecting data on the maximum processing capacity of downstream nodes under different load levels during historical normal operation periods, as well as the maximum allowed request loss rate or response timeout rate in the corresponding Service Level Agreement (SLA). The relationship curve between the downstream node's processing capacity and the request loss rate is fitted by linear regression. The processing capacity utilization rate corresponding to the inflection point on the curve where the loss rate is about to exceed the SLA's allowed threshold is selected as the benchmark value of the rate limiting tolerance. After statistical analysis of multiple sets of historical data, this benchmark value is usually stable between 0.8 and 0.9. In this embodiment, the median of 0.85 is taken as the default value, but it can also be dynamically adjusted according to the business scenario.

[0042] It should be noted that the scaling up action in this step is triggered only when the current load rate of a node is detected to have exceeded the preset target safe load rate. Therefore, before performing the scaling up ratio calculation, it is first determined that... Is it greater than If true, the required expansion ratio is calculated according to the above calculation method. A positive result indicates the multiple of resources that need to be added. If false, it means that the node load is still within a safe range, and the expansion action itself will not be selected as the first action, so there will be no negative ratio.

[0043] Next, based on the dependency matrix, single-node operation or mutual exclusion operation logic is set as a concurrency constraint. To prevent distributed deadlock or data inconsistency caused by simultaneous changes in multiple strongly coupled nodes during recovery, the dependency matrix is... Perform connectivity determination. Extract node pairs corresponding to non-zero elements in the matrix. If the absolute value of the call frequency between two nodes exceeds a preset coupling threshold... This indicates a high-frequency, strong dependency between the two nodes. In this case, the recovery operations for these two nodes are set as mutually exclusive operations, meaning they must be executed sequentially, waiting for the health check of the preceding node to pass completely before the operation of the subsequent node can be triggered. Conversely, if the matrix element values ​​between the nodes are below the coupling threshold... Then, it is divided into independent groups, and the nodes within each group are configured with single-node operation logic that allows parallel processing. A preset coupling threshold is set. The settings are based on a statistical distribution model of historical failure cascading rates, and a fixed threshold value (e.g., 0.6) is selected as the inflection point value that leads to a sharp increase in the probability of cascading downtime. The resulting time-series scheduling rules serve as concurrent execution constraints.

[0044] Then, based on the type of the first action, pre-defined verification rules are matched to generate consistency verification logic. The system maintains a local rule base that precisely maps action types to verification strategies. This rule base is constructed by parsing assertion statements from historically successful operation and maintenance scripts. The type identifier of the first action is extracted and searched in the local rule base. If the action type is a database master-slave switch, the matched pre-defined verification rule is to check the synchronization offset of the master and slave nodes' binary logs (Binlog). Based on this, corresponding consistency verification logic is generated, requiring the synchronization offset to be less than a preset latency threshold (e.g., 100MB) for the action to be considered valid. If the action type is a stateless microservice restart, the matched verification rule is to poll the service health check communication port. The generated consistency verification logic requires the service to return a 200 status code three times consecutively within a specified time window to prevent false liveness.

[0045] Finally, execution parameters, concurrent execution constraints, and consistency verification logic are integrated to generate an optimized recovery path. A directed acyclic graph (DAG) data structure is used to serialize and encapsulate the generated strategies and parameters. The first action is decomposed into specific control instructions recognizable by underlying automation tools (such as Ansible or the Kubernetes API), and execution parameters are used as dynamic input payloads for these instructions. Simultaneously, concurrent execution constraints are used to establish sequential directed connections between control instructions, ensuring actions proceed in a safe order. After each key instruction node in the DAG, the corresponding generated consistency verification logic is attached as a post-monitoring node. The completed DAG is exported as a standardized configuration template file (such as YAML format), and this template file is the final output, generating the optimized recovery path. This path not only contains the execution values ​​of the actions but also establishes concurrent control boundaries and result verification criteria, providing a complete instruction set for the safe delivery of subsequent actions.

[0046] In step S16, the estimated resource consumption vector of the optimized recovery path is quantified, and a resource overload risk verification is performed in conjunction with the real-time snapshot of cluster resources obtained in real time, generating an execution feasibility confirmation signal, including: The resource consumption value of each action is quantified from the instruction set included in the optimized recovery path to construct the estimated resource consumption vector; Collect the resource usage status of each node in the current cluster and generate a real-time snapshot of the cluster resources; The estimated resource consumption vector is superimposed with the real-time snapshot of cluster resources to obtain the total resource usage. By comparing the total resource usage with the preset node carrying capacity limit, the number of resource overload risk points is counted. If the number of resource overload risk points is zero, the execution feasibility confirmation signal is output.

[0047] First, the resource consumption of each action in the instruction set included in the optimized recovery path is quantified to construct an estimated resource consumption vector. Specifically, the optimized recovery path file generated in step S15 is parsed to extract the discrete control instruction sequence. For each control instruction, a pre-set action resource consumption benchmark library is queried. This benchmark library is established by performing sandbox stress tests on various operation and maintenance instructions in an independent test environment, recording the average CPU computing power consumption, physical memory overhead, and peak disk I / O rate required to execute each instruction. After extracting the instantaneous peak resource data corresponding to each instruction, it is aggregated and summed according to the physical or virtual identifier of the execution node. For example, if the instruction set includes starting two new container instances simultaneously on a specific worker node, the peak startup memory of a single container is multiplied by the number of instances. The aggregated resource increments of each node are mapped to feature dimensions to generate an estimated resource consumption vector with a dimension equal to the total number of cluster nodes and containing multi-dimensional estimated resource overhead values. .

[0048] Next, the resource usage status of each node in the current cluster is collected, generating a real-time snapshot of the cluster resources. The instant query interface of the cluster monitoring component is called to obtain global node status data at millisecond latency, just before the action is to be executed. Real-time CPU utilization, available physical memory, and network bandwidth usage of each node are extracted. To filter data distortion caused by momentary fluctuations, the moving median within a preset time window is used as the current status benchmark. The preset time window is determined based on the sampling period of the monitoring probe, typically three times the sampling period (e.g., if the probe samples every 5 seconds, the window is set to 15 seconds) to smooth out single-point data spikes caused by network latency or instantaneous high-frequency requests. The processed real-time resource values ​​of each node are then organized into a... A matrix format with strictly consistent dimensions and arrangement order generates real-time snapshots of cluster resources. .

[0049] Then, the estimated resource consumption vector is numerically superimposed with the real-time snapshot of cluster resources to obtain the total resource usage. Before performing the superposition calculation, the estimated resource consumption vector is... and Perform unit standardization processing. Read the cluster topology configuration table to obtain the total hardware capacity of each node (such as the total number of CPU cores and the total memory capacity), and use this capacity data to... The absolute physical quantities are converted into relative percentages of occupancy. After unifying the dimensions, the quantities are then adjusted in both the node and resource type dimensions. and Perform matrix addition to obtain the total resource usage. The matrix output by this addition operation Each element in This means that if the optimized recovery path is triggered and issued immediately, the first... The node at the th The expected peak utilization rate of these resources.

[0050] It should be noted that the unit unification process first reads the total hardware capacity of each node from the cluster configuration database, including the total number of CPU cores and the total memory capacity (bytes). Then, it divides the absolute physical quantities in the estimated resource consumption vector, such as the 0.5 CPU cores and 2GB of memory required to add a new container instance on a node, by the total capacity of the corresponding node to obtain the estimated incremental percentage of the node in each resource dimension. Finally, it performs a node-by-node and resource-by-resource addition operation on this incremental percentage and the current occupancy percentage recorded in the real-time snapshot of cluster resources to obtain the total resource occupancy after aggregation.

[0051] Finally, by comparing the total resource consumption with the preset node capacity limit, the number of resource overload risk points is calculated. If the number of resource overload risk points is zero, an execution feasibility confirmation signal is output. It is worth noting that the matrix is ​​traversed one by one. Expected occupancy element Compare it with the preset node bearing capacity limit. Numerical comparisons were performed. The preset upper limit of node bearing capacity was determined. It is not a globally uniform constant, but rather dynamically matched and determined based on the service level of the node. For example, for core database cluster nodes, it combines the inflection point data of memory usage that caused memory overflow crashes in historical stress tests, and then... Set to 0.80; for stateless front-end gateway nodes, the tolerance is higher, so set to 0.90. If a certain element in the matrix... Greater than its corresponding upper limit value This will increase the number of resource overload risk points. Perform an increment operation. After the full traversal and calculation are complete, read... The final value. If A value of zero indicates that the estimated peak overhead of the optimized recovery path is completely within the safe zone of the current cluster's available capacity. An execution feasibility confirmation signal containing the current operation timestamp and a verification code is then generated and transferred to the subsequent automated execution interface. If the value is not zero, the distribution process is terminated, an alarm is triggered, and a secondary path replanning is initiated.

[0052] In step S17, in response to the execution feasibility confirmation signal, the optimized recovery path is mapped to atomic operation code, and dynamic configuration parameters are injected to drive automated execution until the real-time calculated fit is within a preset convergence range, at which point the recovery is determined to be complete, including: In response to the execution feasibility confirmation signal, the optimized recovery path is decomposed into independent basic operation units and mapped to atomic operation code; The connection pool size and bandwidth limit values ​​are extracted from the optimized recovery path and injected into the atomic operation code as dynamic configuration parameters to obtain parameterized atomic operation code. Execute the parameterized atomic operation code and collect the real-time execution feedback stream; Calculate the goodness of fit between the real-time execution feedback stream and the preset system health baseline model. When the goodness of fit is within the preset convergence range, the recovery is determined to be complete.

[0053] First, in response to the feasibility confirmation signal, the optimized recovery path is decomposed into independent basic operation units and mapped to atomic operation code. Specifically, the optimized recovery path in the directed acyclic graph format is parsed, its topology is sorted and expanded, and each independent control command node is extracted as a basic operation unit. A pre-built automated script library is invoked, and a semantic parsing engine is used to translate the business logic of the basic operation unit into a low-level scripting language that can be directly interpreted by the target environment. For example, for containerized clusters, it is mapped to standard API call code for Kubernetes clusters; for traditional physical servers, it is mapped to executable Shell or Python script files, thereby completing the generation of basic code logic.

[0054] Then, the connection pool size and bandwidth limit values ​​are extracted from the optimized recovery path and injected into the atomic operation code as dynamic configuration parameters, resulting in parameterized atomic operation code. Using regular expressions or JSON path parsing technology, the attribute configuration tree of the optimized recovery path is traversed to accurately extract values ​​such as the maximum number of database connections and the gateway token bucket issuance rate. A template rendering engine (e.g., Jinja2 architecture) is introduced, using the extracted connection pool size and bandwidth limit values ​​as context variables to replace the placeholder fields in the atomic operation code template. This injection mechanism decouples the code execution logic from the capacity configuration, ensuring that the final parameterized atomic operation code highly matches the specific load requirements of the current fault scenario without requiring manual hard-coding intervention.

[0055] Next, parameterized atomic operation code is executed, and real-time execution feedback streams are collected. Through secure Remote Procedure Call (RPC) or a locally resident operation agent process, the parameterized atomic operation code is securely distributed to the target faulty node and its associated upstream and downstream nodes, triggering execution. During the parallel and serial execution of the instruction sequence, a high-frequency bypass monitoring probe is simultaneously activated. At a sub-second sampling rate, it continuously retrieves the target process's return status codes, standard business output logs, and instantaneous fluctuations in memory throughput. These heterogeneous monitoring and log data are then stream-assembled and vectorized according to strict timestamps to form a continuous data stream—the real-time execution feedback stream—used to dynamically depict the evolution of the cluster state after the recovery action is issued.

[0056] Finally, the goodness of fit between the real-time execution feedback stream and the preset system health baseline model is calculated. When the goodness of fit is within a preset convergence interval, the recovery is considered complete. It is worth noting that the preset system health baseline model is trained based on long-term characteristics from historical fault-free and stable operation periods. Internally, it maintains multi-dimensional Gaussian distribution parameters of the system's core operating indicators, including a baseline mean vector. With reference covariance matrix Extract the multidimensional feature vector of the real-time execution feedback stream within the current scrolling time window. Calculate the Mahalanobis distance between it and the benchmark model. : In the formula, This represents the vector transpose operation. This represents the matrix inversion operation. The Mahalanobis distance is used to calculate the system's goodness of fit. The conversion formula is: The goodness-of-fit value ranges from 0 to 1, with a higher value indicating that the current system state is closer to the historical healthy baseline. The system performs high-frequency polling calculations. If the value is consistently greater than the preset convergence threshold for a preset time period (e.g., a 60-second observation window), If the fit is successful, the fit is considered to have successfully entered the preset convergence interval. The preset convergence threshold is... The determination of the distance range is based on the confidence interval theory in statistics. A critical reciprocal value (e.g., 0.85) is selected as a fixed threshold, which ensures that historical healthy samples have a 95% probability of falling within this distance range. When the real-time fit continuously exceeds this threshold, it indicates that the system state has a greater than 95% probability of returning to the healthy baseline, thus determining that recovery is complete. Once the condition is triggered, it is determined that the anomaly has been completely eliminated and the service state has been smoothly handed over, officially determining that recovery is complete, thus ending the entire closed-loop control process of automatic detection and recovery.

[0057] In summary, this invention discloses an automatic server anomaly detection and recovery method, which includes collecting real-time logs and monitoring data, extracting abnormal load features to construct a system state vector; generating candidate recovery actions using a deep reinforcement learning model, and simulating and predicting stability gains in a digital twin environment; generating an action priority sequence through Monte Carlo tree search by combining high-load resource estimation; optimizing the recovery path based on topological dependencies; verifying overload risks based on real-time resource snapshots; and automatically deploying and executing the method after confirming feasibility. This solves the problem of low success rate of automatic server recovery in existing technologies.

[0058] Reference Figure 2 The second embodiment of the present invention provides a server anomaly automatic detection and recovery system, comprising: The state awareness module is used to acquire real-time log streams and cluster monitoring data, perform abnormal fluctuation detection, lock abnormal server segments, extract feature indicators of the abnormal server segments, and perform feature dimension mapping to obtain the system state vector. The strategy generation module is used to input the system state vector into a pre-trained action generation network to generate an initial recovery action sequence, and to use a value evaluation network to score and filter candidates for recovery actions. The simulation and deduction module is used to simulate the execution of the candidate recovery action in a preset digital twin environment, generate a predicted cluster load distribution matrix, and if the predicted cluster load distribution matrix indicates that there are no overloaded nodes, then the fault propagation link is identified according to the system state vector, and the potential risk value is calculated. The stability is determined by combining the predicted cluster load distribution matrix to obtain a stability benefit score. The global decision module is used to integrate the stability benefit score with the high load resource consumption estimate corresponding to the predicted cluster load distribution matrix, initialize the preset Monte Carlo search tree, and perform multi-step simulation to generate a sorted action priority sequence. The path optimization module is used to determine the corresponding execution parameters, concurrent execution constraints and consistency verification logic based on the first action in the action priority sequence, and encapsulate them to obtain an optimized recovery path; The risk verification module is used to quantify the estimated resource consumption vector of the optimized recovery path, and perform resource overload risk verification in combination with the real-time snapshot of cluster resources collected in real time, and generate an execution feasibility confirmation signal. The execution control module is used to respond to the execution feasibility confirmation signal, map the optimized recovery path into atomic operation code, and inject dynamic configuration parameters to drive automated execution until the real-time calculated fitting degree is in the preset convergence range, and then determine that the recovery is complete.

[0059] It should be noted that the server anomaly automatic detection and recovery system provided in this embodiment of the invention is used to execute all the process steps of the server anomaly automatic detection and recovery method in the above embodiment. The working principle and beneficial effect of the two are one-to-one, so they will not be described again.

[0060] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0061] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for automatic detection and recovery of server anomalies, characterized in that, include: Acquire real-time log streams and cluster monitoring data, perform abnormal fluctuation detection, locate abnormal server segments, extract feature indicators of the abnormal server segments, and perform feature dimension mapping to obtain the system state vector; The system state vector is input into a pre-trained action generation network to generate an initial recovery action sequence, and a value evaluation network is used to score and filter the results to obtain candidate recovery actions. The candidate recovery action is simulated in a pre-set digital twin environment to generate a predicted cluster load distribution matrix. If the predicted cluster load distribution matrix indicates that there are no overloaded nodes, the fault propagation link is identified according to the system state vector, and the potential risk value is calculated. The stability is determined by combining the predicted cluster load distribution matrix to obtain a stability benefit score. By integrating the stability gain score with the high load resource consumption estimate corresponding to the predicted cluster load distribution matrix, a pre-set Monte Carlo search tree is initialized, and multi-step simulation is performed to generate a sorted action priority sequence. The corresponding execution parameters, concurrent execution constraints, and consistency verification logic are determined based on the first action in the action priority sequence, and an optimized recovery path is encapsulated. The estimated resource consumption vector of the optimized recovery path is quantified, and a resource overload risk verification is performed in combination with the real-time snapshot of cluster resources collected in real time, generating an execution feasibility confirmation signal. In response to the execution feasibility confirmation signal, the optimized recovery path is mapped to atomic operation code, and dynamic configuration parameters are injected to drive automated execution until the real-time calculated fit is within the preset convergence range, at which point the recovery is determined to be complete.

2. The method for automatic detection and recovery of server anomalies according to claim 1, characterized in that, The process involves acquiring real-time log streams and cluster monitoring data, performing abnormal fluctuation detection, identifying abnormal server segments, extracting feature indicators from these abnormal segments, and performing feature dimension mapping to obtain a system state vector, including: Obtain the real-time log stream and the cluster monitoring data; For the real-time log stream and the cluster monitoring data, a time-series aligned dataset is generated based on the timestamp index, and fluctuation detection is performed on the time-series aligned dataset to lock the abnormal server segments; Extract the error stack information from the server anomaly fragment to determine the anomaly type indicator; Construct a resource utilization waveform for the abnormal time window corresponding to the abnormal server segment, and calculate the current load characteristics; The system state vector is generated by mapping the anomaly type index and the current load characteristics into a pre-constructed state vector space and then mapping the feature dimensions.

3. The method for automatic detection and recovery of server anomalies according to claim 1, characterized in that, The process involves inputting the system state vector into a pre-trained action generation network to generate an initial recovery action sequence, and then using a value evaluation network for scoring and filtering to obtain candidate recovery actions, including: The system state vector is input into a pre-trained action generation network, which outputs an action probability distribution matrix. Atomic action units are extracted based on the action probability distribution matrix and combined to generate the initial recovery action sequence; The initial recovery action sequence is scored using a pre-trained value evaluation network combined with the system state vector to obtain a sequence value score; Sequences whose sequence value scores exceed a preset score threshold are selected as candidate recovery actions.

4. The method for automatic detection and recovery of server anomalies according to claim 1, characterized in that, The step of simulating the execution of the candidate recovery action in a pre-set digital twin environment to generate a predicted cluster load distribution matrix includes: The candidate recovery actions are rehearsed in a pre-set digital twin environment, node resource usage change information is recorded, and the virtual cluster state after the rehearsal is generated; Extract the resource usage indicators of each node in the virtual cluster state, and construct the predicted cluster load distribution matrix according to the dimensional mapping relationship between node identifier and resource type.

5. The method for automatic detection and recovery of server anomalies according to claim 1, characterized in that, If the predicted cluster load distribution matrix indicates that there are no overloaded nodes, then the fault propagation path is identified based on the system state vector, and a potential risk value is calculated. This value is then combined with the predicted cluster load distribution matrix to determine stability and obtain a stability gain score, including: If the predicted cluster load distribution matrix indicates that there are no overloaded nodes, then the fault propagation link is identified based on the system state vector; Calculate the potential risk value of the candidate recovery action for the fault propagation link; Based on the predicted cluster load distribution matrix, the resource balance is calculated, and a weighted calculation is performed on the resource balance and the potential risk value to obtain the stability benefit score.

6. The method for automatic detection and recovery of server anomalies according to claim 1, characterized in that, The process involves fusing the stability gain score with the estimated high-load resource consumption corresponding to the predicted cluster load distribution matrix, initializing a pre-set Monte Carlo search tree, and performing multi-step simulation to generate a sorted sequence of action priorities, including: Extract the peak resource usage data from the predicted cluster load distribution matrix to determine the estimated value of high load resource usage; The stability gain score and the estimated high load resource consumption are weighted and the result is assigned as the initial state value of the root node of the preset Monte Carlo search tree. Based on the initial state value of the root node, an action search path is generated, and a multi-step simulation is performed on the action search path to obtain the overall stability impact value of the cluster. The overall stable impact value is backpropagated to the nodes along the Monte Carlo search tree, and the cumulative reward value and access count of the nodes along the tree are updated using the incremental averaging algorithm. The actions are sorted according to the accumulated reward value and the access count to generate the action priority sequence.

7. The method for automatic detection and recovery of server anomalies according to claim 1, characterized in that, The step of determining the corresponding execution parameters, concurrent execution constraints, and consistency verification logic based on the first action in the action priority sequence, and encapsulating them to obtain an optimized recovery path, includes: Extract the node connection relationships involved in the first action in the action priority sequence, determine the node topology, and analyze the data flow and call frequency of the nodes involved in the first action to construct a dependency matrix; The expansion ratio or rate limiting threshold is set according to the node topology and used as the execution parameter; Based on the dependency matrix, single-node operation or mutual exclusion operation logic is set as the concurrent execution constraint; The consistency verification logic is generated by matching the preset verification rules with the type of the first action. The optimized recovery path is generated by integrating the execution parameters, the concurrent execution constraints, and the consistency verification logic.

8. The method for automatic detection and recovery of server anomalies according to claim 1, characterized in that, The quantification of the estimated resource consumption vector of the optimized recovery path, combined with real-time snapshots of cluster resources collected in real time, performs resource overload risk verification and generates an execution feasibility confirmation signal, including: The resource consumption value of each action is quantified from the instruction set included in the optimized recovery path to construct the estimated resource consumption vector; Collect the resource usage status of each node in the current cluster and generate a real-time snapshot of the cluster resources; The estimated resource consumption vector is superimposed with the real-time snapshot of cluster resources to obtain the total resource usage. By comparing the total resource usage with the preset node carrying capacity limit, the number of resource overload risk points is counted. If the number of resource overload risk points is zero, the execution feasibility confirmation signal is output.

9. The method for automatic detection and recovery of server anomalies according to claim 1, characterized in that, In response to the feasibility confirmation signal, the optimized recovery path is mapped to atomic operation code, and dynamically configured parameters are injected to drive automated execution until the real-time calculated fit is within a preset convergence range, at which point the recovery is determined to be complete, including: In response to the execution feasibility confirmation signal, the optimized recovery path is decomposed into independent basic operation units and mapped to atomic operation code; The connection pool size and bandwidth limit values ​​are extracted from the optimized recovery path and injected into the atomic operation code as dynamic configuration parameters to obtain parameterized atomic operation code. Execute the parameterized atomic operation code and collect the real-time execution feedback stream; Calculate the goodness of fit between the real-time execution feedback stream and the preset system health baseline model. When the goodness of fit is within the preset convergence range, the recovery is determined to be complete.

10. A server anomaly automatic detection and recovery system, characterized in that, include: The state awareness module is used to acquire real-time log streams and cluster monitoring data, perform abnormal fluctuation detection, lock abnormal server segments, extract feature indicators of the abnormal server segments, and perform feature dimension mapping to obtain the system state vector. The strategy generation module is used to input the system state vector into a pre-trained action generation network to generate an initial recovery action sequence, and to use a value evaluation network to score and filter candidates for recovery actions. The simulation and deduction module is used to simulate the execution of the candidate recovery action in a preset digital twin environment, generate a predicted cluster load distribution matrix, and if the predicted cluster load distribution matrix indicates that there are no overloaded nodes, then the fault propagation link is identified according to the system state vector, and the potential risk value is calculated. The stability is determined by combining the predicted cluster load distribution matrix to obtain a stability benefit score. The global decision module is used to integrate the stability benefit score with the high load resource consumption estimate corresponding to the predicted cluster load distribution matrix, initialize the preset Monte Carlo search tree, and perform multi-step simulation to generate a sorted action priority sequence. The path optimization module is used to determine the corresponding execution parameters, concurrent execution constraints and consistency verification logic based on the first action in the action priority sequence, and encapsulate them to obtain an optimized recovery path; The risk verification module is used to quantify the estimated resource consumption vector of the optimized recovery path, and perform resource overload risk verification in combination with the real-time snapshot of cluster resources collected in real time, and generate an execution feasibility confirmation signal. The execution control module is used to respond to the execution feasibility confirmation signal, map the optimized recovery path into atomic operation code, and inject dynamic configuration parameters to drive automated execution until the real-time calculated fitting degree is in the preset convergence range, and then determine that the recovery is complete.

Images