Fault handling method and electronic device

Abstract

The application discloses a fault processing method and electronic equipment, relates to the technical field of reinforcement learning, and comprises the following steps: collecting monitoring data of a target device and preprocessing the monitoring data to obtain a structured feature vector; constructing a multi-dimensional state vector according to the structured feature vector and the latest processed data; constructing a composite processing space according to a repair action subset for executing different repair operations and a prediction model subset for selecting different fault prediction models; training a processing model through asynchronous reinforcement learning according to the composite processing space and a training data set to obtain a target processing model; and inputting the state vector into the target processing model to obtain a processing result, wherein the processing result comprises a fault prediction result and / or a repair action, so that the technical problem that prediction and repair are disconnected in the prior art is solved, and the technical effect that the processing efficiency and the model training speed in a complex operation and maintenance scene are significantly improved is achieved.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to a fault handling method and electronic device. Background Technology

[0002] With the rapid development of information technology, electronic devices, as a key component of modern production and life, require paramount stability. Traditional methods relying on manual monitoring and handling are no longer sufficient to meet the continuous requirements for reliable equipment operation. Although predictive maintenance technology based on machine learning has emerged, it still has significant limitations: multi-model fusion is often used to improve predictive capabilities, leading to system complexity and high computational resource consumption; after model deployment, there is a lack of dynamic adjustment mechanisms, making it difficult to adapt to changes in the external environment and internal state; at the same time, the degree of system automation is limited, and fault repair still relies on manual intervention. Existing technologies generally suffer from a disconnect between prediction and repair: predictive models often focus only on optimizing accuracy, failing to connect with the actual effectiveness of repair measures; repair strategies are mostly static and rigid, unable to respond dynamically based on prediction results; and the system as a whole lacks a mechanism for continuous learning and iteration from the final business impact, restricting its adaptability to new types of faults and overall operational efficiency.

[0003] Therefore, in view of the shortcomings of existing technical solutions, the present invention provides a fault handling method. Summary of the Invention

[0004] This application provides a fault handling method and electronic device to at least address the problem of a disconnect between prediction and repair in related technologies.

[0005] This application provides a fault handling method, which includes: collecting and preprocessing monitoring data of a target device to obtain a structured feature vector; constructing a multi-dimensional state vector based on the structured feature vector and the most recent processed data; constructing a composite processing space based on a subset of repair actions used to perform different repair operations and a subset of prediction models used to select different fault prediction models; training the processing model through asynchronous reinforcement learning based on the composite processing space and a training dataset to obtain a target processing model; and inputting the state vector into the target processing model to obtain a processing result, wherein the processing result includes a fault prediction result and / or a repair action.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described fault handling methods when executing the computer program.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described fault handling methods.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described fault handling methods.

[0009] This application achieves the following: by collecting and preprocessing monitoring data of the target device to obtain a structured feature vector; constructing a multi-dimensional state vector based on the structured feature vector and the most recent processed data; constructing a composite processing space based on a subset of repair actions used to perform different repair operations and a subset of prediction models used to select different fault prediction models; training the processing model through asynchronous reinforcement learning based on the composite processing space and the training dataset to obtain the target processing model; and inputting the state vector into the target processing model to obtain the processing result, which includes the fault prediction result and / or repair actions. Therefore, by incorporating the selection of the fault prediction model and the repair measures into a unified reinforcement learning action space for collaborative optimization, end-to-end joint dynamic optimization of fault diagnosis and repair strategies is achieved, significantly improving the processing efficiency and model training speed in complex operation and maintenance scenarios, enhancing the accuracy and timeliness of fault handling, reducing the average repair time, and improving the overall availability and stability of electronic equipment clusters. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a fault handling method provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of the system architecture for a fault handling method according to an embodiment of the present application. Figure 3 A schematic diagram of the model training process for a fault handling method provided in an embodiment of this application; Figure 4 A structural block diagram of a fault handling device provided in an embodiment of this application; Figure 5 This is an internal structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The embodiments of this application provide a fault handling method, and the method is described in detail below in conjunction with the execution flow of the fault handling method.

[0017] S101: Collect and preprocess the monitoring data of the target device to obtain a structured feature vector.

[0018] Here, monitoring data can include hardware data, software data, log data, and network data.

[0019] The target device is an electronic device, specifically a server.

[0020] Hardware data can be collected through the hardware monitoring interface of the target device, and may include data such as CPU (Central Processing Unit) temperature, CPU utilization, memory usage, disk I / O (Input / Output), fan speed, and hard drive read / write speed.

[0021] Software data can be collected through application performance monitoring tools and may include data such as application response time, database query speed, and number of threads.

[0022] Log data can be collected through a log management system and may include operating system logs, application logs, error log frequency, and other data.

[0023] Network data can be collected through network monitoring tools and may include data such as packet size, transmission rate, and packet loss rate.

[0024] Specifically, time alignment and synchronization are performed on monitoring data with different sampling frequencies.

[0025] In one embodiment, monitoring data from the target device is collected and preprocessed to obtain a structured feature vector, including: The monitoring data is cleaned and normalized to obtain the first monitoring data; Based on the data type of the first monitoring data, determine the corresponding feature extraction method; Based on the feature extraction method, the data features of the first monitoring data are extracted to obtain a structured feature vector.

[0026] Here, data cleaning is the process of detecting, identifying, correcting, or deleting erroneous, incomplete, duplicate, inconsistent, or irrelevant parts of raw data.

[0027] Data cleaning can include removing noise and outliers from monitoring data, handling missing values, correcting inconsistent data, handling duplicate data, and correcting data types. For example, it can involve deleting obviously erroneous temperature readings or unreasonable log records, and filling in missing data through interpolation or completion methods.

[0028] Here, normalization is the process of scaling numerical features to a uniform and specific range in order to eliminate the influence of differences in units and value ranges between different features.

[0029] The normalization process can include min-max normalization and Z-score normalization.

[0030] Here, feature extraction is the process of extracting useful features from raw data, such as extracting error codes from logs or extracting average latency from network data.

[0031] The data types can include time-series data, text data, and image data.

[0032] Specifically, for time series data, features such as trends, seasonality, and periodicity are extracted; for text data, features such as word frequency, TF-IDF, and word embeddings are extracted using natural language processing techniques; and for image data, high-level features of the image are extracted using convolutional neural networks.

[0033] Specifically, feature extraction also includes statistical features, such as calculating statistical measures of the data, like mean, variance, standard deviation, maximum value, minimum value, etc.

[0034] This can improve the quality and availability of data, and at the same time, through automatic feature extraction, the optimal features can be automatically extracted and selected.

[0035] S102: Construct a multidimensional state vector based on the structured feature vector and the most recently processed data.

[0036] The multidimensional state vector includes current data and historical data, and is a comprehensive vector that integrates objective indicators, subjective cognition, and behavioral history.

[0037] S103: Construct a composite processing space based on a subset of repair actions used to perform different repair operations and a subset of prediction models used to select different fault prediction models.

[0038] The repair action subset may include restarting, resource release and cleanup, configuration management and rollback, execution of repair scripts, re-establishing network connections, alerting administrators, and reducing the load on target devices.

[0039] Each subset of repair actions includes the applicable fault type, triggering conditions, specific repair action sequence, expected effect, and risk assessment.

[0040] The predictive model subset can include traditional machine learning models (such as random forests and gradient boosting decision trees), time series models (such as LSTM (Long Short-Term Memory) networks and autoregressive ensemble moving average models), anomaly detection models, and graph neural networks. For example, LSTM, random forests, isolated forests, and SVMs are used to capture temporal dependencies, handle structured features, detect anomalies, and classify few samples, respectively.

[0041] Specifically, based on the characteristics of different fault types, the association between fault types and corresponding subsets of prediction models is established, and the prediction models are pre-trained offline. Different state vectors s are converted into the input format required by each model, and the outputs (fault probability, type) of each model are normalized to a unified format to establish a candidate model pool.

[0042] In one embodiment, based on the behavioral context features in the current multidimensional state vector, some actions in the composite processing space can be temporarily masked or their probabilistic weights adjusted. This includes: counting the number of consecutive executions of the same action in the behavioral context features; if the number exceeds a preset anti-oscillation threshold, removing the same action from the prediction model subset in the current processing step; and dynamically adjusting the exploration probability weights of various actions in the composite processing space based on the time interval between the last execution of a repair or prediction action in the behavioral context features. Specifically, dynamically adjusting the exploration probability weights of various actions in the composite processing space based on the time interval between the last execution of a repair or prediction action in the behavioral context features includes: if the time interval between the last execution of a repair action is less than a repair cooling threshold, reducing the exploration probability weight of the repair action by a preset ratio; and if the time interval between the last selection of a prediction model is less than a prediction cooling threshold, reducing the exploration probability weight of the selected prediction model by a preset ratio.

[0043] Among them, the anti-vibration shielding means that if the status shows that the same action (such as restarting the service) has been executed three times in a row (anti-vibration threshold), the current step will temporarily shield the action and force the attempt to try other strategies to avoid system oscillation.

[0044] The cooling-off period probability suppression is as follows: if the last time a repair action (such as clearing the cache) was performed was less than 60 seconds ago (repair cooling threshold), the random selection probability weight of all repair actions in the ε-greedy strategy is reduced by 50%; if the last time a prediction action (such as calling the LSTM model) was performed was less than 10 seconds ago (prediction cooling threshold), the random selection probability weight of all prediction actions is reduced by 70%.

[0045] In this way, excessive operation can be suppressed, resource allocation can be optimized and processing rationality can be improved; processing oscillation and invalid loops can be effectively prevented, and the stability of system operation can be guaranteed.

[0046] S104: Based on the composite processing space and the training dataset, the processing model is trained through asynchronous reinforcement learning to obtain the target processing model.

[0047] The training dataset may include historical processing data, simulation data, etc.

[0048] The training dataset is used to generate training state vectors.

[0049] Among them, reinforcement learning can be Q-learning.

[0050] Here, reinforcement learning can be an asynchronous reinforcement learning mechanism, which includes multiple threads that explore multiple copies of the environment in parallel and independently.

[0051] S105: Input the state vector into the target processing model to obtain the processing results, which include fault prediction results and / or repair actions.

[0052] When performing a repair action, the corresponding fault type can be inferred based on the repair action.

[0053] The fault prediction result is obtained by selecting one of multiple candidate fault prediction models for the current state to obtain the fault handling result.

[0054] Among them, the repair action is to perform specific repair operations in response to the predicted fault risks.

[0055] The processing model directly outputs an action corresponding to an action identifier from the composite processing space based on the current state vector, without requiring external logic to determine whether prediction or repair should be performed.

[0056] It should be noted that this application achieves end-to-end joint dynamic optimization of fault diagnosis and repair strategies by incorporating the selection of fault prediction models and repair measures into a unified reinforcement learning action space for collaborative optimization. This significantly improves the processing efficiency and model training speed in complex operation and maintenance scenarios, enhances the accuracy and timeliness of fault handling, reduces the average repair time, and improves the overall availability and stability of the target equipment cluster.

[0057] In some specific implementations, a multidimensional state vector is constructed based on the structured feature vector and the most recently processed data, including: Based on the most recent processing result, collect the predicted data and context data of the most recent processing result to determine the most recent processing data; In response to the detection that the target device is in an initial state, the most recently processed data is set to the default value.

[0058] The prediction data for the most recent processing result may include the failure probability, failure type confidence, and prediction model identifier obtained from the most recent processing action.

[0059] The context data of the processing result may include the type of processing action executed in the previous step, the time interval since the last processing, the number of consecutive identical operations, and the processing round.

[0060] For example, the state vector s can be s=[ #1. Basic Monitoring Features 0.85, #CPU utilization 0.92, #Memory usage 120.5, #Error log / minute ...,# #2. Predicting Cognitive Features 0.93, #fault_prob (probability of failure) 1,0,0,#fault_type (fault type):[Memory leak, disk failure, network error] 0,1,0,0,0,0,#model_used (prediction model): RF (Random Forest) was selected. 0.25, #prediction_age (time since the last prediction) #3. Action Context Features 0,1,0,0,0,0,#last_action (the action performed in the previous step): RF (predictive action) was selected last time. 0, #repair_executed_recently (whether a repair has been performed in the past N seconds) = False 1. #prediction_executed_recently (whether a prediction has been executed in the past M seconds) = True 1, #consecutive_same_action (number of times the same action is executed consecutively) = 1 42#episode_step (current processing round) = 42 ] The method for determining the number of consecutive executions of the same action (consecutive_same_action) is as follows: If the number exceeds a preset anti-vibration threshold, the same action will be removed from the current subset of prediction models. For example, if the anti-vibration blocking status shows that the same action (such as restarting the service) has been executed three times consecutively (anti-vibration threshold), then the action will be temporarily blocked, forcing the attempt of other strategies to avoid system oscillations. The method for determining the time interval since the last execution of a repair or prediction action (prediction_age) dynamically adjusts the exploration probability weight of each action in the composite space. This includes: if the time interval since the last execution of a repair action is less than the repair cooling threshold, then the exploration probability weight of the repair action will be reduced by a preset ratio; if the time interval since the last selection of a prediction model is less than the prediction cooling threshold, then the exploration probability weight of the selected prediction model will be reduced by a preset ratio. For example, if the last execution of a repair action (such as clearing the cache) is less than 60 seconds ago (repair cooling threshold), then the random selection probability weight of all repair actions in the ε-greedy strategy will be reduced by 50%; if the last execution of a prediction action (such as calling the LSTM model) is less than 10 seconds ago (prediction cooling threshold), then the random selection probability weight of all prediction actions will be reduced by 70%.

[0061] Specifically, the default value can be a preset zero vector or a historical average. For example, if no prediction is performed, the probability of failure is set to -1 or 0, all failure types are set to 0, and all prediction models are set to none.

[0062] In this way, by integrating basic monitoring features, predictive cognitive features, and behavioral context features to construct a multi-dimensional state vector, the system can have a comprehensive perception of the target device's operating status, fault recognition stage, and processing history, thereby enhancing the system's adaptability and interpretability to dynamic operation and maintenance scenarios.

[0063] In some specific implementations, before constructing the composite processing space based on subsets of repair actions used to perform different repair operations and subsets of prediction models used to select different fault prediction models, the method further includes: Obtain multiple fault types corresponding to the target device; Based on multiple fault types of the target equipment, determine the corresponding repair actions for each fault type to obtain a subset of repair actions; Based on the multiple fault types of the target equipment, determine the monitoring indicators corresponding to the multiple fault types; Based on the monitoring indicators, multiple fault prediction models corresponding to the target equipment are determined, and a subset of prediction models is obtained.

[0064] Specifically, historical fault data and corresponding monitoring indicators of the target equipment are collected; the predictability of various faults and related indicators are analyzed; and a set of well-performing and complementary prediction models are selected based on data characteristics and fault modes. For example, a time series model (LSTM), a tree model (random forest), and a linear model (logistic regression) can be included as candidates.

[0065] In this way, deep synergy between prediction and repair is achieved, avoiding the inefficiency or conflict caused by the disconnect between the two in traditional systems.

[0066] In some specific implementations, a composite processing space is constructed based on a subset of repair actions used to perform different repair operations and a subset of prediction models used to select different fault prediction models, including: Obtain multiple fault types corresponding to the target device and determine a subset of fault types; A three-dimensional performance evaluation matrix is ​​generated based on the subsets of fault types, repair actions, and prediction models. Based on historical data processing, establish a multi-objective performance evaluation vector; The multi-objective Pareto optimization algorithm is used to calculate the multi-objective performance evaluation vector of each element in the three-dimensional performance evaluation matrix, and the comprehensive performance of each element is obtained. Based on comprehensive performance, a three-dimensional performance evaluation matrix is ​​selected to construct a composite processing space.

[0067] Here, multi-objective Pareto optimization refers to the optimization method of finding a set of Pareto optimal solutions when simultaneously optimizing multiple conflicting objective functions.

[0068] Each element in the three-dimensional performance evaluation matrix represents a composite processing action consisting of a specific fault type, repair action, and prediction model.

[0069] The three-dimensional performance evaluation matrix consists of several fault types for servers, including at least one of hardware failure, software anomaly, performance degradation, and configuration error. The second dimension is a subset of repair actions, including at least one of service restart, cache clearing, resource reallocation, configuration rollback, and component replacement. The third dimension is a subset of prediction models, including at least one of time-series-based prediction models, statistical feature-based classification models, deep learning-based anomaly detection models, and association rule-based analysis models.

[0070] The multi-objective performance evaluation vector can include prediction accuracy metrics, repair effectiveness metrics, resource efficiency metrics, and comprehensive stability metrics. Prediction accuracy metrics are characterized by at least one of the following: accuracy, recall, and F1 score of the associated prediction model on historical data. Repair effectiveness metrics are characterized by at least one of the following: success rate, mean recovery time, and failure recurrence rate of the associated repair actions in historical execution. Resource efficiency metrics are characterized by at least one of the following: computational resources, storage resources, and network resources consumed in executing the composite processing action. Comprehensive stability metrics are characterized by at least one of the following: mean time between failures (MTBF) and performance fluctuation coefficient of the system after execution.

[0071] Specifically, a Pareto optimal solution set is identified by iteratively comparing the multi-dimensional performance evaluation vectors of each candidate composite processing action using a non-dominated sorting genetic algorithm. The Pareto optimal solution set contains multiple non-dominated candidate composite processing actions, each of which is superior to other non-Pareto solutions in at least one objective dimension. All candidate composite processing actions in the Pareto optimal solution set are constructed into a composite processing space, and each action is represented and stored in the space in the form of a multi-dimensional performance evaluation vector.

[0072] In this way, a dynamic composite processing space that is comprehensive, intelligent, and adaptive can be constructed.

[0073] In some specific implementations, asynchronous reinforcement learning includes a first training thread and a second training thread. Based on the composite processing space and the training dataset, the processing model is trained using asynchronous reinforcement learning to obtain the target processing model, including: In response to the detection that the first training thread is running at the first frequency, the processing result is determined based on the current parameters of the processing model and the greedy policy; Based on the processing results, determine the reward value and the next state; Based on the current state, processing result, reward value, and next state, generate a quadruple data and store it in the buffer. In response to the detection that the second training thread is running at the second frequency, the parameters of the processing model are sampled from the buffer, updated by the temporal difference learning algorithm, and the updated parameters are synchronized to the first training thread.

[0074] Here, the first training thread is used to perform processing, and the second training thread is used to update parameters.

[0075] In this process, the first training thread involves high-frequency interaction, while the second training thread involves low-frequency learning. The first training process trains the processing model using asynchronous reinforcement learning based on the composite action space and the training dataset, resulting in the target processing model.

[0076] Here, the greedy strategy always chooses the action with the highest estimated value that is currently known at each step.

[0077] Among them, the temporal difference learning algorithm can be the Q-learning algorithm.

[0078] During the training process, the exploration value The decay process proceeds from large to small.

[0079] The first frequency is greater than the second frequency.

[0080] Specifically, generate a random number, and when the random number is greater than the exploration value... At that time, with probability Perform random exploration; when the random number is less than or equal to When argmaxaQ(st,a;θinference)a is selected (i.e., the maximum value).

[0081] Specifically, if the processing result is a prediction result: call the corresponding prediction model and update the corresponding data fields; if the processing result is a repair action: execute the automated script and wait for a stabilization period (e.g., 30 seconds) before collecting the new status.

[0082] Specifically, the process of the first training thread includes: obtaining the current multidimensional state vector, selecting action at through the ε-greedy policy; performing processing operations and observing feedback; calculating the reward value Rt, and giving a high positive reward when the system finally recovers to stability to ensure end-to-end effect orientation; and writing the quadruple (st, at, Rt, st+1) into the buffer.

[0083] Thus, by adopting an asynchronous experience replay architecture, high-frequency processing and low-frequency parameter optimization are decoupled, balancing real-time response capability and learning stability.

[0084] In some specific implementations, the processing result is determined based on the current parameters of the processing model and the greedy strategy, including: Obtain the base exploration value for the greedy strategy; Monitor the data utilization rate of the buffer zone; When the data utilization rate is less than a preset threshold, the basic exploration value is increased by a fixed amount to determine the initial value; When the data utilization rate is greater than or equal to a preset threshold, the basic exploration value is used as the initial value; Calculate the current exploration value corresponding to the greedy strategy based on the number of training steps, the preset exploration rate decay rate, and the initial value.

[0085] Here, the base exploration value is a preset, fixed initial exploration rate, which is the probability that the processing model will randomly select actions in the early stages of training.

[0086] Here, data utilization rate is the ratio of the number of samples taken from the buffer per unit time to the total capacity of the buffer.

[0087] The fixed range can be 0.05, for example, from 0.9 to 0.95.

[0088] In one embodiment, the base exploration value can be determined based on problem complexity, prior knowledge, training strategy, environmental dynamics, and / or reward sparsity. Specifically, the adjustment rule for problem complexity can be to increase the initial ε to allow for sufficient exploration as the environment becomes more complex, the state space larger, and the action space larger; the adjustment rule for prior knowledge can be to decrease the initial ε if prior knowledge of the problem exists (e.g., knowing that certain strategies are better); the adjustment rule for the training strategy can be to start from completely random (ε=1) and gradually decrease it, or start from a moderate value; the adjustment rule for environmental dynamics can be to increase the initial ε if the environment is non-stationary (i.e., the environment changes over time); and the adjustment rule for reward sparsity can be to increase the initial ε in environments with sparse rewards.

[0089] In this way, the initial exploration value is dynamically set according to the problem complexity and reward sparsity, avoiding suboptimal solutions caused by fixing ε, improving environmental adaptability, and increasing sample efficiency.

[0090] In some specific implementations, the reward value and the next state are determined based on the processing result, including: The next state of the target device after monitoring and processing results; Based on the state vector, processing results, and the next state of the target device, calculate the prediction accuracy, system stability, and / or repair efficiency. The reward value is calculated based on the prediction accuracy, system stability, and / or repair efficiency, combined with the corresponding weighting coefficients.

[0091] Here, the reward value is the immediate reward obtained after performing action a (i.e., processing operation) in state s and transitioning to the new state s'.

[0092] The prediction accuracy can be designed based on indicators such as F1 score and AUC (Area Under the Curve). The prediction accuracy adjustment rule is as follows: if the selected prediction model accurately predicts the subsequent failure, a positive reward is given; if a false alarm or missed alarm occurs, a negative reward is given.

[0093] The system stability adjustment rules are as follows: if the key indicators of the target device (such as CPU temperature and latency) return to the normal range after the repair action is performed, and no new problems are caused, a positive reward will be given; if the system status deteriorates or new anomalies occur, a negative reward will be given.

[0094] The repair efficiency adjustment rule is as follows: if the repair action is successful and takes little time, a positive reward is given; if the action is ineffective or consumes too many resources, a negative reward is given.

[0095] The weighting coefficient can be configured according to the business priority.

[0096] Specifically, the reward value can be calculated using the following formula: R(s,a)=α·accuracy(s,a)+β·stability(s,a)+γ·efficiency(s,a), where accuracy(s,a) is the prediction accuracy, stability(s,a) is the system stability, efficiency(s,a) is the repair efficiency, and α, β and γ are adjustable weight coefficients.

[0097] In this way, by integrating reward functions from multiple dimensions, the system can simultaneously optimize multiple operational goals during the learning process, and flexibly adjust the optimization focus according to business needs through weight coefficients, thereby improving the system's practicality and configurability.

[0098] In some specific implementations, the parameters of the processing model are updated using a temporal difference learning algorithm, including: Sample a training sample from the buffer; Determine whether the next state of the training sample after processing is a termination state; In response to the next state being the termination state, the reward value is calculated as the maximum expected long-term reward value of the training sample. In response to the next state not being a terminating state, the target expected long-term reward value of the training samples is calculated using the following formula: y = R + γ·maxa'Q(s',a';θtarget); where y is the target expected long-term reward value; R is the reward value; s' is the next state; γ is a discount factor used to control the importance of future rewards; maxa'Q(s',a';θtarget) is the maximum expected long-term reward value of the next state calculated by the target network; and θtarget is the target network parameter, which is periodically updated to the online network parameter. The expected long-term online return value of the training samples is calculated using the parameters of the online network. Calculate the loss between the target expected long-term return and the online expected long-term return, and update the online network parameters using gradient descent based on the loss; Upon completion of the update, the parameters in the processing model are synchronized to the online network parameters.

[0099] During initialization, the target network parameters, online network parameters, and processing model parameters are the same.

[0100] The next state is the state that is transitioned to after the action (i.e. the processing result) is performed.

[0101] The target network parameters are synchronized every preset number of steps, such as 100 steps; the online network parameters are updated with each batch of data; and the parameters of the processing model are synchronized after each training iteration.

[0102] Specifically, the process of the second training thread includes: randomly sampling small batches of data from the buffer, calculating the target Q value for each sample, calculating the loss and updating the parameters, and synchronizing the policy parameters.

[0103] This enables the model to accurately assess the long-term cumulative benefits of actions, guiding the system to learn to not only focus on immediate repair effects but also take into account long-term system stability, thus enhancing the foresight of the processing.

[0104] In some specific implementations, the state vector is input into the target processing model to obtain the processing result, including: Based on the state vector, the expected long-term return assessment value of each processing result in the composite processing space is calculated using the target processing model. Based on the expected long-term return assessment value, select and determine the target repair operation and / or the target fault prediction model from the composite treatment space and obtain the prediction results.

[0105] Specifically, the action with the highest expected long-term return is selected as the processing result.

[0106] In one embodiment, Figure 2 This is a schematic diagram of the system architecture in an embodiment of this application, such as... Figure 2 As shown, the system architecture in this application includes: a data collection module, a data preprocessing module, a reinforcement learning module, a fault prediction / repair module, and a continuous optimization module.

[0107] Specifically, the data collection module includes hardware monitoring, software monitoring, a logging system, and network monitoring.

[0108] Specifically, the data preprocessing module includes data cleaning, data normalization, feature extraction, context construction, and state output.

[0109] Specifically, the continuous optimization module is used to optimize the feature set and feedback data, and update model parameters. The execution flow of the continuous optimization module includes: continuously recording the entire process data of each "prediction-repair" cycle, including: input state s, selected prediction / repair action, prediction probability, whether a failure actually occurred, system state s' after repair, and calculated reward R; merging newly collected feedback data with historical data to form an updated dataset; using this dataset to retrain or incrementally learn the reinforcement learning module; evaluating the performance of the updated model on the validation set using metrics such as F1 score and accuracy; when the new model significantly outperforms the old model, deploying it to the production environment to replace the old model; and dynamically adjusting the exploration value based on real-time data streams. Parameters such as these are adjusted to adapt to environmental changes.

[0110] In one embodiment, Figure 3 This is a schematic diagram of the model training process in the embodiments of this application, such as... Figure 3 As shown, the model training process in this application includes: obtaining the server (i.e., target device) state S (i.e., structure vector), obtaining random numbers, and determining whether the random numbers are greater than 1. (i.e., exploration value). When the random number is greater than the exploration value, execute A=argmaxQ(s,a). When the random number is less than or equal to the exploration value, randomly select an action a, observe the new state and reward R, update the Q value, update the policy, and determine whether convergence or the maximum number of iterations has been reached. If yes, end; if no, return to obtain the server (i.e., target device) state S.

[0111] It should be understood that, although Figures 1-3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Furthermore, Figures 1-3At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0113] Embodiments of this application also provide a fault handling apparatus, comprising: a first processing module 401 for collecting and preprocessing monitoring data of a target device to obtain a structured feature vector; a second processing module 402 for constructing a multi-dimensional state vector based on the structured feature vector and the most recent processing data; a third processing module 403 for constructing a composite processing space based on a subset of repair actions for performing different repair operations and a subset of prediction models for selecting different fault prediction models; a fourth processing module 404 for training a processing model through asynchronous reinforcement learning based on the composite processing space and a training dataset to obtain a target processing model; and a fifth processing module 405 for inputting the state vector into the target processing model to obtain a processing result, wherein the processing result includes a fault prediction result and / or a repair action.

[0114] In a preferred embodiment of this application, the second processing module 402 is specifically used to: collect the prediction data of the most recent processing result and the context data of the processing result based on the most recent processing result, and determine the most recent processing data; and set the most recent processing data as a default value in response to detecting that the target device is in an initial state.

[0115] In a preferred embodiment of this application, the third processing module 403 is specifically used to: acquire multiple fault types corresponding to the target device; determine repair actions corresponding to the multiple fault types based on the multiple fault types of the target device, and obtain a subset of repair actions; determine monitoring indicators corresponding to the multiple fault types based on the multiple fault types of the target device; and determine multiple fault prediction models corresponding to the target device based on the monitoring indicators, and obtain a subset of prediction models.

[0116] As a preferred implementation, in this embodiment of the application, the third processing module 403 is further configured to: acquire multiple fault types corresponding to the target device and determine a subset of fault types; generate a three-dimensional performance evaluation matrix based on the subset of fault types, the subset of repair actions, and the subset of prediction models; establish a multi-objective performance evaluation vector based on historical processing data; calculate the multi-objective performance evaluation vector of each element in the three-dimensional performance evaluation matrix using a multi-objective Pareto optimization algorithm to obtain the comprehensive performance of each element; and filter the three-dimensional performance evaluation matrix based on the comprehensive performance to construct a composite processing space.

[0117] In a preferred embodiment of this application, the fourth processing module 404 is specifically configured to: in response to detecting that the first training thread is running at a first frequency, determine the processing result based on the current parameters of the processing model and the greedy strategy; determine the reward value and the next state according to the processing result; generate quadruple data and store it in a buffer according to the current state, the processing result, the reward value and the next state; and in response to detecting that the second training thread is running at a second frequency, sample from the buffer, update the parameters of the processing model through a temporal difference learning algorithm, and synchronize the updated parameters to the first training thread.

[0118] In a preferred embodiment of this application, the fourth processing module 404 is further configured to: obtain the basic exploration value of the greedy strategy; monitor the data utilization rate of the buffer; in response to the data utilization rate being less than a preset threshold, increase the basic exploration value by a fixed amount to determine the initial value; in response to the data utilization rate being greater than or equal to the preset threshold, use the basic exploration value as the initial value; and calculate the current exploration value corresponding to the greedy strategy based on the number of training steps, the preset exploration rate decay rate, and the initial value.

[0119] In a preferred embodiment of this application, the fourth processing module 404 is further configured to: monitor the next state of the target device after the processing result; calculate the prediction accuracy, system stability, and / or repair efficiency based on the state vector, the processing result, and the next state of the target device; and calculate the reward value based on the prediction accuracy, system stability, and / or repair efficiency, combined with the corresponding weight coefficients.

[0120] In a preferred implementation, in this embodiment, the fourth processing module 404 is further configured to: sample a training sample from the buffer; determine whether the next state of the training sample after processing is a termination state; in response to the next state being a termination state, calculate the reward value as the maximum expected long-term return value of the training sample; in response to the next state not being a termination state, calculate the target expected long-term return value of the training sample using the following formula: y = R + γ·maxa'Q(s',a';θtarget); where y is the target expected long-term return value; R is the reward value; s' is the next state; γ is a discount factor used to control the importance of future rewards; maxa'Q(s',a';θtarget) is the maximum expected long-term return value of the next state calculated by the target network; θtarget is the target network parameter, periodically updated to the online network parameter; calculate the online expected long-term return value of the training sample using the parameters of the online network; calculate the loss between the target expected long-term return and the online expected long-term return, and update the online network parameter using gradient descent based on the loss; in response to the completion of the update, synchronize the parameters in the processing model to the online network parameter.

[0121] As a preferred implementation, in this embodiment of the application, the fifth processing module 405 is specifically used to: calculate the expected long-term return evaluation value of each processing result in the composite processing space based on the state vector using the target processing model; and select and determine the target repair operation and / or determine the target fault prediction model from the composite processing space based on the expected long-term return evaluation value and obtain the prediction result.

[0122] For a description of the features in the embodiment corresponding to the fault handling device, please refer to the relevant description of the embodiment corresponding to the fault handling method, which will not be repeated here.

[0123] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described fault handling method embodiments.

[0124] The electronic device can be the target device, and its internal structure diagram can be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores fault handling data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a fault handling method.

[0125] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described fault handling method embodiments when it is run.

[0126] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0127] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described fault handling method embodiments.

[0128] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described fault handling method embodiments.

[0129] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0130] The foregoing has provided a detailed description of a fault handling method, electronic device, storage medium, and computer program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application; the descriptions of the above embodiments are only intended to aid in understanding the method and core ideas of this application.

Claims

1. A fault handling method, characterized in that, The method includes: Collect and preprocess monitoring data from the target device to obtain structured feature vectors; Based on the structured feature vector and the most recently processed data, construct a multidimensional state vector; A composite processing space is constructed based on a subset of repair actions used to perform different repair operations and a subset of prediction models used to select different fault prediction models. Based on the composite processing space and the training dataset, the processing model is trained through asynchronous reinforcement learning to obtain the target processing model; The state vector is input into the target processing model to obtain the processing result, wherein the processing result includes fault prediction result and / or repair action.

2. The fault handling method according to claim 1, characterized in that, The step of constructing a multidimensional state vector based on the structured feature vector and the most recently processed data includes: Based on the most recent processing result, collect the predicted data of the most recent processing result and the context data of the processing result to determine the most recent processing data; In response to detecting that the target device is in an initial state, the most recently processed data is set to a default value.

3. The fault handling method according to claim 1, characterized in that, Before constructing the composite processing space based on subsets of repair actions used to perform different repair operations and subsets of prediction models used to select different fault prediction models, the method further includes: Obtain multiple fault types corresponding to the target device; Based on the multiple fault types of the target device, determine the corresponding repair actions for the multiple fault types, and obtain the subset of repair actions; Based on the multiple fault types of the target device, determine the monitoring indicators corresponding to the multiple fault types; Based on the monitoring indicators, multiple fault prediction models corresponding to the target device are determined, and a subset of the prediction models is obtained.

4. The fault handling method according to claim 1, characterized in that, The construction of a composite processing space based on subsets of repair actions used to perform different repair operations and subsets of prediction models used to select different fault prediction models includes: Obtain multiple fault types corresponding to the target device and determine a subset of fault types; A three-dimensional performance evaluation matrix is ​​generated based on the fault type subset, the repair action subset, and the prediction model subset. Based on historical data processing, establish a multi-objective performance evaluation vector; The multi-objective performance evaluation vector of each element in the three-dimensional performance evaluation matrix is ​​calculated using a multi-objective Pareto optimization algorithm to obtain the comprehensive performance of each element. Based on the overall performance, the three-dimensional performance evaluation matrix is ​​selected to construct the composite processing space.

5. The fault handling method according to claim 1, characterized in that, The asynchronous reinforcement learning includes a first training thread and a second training thread. The step of training the processing model using asynchronous reinforcement learning based on the composite processing space and the training dataset to obtain the target processing model includes: In response to the detection that the first training thread is running at a first frequency, the processing result is determined based on the current parameters of the processing model and the greedy strategy; Based on the processing results, determine the reward value and the next state; Based on the current state, the processing result, the reward value, and the next state, generate a quadruple data set and store it in the buffer. In response to detecting that the second training thread is running at the second frequency, the parameters of the processing model are sampled from the buffer, updated by the temporal difference learning algorithm, and the updated parameters are synchronized to the first training thread.

6. The fault handling method according to claim 5, characterized in that, The process of determining the processing result based on the current parameters of the processing model and the greedy strategy includes: Obtain the base exploration value of the greedy strategy; Monitor the data utilization rate of the buffer; When the data utilization rate is less than a preset threshold, the basic exploration value is increased by a fixed amount to determine the initial value; When the data utilization rate is greater than or equal to the preset threshold, the basic exploration value is used as the initial value; The current exploration value corresponding to the greedy strategy is calculated based on the number of training steps, the preset exploration rate decay rate, and the initial value.

7. The fault handling method according to claim 5, characterized in that, The step of determining the reward value and the next state based on the processing result includes: The next state of the target device after monitoring the processing result; Based on the state vector, the processing result, and the next state of the target device, calculate the prediction accuracy, system stability, and / or repair efficiency. The reward value is calculated based on the prediction accuracy, the system stability, and / or the repair efficiency, combined with the corresponding weighting coefficients.

8. The fault handling method according to claim 5, characterized in that, The step of updating the parameters of the processing model using a time-difference learning algorithm includes: Sample a training sample from the buffer; Determine whether the next state of the training sample after the processing result is a termination state; In response to the next state being a termination state, a reward value is calculated as the maximum expected long-term return value of the training sample; In response to the next state not being a termination state, the target expected long-term return value of the training sample is calculated using the following formula: y = R + γ·maxa'Q(s',a';θtarget); where y is the target expected long-term return value; R is the reward value; s' is the next state; γ is a discount factor used to control the importance of future rewards; maxa'Q(s',a';θtarget) is the maximum expected long-term return value of the next state calculated by the target network; θtarget is the target network parameter, which is periodically updated to the online network parameter. The expected long-term online return value of the training samples is calculated using the parameters of the online network; Calculate the loss of the target expected long-term return and the online expected long-term return, and update the online network parameters using gradient descent based on the loss; Upon completion of the update, the parameters in the processing model are synchronized with the online network parameters.

9. The fault handling method according to claim 1, characterized in that, The step of inputting the state vector into the target processing model to obtain the processing result includes: The target processing model calculates the expected long-term return assessment value of each processing result in the composite processing space based on the state vector. Based on the expected long-term return assessment value, target repair operations and / or target fault prediction models are selected from the composite processing space and prediction results are obtained.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the fault handling method as described in any one of claims 1 to 9 when executing the computer program.

Images