Server fault processing method, apparatus, device, and storage medium

Abstract

The application discloses a kind of server fault processing method, device, equipment and storage medium, the method includes: determining the configuration information of target domain name and target gateway address, and based on the configuration information of two-way communication tunnel issue;According to the abnormal event of the index data collected to abnormal detection of pre-set abnormal detection model, obtain server, the pre-set abnormal detection model is determined by unsupervised algorithm and is trained to obtain abnormal data;Based on two-way communication tunnel reception exception event, and according to the configuration information of exception event update described.The application issues the configuration information of target domain name and target gateway according to two-way communication tunnel, and detects exception event according to pre-set abnormal detection model, so as to be able to update configuration information for exception event, without manual intervention, improve the efficiency of server fault processing.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a server fault handling method, apparatus, device, and storage medium. Background Technology

[0002] Currently, when the containers providing business services within the server cluster are all malfunctioning and unable to provide services, the server health monitoring module will be unable to detect the service anomaly. Manual intervention is required to manually modify the server to remove the IP address of the faulty business center. However, manual intervention is time-consuming, labor-intensive, and prone to errors. Summary of the Invention

[0003] The main objective of this invention is to provide a server fault handling method, apparatus, device, and storage medium, aiming to solve the technical problem of how to improve the efficiency of server fault handling.

[0004] To achieve the above objectives, the present invention provides a server fault handling method, which includes the following steps:

[0005] Determine the configuration information of the target domain name and target gateway address, and send the configuration information based on the bidirectional communication tunnel;

[0006] Anomalies are detected in the collected indicator data according to a preset anomaly detection model to obtain abnormal events of the server. The preset anomaly detection model is obtained by determining abnormal data through an unsupervised algorithm and training on the abnormal data.

[0007] The abnormal event is received based on the bidirectional communication tunnel, and the configuration information is updated according to the abnormal event.

[0008] Optionally, the abnormal event includes: a single abnormal component;

[0009] Following the step of receiving the abnormal event based on the bidirectional communication tunnel, the method includes:

[0010] When the host corresponding to the single abnormal component is detected to be operating normally, the single abnormal component is deleted.

[0011] When a host corresponding to a single abnormal component is detected to be malfunctioning, the host that is malfunctioning is marked as unschedulable.

[0012] Place the single abnormal component on a normally functioning host and reschedule the single abnormal component.

[0013] Reset the host that is malfunctioning and mark it as schedulable.

[0014] Optionally, the abnormal event includes: multiple abnormal components;

[0015] The step of updating the configuration information based on the abnormal event includes:

[0016] Based on the multiple abnormal components reported by the monitoring agent to the service scheduler, delete the configuration information corresponding to the multiple abnormal components in the configuration information.

[0017] The multiple abnormal components are rescheduled, and the configuration information corresponding to the multiple abnormal components is restored in the configuration information when the multiple abnormal components resume running.

[0018] Optionally, the abnormal event includes: gateway malfunction;

[0019] The step of updating the configuration information based on the abnormal event includes:

[0020] When an abnormality is detected in the gateway, the configuration information corresponding to the abnormality in the configuration information is deleted.

[0021] When the abnormal gateway recovers, the configuration information corresponding to the abnormal gateway is restored in the configuration information.

[0022] Optionally, the step of performing anomaly detection on the collected indicator data according to a preset anomaly detection model to obtain abnormal events of the server includes:

[0023] The indicator data is tested to obtain quantitative and rate data;

[0024] Anomalies are detected in the quantity data and rate data according to a preset anomaly detection model to obtain abnormal events of the server.

[0025] Optionally, before the step of detecting the indicator data to obtain the quantity data and rate data, the method further includes:

[0026] Anomalies are identified by using an unsupervised algorithm and filtering out data from the indicator data, and these anomalies are then labeled.

[0027] A pre-defined anomaly detection model is obtained by offline training of labeled abnormal data using a supervised algorithm.

[0028] Optionally, after the steps of receiving the abnormal event based on the bidirectional communication tunnel and updating the configuration information according to the abnormal event, the method further includes:

[0029] Multiple time series models are established, and the multiple time series models are trained based on the training residuals of the previous round of the multiple time series models;

[0030] A target anomaly prediction model is obtained by weighted ensemble of multiple trained time series models.

[0031] The probability of an abnormal event is obtained by predicting the index data based on the target anomaly prediction model.

[0032] In addition, to achieve the above objectives, the present invention also proposes a server fault handling device, which includes: a data transmission module, a data detection module, and a data update module;

[0033] The data transmission module is used to determine the configuration information of the target domain name and the target gateway address, and to send the configuration information based on the bidirectional communication tunnel;

[0034] The data detection module is used to perform anomaly detection on the collected indicator data according to a preset anomaly detection model to obtain abnormal events of the server. The preset anomaly detection model is obtained by determining abnormal data through an unsupervised algorithm and training on the abnormal data.

[0035] The data update module is used to receive the abnormal event based on the bidirectional communication tunnel and update the configuration information according to the abnormal event.

[0036] Furthermore, to achieve the above objectives, the present invention also proposes a server fault handling device, which includes a memory, a processor, and a server fault handling program stored in the memory and capable of running on the processor. The server fault handling program is configured to implement the server fault handling method described above.

[0037] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a server fault handling program, which, when executed by a processor, implements the server fault handling method as described above.

[0038] This invention discloses a server fault handling method, apparatus, device, and storage medium. The method includes: determining the configuration information of a target domain name and a target gateway address, and sending the configuration information based on a bidirectional communication tunnel; performing anomaly detection on collected indicator data according to a preset anomaly detection model to obtain abnormal events of the server, wherein the preset anomaly detection model is obtained by determining abnormal data through an unsupervised algorithm and training on the abnormal data; receiving abnormal events based on the bidirectional communication tunnel, and updating the configuration information according to the abnormal events. This invention sends the configuration information of the target domain name and target gateway through a bidirectional communication tunnel and detects abnormal events according to the preset anomaly detection model, thereby enabling the configuration information to be updated in response to abnormal events without manual intervention, thus improving the efficiency of server fault handling. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the structure of a server fault handling device in the hardware operating environment involved in the embodiments of the present invention;

[0040] Figure 2 This is a flowchart illustrating the first embodiment of the server fault handling method of the present invention;

[0041] Figure 3 This is a flowchart illustrating the second embodiment of the server fault handling method of the present invention;

[0042] Figure 4 This is a flowchart illustrating the third embodiment of the server fault handling method of the present invention;

[0043] Figure 5 This is a flowchart illustrating the fourth embodiment of the server fault handling method of the present invention;

[0044] Figure 6 This is a schematic diagram of a server fault handling system architecture according to an embodiment of the server fault handling method of the present invention;

[0045] Figure 7 This is a target anomaly prediction model generation diagram according to an embodiment of the server fault handling method of the present invention;

[0046] Figure 8 This is an automated fault handling flowchart of an embodiment of the server fault handling method of the present invention;

[0047] Figure 9 This is a structural block diagram of the first embodiment of the server fault handling device of the present invention.

[0048] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0050] Reference Figure 1 , Figure 1 This is a schematic diagram of the server fault handling device structure in the hardware operating environment involved in the embodiments of the present invention.

[0051] like Figure 1As shown, the server fault handling device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen, and optionally, it may also include a standard wired interface or a wireless interface. In this invention, the wired interface of the user interface 1003 may be a USB interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0052] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the server fault handling equipment and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0053] like Figure 1 As shown, the memory 1005, which is identified as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a server fault handling program.

[0054] exist Figure 1 In the server fault handling device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the user equipment; the server fault handling device calls the server fault handling program stored in the memory 1005 through the processor 1001 and executes the server fault handling method provided in the embodiment of the present invention.

[0055] Based on the above hardware structure, an embodiment of the server fault handling method of the present invention is proposed.

[0056] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the server fault handling method of the present invention, which presents the first embodiment of the server fault handling method of the present invention.

[0057] Step S10: Determine the configuration information of the target domain name and the target gateway address, and send the configuration information based on the bidirectional communication tunnel.

[0058] It should be noted that the execution subject of this embodiment may be a computer software service device with data processing, network communication and program running functions, such as a server fault handling device, or other electronic devices that can achieve the same or similar functions. This embodiment does not limit this.

[0059] It should be understood that currently, application clients can access the business services provided by the business center's Kubernetes clusters A and B via domain names through the DNS server. When the ingress server fails, server health checks can detect and identify the fault and return the fault result IP address to the DNS server, thus achieving automatic traffic switching capability in the event of server-level failure.

[0060] Currently, the failover mechanism for the business center ingress server is as follows: 1. Client Access Method: When clients access the multi-center business cluster, they access the hard load balancing via DNS domain names, sending requests to the corresponding backend cluster. This avoids applications accessing the cluster directly through the IP addresses of multiple business centers, preventing the need to modify client access IPs in case of a center failure. 2. Load Balancing: This implements traffic load balancing across business centers, distributing access traffic across multiple business centers. Load balancing supports various load balancing strategies, including simple round-robin or intelligent traffic scheduling based on the current load of multiple business centers. It includes DNS service resolution and server health checks corresponding to the IP addresses of the domain name resolution entries. 3. DNS Server: This provides DNS domain name resolution capabilities and enables automatic service switching based on the server status reported by server health checks. The DNS server records the entry IP addresses corresponding to multiple business centers. When a client accesses and resolves the IP address of a service, the DNS server returns a different IP address to the client according to the configured strategy. 4. Server Health Check: Server health checks perform health checks on the servers and report the results to the DNS server. The system monitors the health status of servers within its jurisdiction in real time, using technologies such as ICMP, TCP, HTTP, and FTP. It also records the load on each server and provides load balancing strategy data to the DNS server based on access traffic. Fifth, failover: When a server health check detects an anomaly, the server health check module sends the failure result back to the DNS server, which then places the IP address of that service center in an abnormal state. Subsequent client accesses to resolve the IP address of the service center will no longer return that service center's IP address; all resolutions will only return the IP address of the service center in a normal state. When the faulty server recovers, the server health check module sends recovery information back to the DNS server, which then places the IP address of that service center in a normal state, allowing it to be resolved again and returned. This achieves automatic traffic switching capability in case of server failure.

[0061] It should be noted that the business center application service provides external service access capabilities through the Kubernetes ingress server. The instances registered to the ingress server to provide services are POD1 and POD2 within the cluster. When POD1 or POD2 fails, server health checks cannot connect to or view the abnormal status of the PODs, rendering them unable to provide services normally. However, the ingress server is running normally. In this situation, health checks cannot detect the service failure and provide feedback. The DNS server will not automatically remove the faulty IP address from the configuration information. This situation may cause service access through the DNS server to be resolved to an unavailable ingress node, triggering a business failure. The service failure must be manually detected and the DNS server configuration information manually changed; automatic failover is not possible. After manual investigation and resolution of the failure cause, and the PODs are restored to normal operation and service provision, the DNS server configuration information is manually restored.

[0062] As you can understand, a Pod is the smallest / simplest basic unit created or deployed in Kubernetes (the application deployment engine). A Pod represents a process running on the cluster. A Pod encapsulates an application container (or multiple containers), storage resources, a unique network IP address, and policy options for managing and controlling how the container runs. A Pod represents a unit of deployment: a single instance of an application in Kubernetes, which may consist of a single container or resources shared by multiple containers.

[0063] To overcome the aforementioned shortcomings, this embodiment aims to automatically update DNS server configuration information and remove faulty configuration information when application services within the central cluster experience failures, thereby automating service failure isolation. Furthermore, it automatically restores the removed faulty service configuration information once the application services regain their service capabilities. All operations are automated and require no manual intervention.

[0064] This embodiment, building upon existing load-balanced DNS resolution, provides a DNS resolution cluster, a service scheduler, and a monitoring agent to meet user needs. The monitoring agent, deployed within the cluster, monitors the status of the service POD instances provided by the business center ingress server and applications, and feeds back the status of each business center instance to the service scheduler in real time. Based on the feedback from the cluster agent, the service scheduler adjusts the configuration information of the corresponding DNS server, promptly updating the DNS configuration information of faulty nodes, thus achieving automatic failover and recovery of DNS records. Simultaneously, the DNS resolution cluster notifies the service scheduler to automatically and elastically scale the service POD instances based on access traffic and the configured scheduling policy.

[0065] Specifically, the user configures the business center ingress server IP and DNS domain name to the DNS resolution cluster. The service scheduler initiates a long connection to the monitoring agent service in the Kubernetes central cluster to be monitored, establishing a two-way authenticated TLS tunnel to collect event information generated by the Kubernetes central cluster. After the long connection is established, the domain name IP configuration information is sent out, and the monitoring agent generates the POD instance information corresponding to the DNS domain name to be monitored. The monitoring agent monitors the running status of the POD in the Kubernetes central cluster. When the running status of all PODs corresponding to a certain DNS domain name is abnormal, it sends a DNS domain name abnormal information event back to the service scheduler. After receiving the abnormal information event, the service scheduler updates the DNS domain name resolution entries that generated abnormalities in the DNS resolution cluster and removes the DNS resolution entries of the faulty nodes.

[0066] The monitoring agent's anomaly monitoring thread continues to monitor services where the PODs corresponding to DNS domain names are all in an abnormal running state. When the Kubernetes central cluster reschedules the PODs and their running state returns to normal and they can resume providing services, it sends a DNS domain name fault recovery information event back to the service scheduler. Upon receiving the fault recovery information event, the cluster service scheduler updates the fault recovery configuration information in the DNS resolution cluster.

[0067] When application clients access the business center service through a load balancer, they can be prevented from being redirected to an unavailable node, thus avoiding business failures. Once the service is restored and services are re-provided, external access capability can be automatically restored.

[0068] Based on the automatic elastic configuration strategy and monitoring agent feedback information, the service scheduler automatically triggers the scaling capabilities of service instances in the business center according to service call traffic.

[0069] For ease of understanding, please refer to Figure 6 To explain, Figure 6 This diagram illustrates the architecture of a server fault handling system. Based on the existing load balancer, a DNS resolution cluster is set up. The DNS resolution cluster contains DNS and ingress-IP configuration information. A two-way authenticated TLS tunnel is established between the service scheduler and the monitoring agent. Configuration information is distributed to the monitoring agent via this TLS tunnel. The monitoring agent monitors the POD's operational status and reports any abnormal events to the service scheduler. Upon receiving the abnormal event, the service scheduler updates the DNS resolution cluster's configuration information. The diagram also includes Kubernetes clusters A and B, each containing a gateway ingress and POD1 and POD2 for the business service center.

[0070] It should be noted that the configuration information for the target domain name and target gateway address is configured in the DNS resolution cluster. The DNS resolution cluster is implemented using CoreDNS (a flexibly scalable server) within the Kubernetes central cluster. CoreDNS is a pluggable DNS server written in Golang and is the default DNS server built into Kubernetes v1.13 and later. It is licensed under the Apache License Version 2 CoreDNS and supports cross-platform compatibility with various CPUs and operating systems.

[0071] Understandably, after the DNS resolution cluster is configured, it transmits the information to the service scheduler so that the service scheduler can save the configuration information to the scheduling list.

[0072] Understandably, establishing a two-way TLS-certified tunnel between the service scheduler and the monitoring agent enables secure communication and interoperability for secure transaction processing, including the use of cipher suites, certificate formats, signature algorithms, and session suspension. TLS also defines a method for establishing end-to-end secure TLS tunnels at the transport layer. It specifies a mechanism for providing a layered security layer for data before the application protocol and TCP / IP. TLS sits as a virtual layer between the application layer and the transport layer, accepting requests from the application layer and forwarding them to TCP for transmission to the server.

[0073] It should be noted that TLS tunneling can be implemented through the javax.net.ssl ​​package in the Java SDK, which provides relevant classes for establishing TLS connections. These include: secure sockets and secure server-side sockets, a non-blocking SSL / TLS data processing engine (SSLEngine), a socket creation factory to generate SSL sockets and server-side sockets, a socket context to store information used during creation and data engine processing, X.509 compliant cryptographic keys, and a security management interface.

[0074] Step S20: Perform anomaly detection on the collected indicator data according to the preset anomaly detection model to obtain abnormal events of the server. The preset anomaly detection model is obtained by determining abnormal data through an unsupervised algorithm and training on the abnormal data.

[0075] It should be noted that after receiving the configuration information from the service scheduler, the monitoring agent places it in the normal service instance queue. Simultaneously, a normal monitoring thread is started to monitor the normal service instance queue, primarily monitoring the ingress server and POD application. The monitoring content mainly includes monitoring the Node node and containers within the POD. Based on the abnormal events reported by the intelligent anomaly detection module, the corresponding DNS service is moved to the abnormal service instance queue and handed over to the abnormal service monitoring thread for monitoring, with feedback then returned to the service scheduler.

[0076] It should be noted that the anomaly detection module's event monitoring achieves complex scenario anomaly event monitoring by defining KPI indicators and constructing an intelligent anomaly detection process. It collects indicators across six dimensions: real-time computing resources, storage resources, disk I / O, processes, network, and Docker containers, establishes a preset anomaly detection model, and performs anomaly detection on the indicator data using this model. The preset anomaly detection model includes single POD anomaly detection, batch POD anomaly detection, and gateway node anomaly detection.

[0077] Furthermore, in order to improve the accuracy of abnormal event detection, step S20 of this embodiment may further include:

[0078] The indicator data is tested to obtain quantitative and rate data;

[0079] Anomalies are detected in the quantity data and rate data according to a preset anomaly detection model to obtain abnormal events of the server.

[0080] Understandably, online measurement and rate value testing is performed on indicator data. Measurement data testing is primarily used for KPI indicator testing, focusing on reporting business-specific numerical values ​​such as connection volume, transaction volume, and request count. Rate value testing involves detecting percentage data, such as success rate and return on investment. This measurement and rate value testing is conducted online.

[0081] Furthermore, to improve the accuracy of anomaly detection, the method further includes the following step before detecting the indicator data and obtaining the quantity and rate data:

[0082] Anomalies are identified by using an unsupervised algorithm and filtering out data from the indicator data, and these anomalies are then labeled.

[0083] A pre-defined anomaly detection model is obtained by offline training of labeled abnormal data using a supervised algorithm.

[0084] It should be noted that outliers in the indicator data can be identified using unsupervised algorithms and statistical discrimination, and then supervised algorithms can be used for offline training on these outliers. Unsupervised algorithms include the 3-Sigma algorithm and the Isolation Forest algorithm, while supervised algorithms can include the XGBoost algorithm. The principle of the 3-Sigma algorithm is that the data should follow a normal distribution; if a data value exceeds three times the standard deviation, it can be considered an outlier. The principle of the Isolation Forest algorithm is to assume that a random hyperplane is used to divide the data space. Each division generates two subspaces, and then another random hyperplane is used to divide each subspace, and so on, until each subspace contains only one data point (an outlier).

[0085] Understandably, when abnormal data is input into a positive and negative sample library, the labeling of positive and negative samples in the sample library can be presented through a visual interface, showing the labeling interface of abnormal samples (positive samples), with the peak part being the abnormal point.

[0086] Step S30: Receive the abnormal event based on the bidirectional communication tunnel, and update the configuration information according to the abnormal event.

[0087] It should be noted that the anomaly monitoring thread in the monitoring agent is responsible for monitoring the queue of abnormal service instances. After the intelligent anomaly detection module reports that the service indicator status is normal and the service can be provided normally, when an anomaly event occurs, the anomaly event is fed back to the service scheduler through a two-way communication tunnel. The server updates the configuration information of the DNS resolution cluster according to the anomaly event.

[0088] It is understandable that abnormal events can be caused by a single faulty component, multiple faulty components, or gateway malfunction. A single faulty component could mean that a single POD in the business center is not providing services, while multiple faulty components could mean that all PODs in the business center are not providing services.

[0089] Furthermore, in order to intervene in the occurrence of abnormal events in a timely manner, this embodiment may further include the following after step S30:

[0090] Multiple time series models are established, and the multiple time series models are trained based on the training residuals of the previous round of the multiple time series models;

[0091] A target anomaly prediction model is obtained by weighted ensemble of multiple trained time series models.

[0092] The probability of an abnormal event is obtained by predicting the index data based on the target anomaly prediction model.

[0093] Understandably, predictions are made about the current state of nodes based on historical anomalies, using intelligent predictions based on metrics such as computing resources, storage resources, disk I / O, processes, network, and Docker containers. Metric data prediction refers to using historical metric data to predict metric data at a future point in time.

[0094] It should be noted that the time series model uses logistic regression to build the trend model and Fourier series to build the cycle model, while the impact range of holidays is represented by indicator functions and configuration parameters. The time series model is expressed by the following formula:

[0095] y(t)=g(t)+s(t)+h(t)+∈ t ;

[0096] Where, ∈ t This represents the error term, also known as the residual term. g(t) is the trend term, representing the trend of the time series in a non-periodic manner; specifically...

[0097]

[0098] C(t) represents the maximum asymptotic value of the function, k represents the growth rate, m represents the midpoint of the curve, and δ represents the change in the growth rate k. The growth rate at time t is k + a. T δ, the variation of the time series s j This represents the point of change at timestamp S, where S represents the point of change, and a(t) = ((a1(t),...,a...). j (t),...,a s (t))) T , γ represents the boundary of the line segment;

[0099] For periodic terms, the model uses Fourier series to simulate the periodicity of the time series. P represents the time of the period; if the period is in years, then P equals 365.25; if the period is in weeks, then P equals 7. n and b n Let n represent the Fourier coefficients, where n = 0, 1, 2, ...

[0100] The "Holiday" item indicates whether a holiday applies on that day. K = (K i ,...,K L ) T K ~ Normal(0,v) 2Furthermore, this normal distribution is affected by the index v (default value is 10). The larger the v, the greater the impact of holidays on the model. K represents the influencing factors of holidays, and D... L It refers to the period of time before and after a holiday.

[0101] It should be noted that training the multiple time series models based on the residuals from the previous training round, and then weighting and ensembling these trained models, can employ the concept of ensemble learning. This involves using the time series models as weak learners, with each weak learner trained on the residuals (∈_t) from the previous round to obtain the final ensemble model. The entire training process continuously improves the prediction accuracy of the ensemble model by reducing bias. The number of weak learners can be set as needed. The final ensemble model is obtained by weighted summation of the weak learners obtained from each training round. The objective function used in the time series fusion index data prediction algorithm is as follows:

[0102]

[0103] in, and y i These represent the predicted value and the actual value, respectively. This represents the residual between the predicted value and the actual value. ω represents the L2 regularization term, where ω represents the weights of the time series model. The L2 regularization term smooths the model weights to prevent overfitting.

[0104] For ease of understanding, please refer to Figure 7 Please provide an explanation, such as Figure 7 A graph is generated for the target anomaly prediction model. The graph shows three time-series models (weak learners). The total sample refers to the set of all samples, where each sample represents a specific value of the indicator data at a given time (e.g., the amount of data). The entire time-series prediction process essentially predicts the specific values ​​of the indicator data, which is a regression prediction process. The time-series models are trained, and the residuals from the previous model are used to train the next model. Finally, an ensemble model, the target anomaly prediction model, is generated based on the weighted deweighting of the time-series models.

[0105] It should be noted that the following measures are taken to handle potentially risky nodes based on the prediction results: 1. When the prediction result indicates that the probability of POD1 experiencing an anomaly in the next time period exceeds the configured threshold, the following methods are used to handle potential anomalies: Expand the service instance in the business center to alleviate the pressure on the POD1 node. After the anomaly probability decreases, scale down the expanded service instance to reduce resource waste. 2. When the prediction result indicates that the probability of a sudden increase in service call volume in the next time period exceeds the configured threshold, the following methods are used to handle potential anomalies: Expand the gateway ingress instance node. The monitoring agent will report the ingress-IP of the new instance node to the service scheduler and add its IP to the configuration information. Simultaneously, expand the service instance in the business center to cope with the traffic surge. After the traffic returns to normal, notify the service scheduler to remove the newly expanded ingress-IP from the configuration information and scale down the gateway ingress instance node and the service instance in the business center. 3. When the prediction result indicates that the probability of an anomaly occurring on the host node where the POD1 instance is located exceeds the configured threshold, the following methods will be used to handle potential anomalies: expand the service instance of the business center, schedule the instance on other normal host nodes, and monitor high-probability abnormal host nodes in real time. Once a crash occurs, set it to be unschedulable, perform a POD eviction operation, reset the abnormal host node, set it to be schedulable again, and restore the host node's running status.

[0106] For ease of understanding, please refer to Figure 8 To explain, Figure 8 The flowchart illustrates the automated fault handling process. The DNS resolution cluster transmits configuration information to the service scheduler. The service scheduler establishes a TLS connection with the monitoring agent and distributes the configuration information to the monitoring agent. The monitoring agent monitors the status of the ingress server and the business application POD instance, and reports DNS domain name anomaly events to the service scheduler. The service scheduler updates the configuration information corresponding to the anomaly events in the DNS resolution cluster. The monitoring agent monitors the abnormal service instance and, upon fault recovery, transmits DNS domain name fault recovery information events to the service scheduler. The service scheduler then restores the configuration information corresponding to the fault recovery information events in the DNS resolution cluster.

[0107] This embodiment determines the configuration information of the target domain name and target gateway address, and sends the configuration information based on a bidirectional communication tunnel. It then performs anomaly detection on the collected indicator data using a preset anomaly detection model to obtain abnormal events on the server. This preset anomaly detection model is obtained by training on abnormal data using an unsupervised algorithm. Finally, it receives abnormal events through the bidirectional communication tunnel and updates the configuration information accordingly. This embodiment sends the configuration information of the target domain name and target gateway through the bidirectional communication tunnel and detects abnormal events based on the preset anomaly detection model. This allows for updating the configuration information in response to abnormal events without manual intervention, improving the efficiency of server fault handling.

[0108] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the server fault handling method of the present invention, based on the above. Figure 2 The first embodiment shown presents a second embodiment of the server fault handling method of the present invention.

[0109] In the second embodiment, step S30 includes:

[0110] Step S301: When it is detected that the host corresponding to the single abnormal component is operating normally, delete the single abnormal component;

[0111] Step S302: When an abnormal host operation is detected corresponding to the single abnormal component, the abnormal host is marked as unschedulable;

[0112] Step S303: Place the single abnormal component on a normally functioning host and reschedule the single abnormal component;

[0113] Step S304: Reset the host that is malfunctioning and mark it as schedulable.

[0114] In the specific implementation, when a POD in the business center is not providing services, and the abnormal node is the POD1 instance, the first processing method is adopted: automatically detect whether the host node where POD1 is located is running normally. If it is normal, delete POD1 and re-schedule and start it. If the host node is abnormal, set the host node to unschedulable, perform a POD eviction operation, and reschedule POD1 on a normally running host node. After resetting the abnormal host node, set it back to schedulable and restore the host node's running state.

[0115] In this embodiment, when the host corresponding to a single abnormal component is detected to be operating normally, the single abnormal component is deleted; when the host corresponding to the single abnormal component is detected to be operating abnormally, the abnormal host is marked as unschedulable; the single abnormal component is placed on a normally operating host and rescheduled; the abnormal host is reset and marked as schedulable. This embodiment determines whether the host is operating normally when a single abnormal component is not providing service; if the host is operating normally, the single abnormal component is deleted. If the host is operating abnormally, the single abnormal component is placed on a normally operating host and the abnormal host is reset. This eliminates the need for manual intervention and improves the accuracy of handling abnormal events.

[0116] Reference Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the server fault handling method of the present invention, based on the above. Figure 2 The first embodiment shown presents a third embodiment of the server fault handling method of the present invention.

[0117] In the third embodiment, step S30 further includes:

[0118] Step S301': Based on the multiple abnormal components reported by the monitoring agent to the service scheduler, delete the configuration information corresponding to the multiple abnormal components in the configuration information;

[0119] Step S302': Reschedule the plurality of abnormal components, and restore the configuration information corresponding to the plurality of abnormal components in the configuration information when the plurality of abnormal components resume running.

[0120] In the specific implementation, when all PODs in the business center are not providing services, and the abnormal node locations are POD1 and POD2, the second processing method is adopted: the monitoring agent program reports DNS domain name abnormality information events to the service scheduler. After receiving the abnormality information event, the service scheduler deletes the configuration information of the faulty node in the DNS resolution cluster. The POD is rescheduled, and after the POD recovers and resumes providing services, it sends a fault recovery information event back to the service scheduler, which then restores the configuration information of the faulty node.

[0121] This embodiment, based on the multiple abnormal components reported by the monitoring agent to the service scheduler, deletes the configuration information corresponding to the multiple abnormal components from the configuration information; reschedules the multiple abnormal components; and restores the configuration information corresponding to the multiple abnormal components in the configuration information when the multiple abnormal components are not providing services. This embodiment deletes the configuration information corresponding to the multiple abnormal components when they are not providing services, reschedules the multiple abnormal components, and restores the configuration information corresponding to the multiple abnormal components when they resume normal operation, thus eliminating the need for manual intervention and improving the accuracy of abnormal event handling.

[0122] Reference Figure 5 , Figure 5 This is a flowchart illustrating the third embodiment of the server fault handling method of the present invention, based on the above. Figure 2 The first embodiment shown presents a fourth embodiment of the server fault handling method of the present invention.

[0123] In the fourth embodiment, step S30 further includes:

[0124] Step S301”: When the gateway is detected to be malfunctioning, delete the configuration information corresponding to the gateway malfunction in the configuration information;

[0125] Step S301”: When the abnormal gateway recovers to normal, restore the configuration information corresponding to the abnormal gateway in the configuration information.

[0126] In the specific implementation, when the gateway ingress malfunctions, and the malfunctioning node is the gateway node, a third approach is used: When the anomaly detection module detects an anomaly in the gateway ingress instance or an anomaly in the host running the ingress, it reports a DNS domain name anomaly event to the service scheduler. The malfunctioning gateway is then removed from the configuration information. After restarting the ingress instance, the DNS resolution entries of the malfunctioning gateway are restored.

[0127] In this embodiment, when a gateway malfunction is detected, the configuration information corresponding to the malfunctioning gateway is deleted from the configuration information; when the malfunctioning gateway recovers, the configuration information corresponding to the malfunctioning gateway is restored in the configuration information. This embodiment removes the malfunctioning gateway from the configuration information when it is detected and restores its configuration information when it recovers, thus eliminating the need for manual intervention and improving the accuracy of handling abnormal events.

[0128] Furthermore, this embodiment of the invention also proposes a storage medium storing a server fault handling program, which, when executed by a processor, implements the server fault handling method described above.

[0129] In addition, refer to Figure 9 The present invention also proposes a server fault handling device, which includes: a data transmission module 10, a data detection module 20 and a data update module 30;

[0130] The data transmission module 10 is used to determine the configuration information of the target domain name and the target gateway address, and to send the configuration information based on the bidirectional communication tunnel;

[0131] The data detection module 20 is used to perform anomaly detection on the collected indicator data according to a preset anomaly detection model to obtain abnormal events of the server. The preset anomaly detection model is obtained by determining abnormal data through an unsupervised algorithm and training on the abnormal data.

[0132] The data update module 30 is used to receive the abnormal event based on the bidirectional communication tunnel and update the configuration information according to the abnormal event.

[0133] This embodiment determines the configuration information of the target domain name and target gateway address, and sends the configuration information based on a bidirectional communication tunnel. It then performs anomaly detection on the collected indicator data using a preset anomaly detection model to obtain abnormal events on the server. This preset anomaly detection model is obtained by training on abnormal data using an unsupervised algorithm. Finally, it receives abnormal events through the bidirectional communication tunnel and updates the configuration information accordingly. This embodiment sends the configuration information of the target domain name and target gateway through the bidirectional communication tunnel and detects abnormal events based on the preset anomaly detection model. This allows for updating the configuration information in response to abnormal events without manual intervention, improving the efficiency of server fault handling.

[0134] Based on the first embodiment of the server fault handling device of the present invention described above, a second embodiment of the server fault handling device of the present invention is proposed.

[0135] In this embodiment, the data update module 30 is used to delete the single abnormal component when it is detected that the host corresponding to the single abnormal component is running normally.

[0136] Furthermore, the data update module 30 is also used to mark the host with the abnormal operation as unschedulable when an abnormal operation of the host corresponding to the single abnormal component is detected.

[0137] Furthermore, the data update module 30 is also used to place the single abnormal component on a normally functioning host and reschedule the single abnormal component.

[0138] Furthermore, the data update module 30 is also used to reset the abnormal host and mark it as schedulable.

[0139] Furthermore, the data update module 30 is also used to delete the configuration information corresponding to the multiple abnormal components in the configuration information based on the multiple abnormal components reported by the monitoring agent to the service scheduler.

[0140] Furthermore, the data update module 30 is also used to reschedule the plurality of abnormal components, and restore the configuration information corresponding to the plurality of abnormal components in the configuration information when the plurality of abnormal components resume running.

[0141] Furthermore, the data update module 30 is also used to delete the configuration information corresponding to the gateway malfunction when the gateway malfunction is detected to be abnormal.

[0142] Furthermore, the data update module 30 is also used to restore the configuration information corresponding to the abnormal gateway in the configuration information when the abnormal gateway recovers to normal.

[0143] Furthermore, the data detection module 20 is also used to detect indicator data to obtain quantity data and rate data.

[0144] Furthermore, the data detection module 20 is also used to perform anomaly detection on the quantity data and the rate data according to a preset anomaly detection model to obtain abnormal events of the server.

[0145] Furthermore, the data detection module 20 is also used to filter out abnormal data from the indicator data based on an unsupervised algorithm, and to label the abnormal data.

[0146] Furthermore, the data detection module 20 is also used to perform offline training on the labeled abnormal data according to a supervised algorithm to obtain a preset anomaly detection model.

[0147] Furthermore, the data update module 30 is also used to establish multiple time series models and train the multiple time series models based on the previous training residuals of the multiple time series models.

[0148] Furthermore, the data update module 30 is also used to perform weighted integration of multiple trained time series models to obtain a target anomaly prediction model.

[0149] Furthermore, the data update module 30 is also used to predict the indicator data according to the target anomaly prediction model to obtain the probability of an anomaly event.

[0150] Other embodiments or specific implementations of the server fault handling device described in this invention can be found in the above-described method embodiments, and will not be repeated here.

[0151] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0152] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0154] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A server fault handling method, characterized in that, The server fault handling method includes the following steps: The configuration information of the target domain name and target gateway address is determined, and the configuration information is sent to the monitoring agent deployed in the running cluster based on the bidirectional communication tunnel, so that the monitoring agent can monitor the status of the service POD instances provided by the business center ingress server and the application, and provide real-time feedback on the status of each business center instance. Anomaly detection is performed on the collected indicator data according to a preset anomaly detection model to obtain abnormal events of the server. The preset anomaly detection model is obtained by identifying abnormal data through an unsupervised algorithm and training on the abnormal data. The indicator data includes quantitative data and rate data. The quantitative data is the detection of characteristic numerical indicators of the business, and the rate data is the detection of percentage data. The preset anomaly detection model includes single POD anomaly detection, batch POD anomaly detection, and gateway node anomaly detection. The abnormal event is received based on the bidirectional communication tunnel, and the configuration information is updated according to the abnormal event to remove the faulty configuration information; Following the step of receiving the abnormal event based on the bidirectional communication tunnel, the method includes: When the abnormal event includes a single abnormal component, the single abnormal component is deleted when it is detected that the host corresponding to the single abnormal component is operating normally. When a host corresponding to a single abnormal component is detected to be malfunctioning, the host that is malfunctioning is marked as unschedulable. Place the single abnormal component on a normally functioning host and reschedule the single abnormal component. Reset the host that is malfunctioning and mark it as schedulable; The step of updating the configuration information based on the abnormal event includes: When the abnormal event includes multiple abnormal components, the configuration information corresponding to the multiple abnormal components is deleted from the configuration information according to the multiple abnormal components reported by the monitoring agent to the service scheduler. The multiple abnormal components are rescheduled, and the configuration information corresponding to the multiple abnormal components is restored in the configuration information when the multiple abnormal components resume running.

2. The server fault handling method as described in claim 1, characterized in that, The abnormal events include: gateway malfunction; The step of updating the configuration information based on the abnormal event includes: When an abnormality is detected in the gateway, the configuration information corresponding to the abnormality in the configuration information is deleted. When the abnormal gateway recovers, the configuration information corresponding to the abnormal gateway is restored in the configuration information.

3. The server fault handling method as described in claim 1, characterized in that, The step of performing anomaly detection on the collected indicator data according to a preset anomaly detection model to obtain abnormal events of the server includes: The indicator data is tested to obtain quantitative and rate data; Anomalies are detected in the quantity data and rate data according to a preset anomaly detection model to obtain abnormal events of the server.

4. The server fault handling method as described in claim 3, characterized in that, Before the step of detecting the indicator data to obtain the quantity data and rate data, the method further includes: Anomalies are identified from the indicator data using an unsupervised algorithm, and these anomalies are then labeled. A pre-defined anomaly detection model is obtained by offline training of labeled abnormal data using a supervised algorithm.

5. The server fault handling method according to any one of claims 1 to 4, characterized in that, After the steps of receiving the abnormal event based on the bidirectional communication tunnel and updating the configuration information according to the abnormal event, the method further includes: Multiple time series models are established, and the multiple time series models are trained based on the training residuals of the previous round of training of the multiple time series models; A target anomaly prediction model is obtained by weighted ensemble of multiple trained time series models. The probability of an abnormal event is obtained by predicting the index data based on the target anomaly prediction model.

6. A server fault handling device, characterized in that, The server fault handling device includes: a data transmission module, a data detection module, and a data update module; The data transmission module is used to determine the configuration information of the target domain name and the target gateway address, and to send the configuration information to the monitoring agent deployed in the running cluster based on the bidirectional communication tunnel, so that the monitoring agent can monitor the status of the service POD instances provided by the business center ingress server and the application, and provide real-time feedback on the status of each business center instance. The data detection module is used to perform anomaly detection on the collected indicator data according to a preset anomaly detection model to obtain abnormal events of the server. The preset anomaly detection model is obtained by determining abnormal data through an unsupervised algorithm and training on the abnormal data. The indicator data includes quantitative data and rate data. The quantitative data is the detection of characteristic numerical indicators of the business, and the rate data is the detection of percentage data. The preset anomaly detection model includes single POD anomaly detection, batch POD anomaly detection, and gateway node anomaly detection. The data update module is used to receive the abnormal event based on the bidirectional communication tunnel, and update the configuration information according to the abnormal event to remove the faulty configuration information; The data update module is further configured to, when the abnormal event includes a single abnormal component, delete the single abnormal component when the host corresponding to the single abnormal component is detected to be running normally; mark the host running abnormally as unschedulable when the host corresponding to the single abnormal component is detected to be running abnormally; place the single abnormal component on the host running normally and reschedule the single abnormal component; reset the host running abnormally and mark it as schedulable. The data update module is further configured to, when the abnormal event includes multiple abnormal components, delete the configuration information corresponding to the multiple abnormal components in the configuration information according to the multiple abnormal components fed back to the service scheduler by the monitoring agent; reschedule the multiple abnormal components; and restore the configuration information corresponding to the multiple abnormal components in the configuration information when the multiple abnormal components resume running.

7. A server fault handling device, characterized in that, The server fault handling device includes: a memory, a processor, and a server fault handling program stored in the memory and executable on the processor. When the server fault handling program is executed by the processor, it implements the steps of the server fault handling method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a server fault handling program, which, when executed by a processor, implements the steps of the server fault handling method as described in any one of claims 1 to 5.

Images