Fault diagnosis automatic recovery system and method and electronic equipment

Abstract

The embodiment of the invention provides a fault diagnosis automatic recovery system and method and electronic equipment, the system separately deploys each functional component in a financial webpage system to a Kubernetes cluster for operation in a container mirror image mode, and a real-time monitoring module deploys a monitoring service unit in the Kubernetes; the deployment monitoring service unit collects performance indexes and log records of each container mirror image, the fault automatic discovery module determines whether a target container mirror image meeting a fault detection rule exists or not according to the collected performance indexes, if yes, the fault type of the target container mirror image is further determined, and according to the specific type, the target container mirror image is detected. And determining a corresponding set recovery record, and carrying out multiple types of fault recovery such as Pod restart, rolling update, fault transfer, backup and recovery on the recovery record. By selecting the embodiment of the invention, the fault diagnosis and automatic recovery of each component in the financial webpage system can be covered, so that the fault recovery efficiency is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a fault diagnosis and automatic recovery system, method and electronic equipment. Background Art

[0002] In the financial field, a financial Web system (or financial web page system) is an electronic platform or application for implementing financial services and transactions. It is usually developed by financial institutions to provide online banking, stock and bond trading, insurance services, etc. Among them, in financial web page systems, especially in complex and large-scale financial web page systems, fault diagnosis and recovery often rely on manual operations. Although there are some monitoring and log analysis tools that can assist operation and maintenance personnel in identifying faults, there are still many limitations and challenges.

[0003] Specifically, existing analysis tools usually can only provide limited information and may not cover all fault scenarios. In addition, these tools may have false positives or false negatives, which may interfere with fault diagnosis and recovery. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a fault diagnosis and automatic recovery system, method and electronic device to automatically diagnose and automatically recover faults existing in a financial web page system, thereby improving fault recovery efficiency.

[0005] In a first aspect, an embodiment of the present application provides a fault diagnosis and automatic recovery system, method and electronic device, wherein the system includes: a containerized deployment module, a real-time monitoring module, an automatic fault discovery module and a fault recovery module, wherein:

[0006] The containerized deployment module is used to create a container image for each functional component in the financial webpage system, and deploy each functional component into each container image, wherein each container image is deployed in a Kubernetes cluster, and each container image contains the operating environment that each functional component depends on;

[0007] The real-time monitoring module is used to deploy a monitoring service unit in the Kubernetes cluster, and the monitoring service unit is used to collect performance indicators and log records of each container image; and the performance indicators and log records collected by the monitoring service unit are displayed through a visual chart display unit;

[0008] The automatic fault discovery module is used to set a fault detection rule in the monitoring service unit, and trigger the fault recovery module if the performance indicator collected by the monitoring service unit meets the fault detection rule;

[0009] The fault recovery module is used to determine the fault type of the target container image that meets the fault detection rule, and recover the target container image according to the set fault recovery record, wherein the set fault recovery record includes: Pod restart record, rolling update record, failover record, backup and recovery record.

[0010] In some possible embodiments, the automatic fault discovery module is also used to train a preset machine learning model based on the performance indicators and the log records to obtain a target fault identification model, and use the target fault identification model to identify potential fault modes.

[0011] In some possible embodiments, the real-time monitoring module is further used to receive a monitoring script input by a user and execute the monitoring script, and collect performance indicators or log records specified in the monitoring script.

[0012] In some possible embodiments, the containerized deployment module is further used to generate a Kubernetes deployment list according to the deployment status of each container image, and the Kubernetes deployment list includes Pod templates and service resources of each functional component.

[0013] In some possible embodiments, the fault recovery module is specifically used to:

[0014] If the failure type of the target container image is application unresponsiveness or memory leakage, the target container image is restored using the Pod restart record;

[0015] If the failure type of the target container image is update code, each Pod instance in the target container image is replaced by using the rolling update record;

[0016] If the failure type of the target container image is component replacement, the failover record is used to transfer the Pod instance running in the target container image;

[0017] If the failure type of the target container image is backup and recovery, a new backup container image is determined for the target container image using the backup and recovery record, and the data in the target container image is backed up and restored to the new backup container image.

[0018] In some possible embodiments, the functional components include: a financial gateway, a redis cluster, an index warehouse, a message cluster, a database read-write separation component, and an application server cluster.

[0019] In a second aspect, an embodiment of the present application provides a fault diagnosis and automatic recovery method, wherein the method comprises:

[0020] Create a container image for each functional component in the financial webpage system, and deploy each functional component into each container image, wherein each container image is deployed in a Kubernetes cluster, and each container image contains the operating environment that each functional component depends on;

[0021] Deploy a monitoring service unit in the Kubernetes cluster, wherein the monitoring service unit is used to collect performance indicators and log records of each of the container images; and display the performance indicators and log records collected by the monitoring service unit through a visual chart display unit;

[0022] A fault detection rule is set in the monitoring service unit. If the performance indicators collected by the monitoring service unit meet the fault detection rule, the fault type of the target container image that meets the fault detection rule is determined, and the target container image is restored according to the set fault recovery record, wherein the set fault recovery record includes: Pod restart record, rolling update record, failover record, backup and recovery record.

[0023] In a third aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes:

[0024] Processor; and

[0025] Memory for storing programs,

[0026] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes the fault diagnosis and automatic recovery method described in the second aspect.

[0027] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the fault diagnosis and automatic recovery method described in the second aspect.

[0028] Beneficial effects of this application:

[0029] The embodiments of the present application provide a fault diagnosis and automatic recovery system, method and electronic device, wherein the system separately deploys each functional component in the financial webpage system in the form of container images to a Kubernetes cluster for operation, and a real-time monitoring module deploys a monitoring service unit in the Kubernetes, and the deployed monitoring service unit collects performance indicators and log records of each container image, and an automatic fault discovery module determines whether there is a target container image that meets the fault detection rules based on the collected performance indicators, and if so, further determines the fault type of the target container image, and then determines the corresponding set recovery record according to the specific type of the fault, and performs multiple types of fault recovery such as Pod restart, rolling update, failover, backup and recovery.

[0030] By selecting an embodiment of the present application, the financial web page system is split into various functional components, and then container images are built for different functional components. Then, indicators are collected for each container image respectively. In this way, fault diagnosis and automatic recovery of each component in the financial web page system can be covered, thereby effectively improving the fault recovery efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0032] Figure 1 A schematic diagram of a system architecture of a fault diagnosis and automatic recovery system provided in an embodiment of the present application is shown;

[0033] Figure 2 A schematic diagram of an engineering architecture of a fault diagnosis and automatic recovery system provided in an embodiment of the present application is shown;

[0034] Figure 3 A schematic diagram of a process flow of a fault diagnosis and automatic recovery method provided in an embodiment of the present application is shown;

[0035] Figure 4 A schematic diagram of a logical structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0037] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0038] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0040] As described in the background technology, when most existing financial web systems fail, maintenance personnel or developers usually manually analyze logs, performance indicators and other relevant data to determine the specific cause and location of the failure. This method is not only time-consuming, but also relies on manual experience, which can easily lead to inaccurate diagnostic results or omission of key information. In addition, there is a lack of an automatic fault recovery mechanism, that is, after the existing fault is determined, it is still necessary to manually restart, replace the faulty component, roll back the version, etc. This fault recovery process is not only inefficient, but also prone to misoperation or delayed response in emergency situations, thereby increasing financial risks.

[0041] Although there are some monitoring and diagnostic tools that can help identify and resolve failures in financial web systems, these tools usually only provide limited information and may not cover all failure scenarios. In addition, these tools may also have false positives or false negatives, which may interfere with the diagnosis and recovery of failures. In addition, since the diagnosis and recovery process of failures may take a long time, users may face service interruptions or performance degradation during use, which can easily affect the user experience.

[0042] In view of this, the present application provides a fault diagnosis and automatic recovery system, method and electronic device, wherein the fault diagnosis and automatic recovery system is deployed on a financial webpage system monitoring device, and the monitoring device can be any electronic device with financial webpage system performance monitoring, including but not limited to a personal mobile terminal, a computer or a server. As an implementation method, the fault diagnosis and automatic recovery system is a complete application system, which can be installed in the monitoring device through an installation package, and the application system and each functional component in the financial webpage system exchange data through a preset data interface, and the fault diagnosis and automatic recovery system can collect the operating status of each functional component through the preset data interface.

[0043] In some possible embodiments, the fault diagnosis and automatic recovery system can be as follows: Figure 1 As shown, it includes the following modules:

[0044] Containerized deployment module, real-time monitoring module, automatic fault discovery module, and fault recovery module. Among them:

[0045] The containerized deployment module is used to create a container image for each functional component in the financial webpage system, and deploy each functional component into each container image, wherein each container image is deployed in a Kubernetes cluster, and each container image contains the operating environment that each functional component depends on;

[0046] The real-time monitoring module is used to deploy a monitoring service unit in the Kubernetes cluster, and the monitoring service unit is used to collect performance indicators and log records of each container image; and the performance indicators and log records collected by the monitoring service unit are displayed through a visual chart display unit;

[0047] The automatic fault discovery module is used to set a fault detection rule in the monitoring service unit, and trigger the fault recovery module if the performance indicator collected by the monitoring service unit meets the fault detection rule;

[0048] The fault recovery module is used to determine the fault type of the target container image that meets the fault detection rule, and recover the target container image according to the set fault recovery record, wherein the set fault recovery record includes: Pod restart record, rolling update record, failover record, backup and recovery record.

[0049] In an embodiment of the present application, each functional component in the financial web page system is separately deployed in the form of a container image to a Kubernetes cluster for operation, and a real-time monitoring module deploys a monitoring service unit in the Kubernetes. The deployed monitoring service unit collects performance indicators and log records of each container image, and the automatic fault discovery module determines whether there is a target container image that meets the fault detection rules based on the collected performance indicators. If so, the fault type of the target container image is further determined. Then, according to the specific type of the fault, the corresponding set recovery record is determined to perform multiple types of fault recovery such as Pod restart, rolling update, failover, backup and recovery.

[0050] By selecting an embodiment of the present application, the financial web page system is split into various functional components, and then container images are built for different functional components. Then, indicators are collected for each container image respectively. In this way, fault diagnosis and automatic recovery of each component in the financial web page system can be covered, thereby effectively improving the fault recovery efficiency.

[0051] The following is an exemplary description of the fault diagnosis and automatic recovery system provided by the present application:

[0052] In the embodiment of the present application, the financial webpage system can be a webpage version of the system or an APP on a mobile terminal. The various functional components in the financial webpage system can be divided by developers or operation and maintenance personnel according to the various functions provided by the financial webpage system. For example, the software and hardware modules that implement the payment function can be divided into payment function components. As an implementation method, Figure 2 As shown, the various functional component modules of the financial webpage system may include: financial gateway, redis cluster, index warehouse, message cluster, database read-write separation component, and application server cluster.

[0053] Among them, in the embodiment of the present application, the financial gateway is responsible for receiving the data related to the normal operation of the financial web system, which is transmitted back by the web version of the financial web system and the mobile terminal version of the financial web system through the signal link and the cloud service. The redis cluster is a data cluster used to provide underlying data support for data processing tasks. The index information of each data stored in each data cluster is stored in the index warehouse. When the user reads and writes the data, the specific storage location of the target data can be determined according to the index information stored in the index warehouse. The message cluster is used to record and process various types of messages to the outside of the entire financial web system. The database read-write separation component is used to control the isolation between the read and write operations of the data in each data cluster. The application server cluster is a server cluster that is pre-allocated and configured for the financial web system and belongs to a hardware cluster. In the embodiment of the present application, each functional component is deployed in the Kubernetes cluster in the form of a Pod instance. As an implementation method, the containerized deployment module creates a container image in the Kubernetes cluster for each functional component in advance, and then deploys each functional component in the form of a Pod instance to each container image. Among them, the container image can be a Docker image, which is specifically a lightweight, executable independent software package. The container image corresponding to each functional component contains all the content required by each functional component, such as code, runtime, library, environment variables and configuration files. The Pod instance refers to the smallest unit of service provided by the Kubernetes cluster.

[0054] Wherein, the containerized deployment module can be modified based on the existing Docker image to modify the content required by each functional component into the existing Docker image. As another implementation, the Docker image corresponding to the functional component can be created by writing a Dockerfile text file.

[0055] By selecting an embodiment of the present application and using a containerized deployment module to create corresponding container images for each functional component in the financial web page system, it is possible to isolate the processes corresponding to each functional component from each other, that is, to achieve process-level isolation, which can prevent the failure of a functional component from affecting the operation of other functional components and enhance security.

[0056] On this basis, in some embodiments, the containerized deployment module is also used to:

[0057] A Kubernetes deployment list is generated according to the deployment status of each container image, and the Kubernetes deployment list includes Pod templates and service resources of each functional component.

[0058] In the embodiment of the present application, the Kubernetes deployment list is a summary file of the configuration information of each functional component allocated and deployed in the Kubernetes cluster. As an implementation method, the Kubernetes cluster abstracts various components deployed in the Kubernetes cluster into various resources. The containerized deployment module can obtain the various resources abstracted by the Kubernetes to summarize the information, and then write it in the file format YAML format, sort out the specific situation of the Pod instance deployed by each functional component, generate the Pod template of the functional component, and summarize and sort out the Services, Deployments, Ingress and other resources to obtain the Kubernetes deployment list.

[0059] In an embodiment of the present application, after each functional component is deployed to each container image, the Kubernetes deployment list can be executed through the Kubect1 command or the CI (Continuous Integration) tool / CD (Continuous Delivery) tool to deploy each functional component to the Kubernetes cluster.

[0060] After each functional component is deployed to the Kubernetes cluster in the form of a container image, the specific deployment location and specific data interface of each functional component are known. At this time, the real-time monitoring module can monitor the operating status of each functional component with the help of the specific deployment location and specific data interface of each functional component in each container image. Specifically, the real-time monitoring module deploys a monitoring service unit in the Kubernetes cluster, and the monitoring service unit collects performance indicators and log records of each container image.

[0061] As an implementation method, the monitoring service unit is any functional unit capable of monitoring the performance of Pod service instances in a Kubernetes cluster. Exemplarily, the monitoring service unit may be a Prometheus service. By configuring the container images corresponding to each functional component in the Prometheus service as monitoring data collection points (Exporter), each monitoring data collection point is used to collect performance indicators and log records of each container image. Among them, the performance indicators of each container image include: CPU usage, memory usage, network throughput, and disk I / O status. The log records mainly include: standard output, standard error output, and optional timestamp information. Among them, the optional timestamp information is added to the log record for a specific event by using the "timestamps" parameter, so that the problem can be located, tracked, and diagnosed according to the timestamp.

[0062] In an embodiment of the present application, the performance indicators and log records collected by the monitoring service unit can be displayed in a visual chart display unit in the form of visual charts. Among them, the visual chart display unit can specifically be a display unit provided by the fault diagnosis and automatic recovery system, which is used to intuitively display various data to the user. As an implementation method, the visual chart display unit can be implemented based on Grafana technology, and the data source collected by the Prometheus service is connected to the visual chart display unit implemented based on Grafana technology. The operating status and performance indicators of each functional component in the entire financial web page system can be displayed through the visual chart display unit.

[0063] In some embodiments, the real-time monitoring module is also used to receive the monitoring script passed in by the user and execute the monitoring script, and collect the performance indicators or log records specified in the monitoring script. Among them, the real-time monitoring module is connected to the user interaction module in the fault diagnosis and automatic recovery system, and the user interaction module provides an interface or interface for user interaction, and the user can submit a customized monitoring script to the interface or on the interface. Specifically, the monitoring script is a script file that the user customizes for the functional components to be monitored according to their own business needs, and can be any one of Shell scripts, Python scripts or JavaScript scripts. In this way, the performance indicators or log records specified by the user can be collected by running the customized monitoring script, and the collected information can be pushed to the Prometheus service or other monitoring system to monitor the customized faults, thereby achieving the effect of customized fault diagnosis and automatic recovery.

[0064] On the basis that the above-mentioned real-time monitoring module can obtain the specific performance indicator data and log records of the container image corresponding to each functional component, the automatic fault discovery module can set the fault detection rule in the monitoring service unit, and then judge whether the functional component has a running state fault according to the fault detection rule. Exemplarily, the automatic fault discovery module can set the fault detection rule in the Prometheus service: CPU usage>90%. At this time, when the Prometheus service monitors that the CPU usage of a container image is greater than 90%, it can automatically determine that the container image meets the fault detection rule. Among them, the fault detection rules will be different according to different functional components, and can be set according to historical data or historical experience. Exemplarily, the accuracy of the fault detection rules of the core business (such as CPU usage>80%) will be higher than the accuracy of the fault detection rules of the non-core business (such as CPU usage>90%).

[0065] Once the fault automatic discovery module finds that the collected performance indicators meet the fault detection rules, it is necessary to immediately restore the target container image with the fault, and then trigger the fault recovery module to restore the target container image. In the embodiment of the present application, each module in the fault diagnosis and automatic recovery system is a logical function module, which refers to a virtual module after packaging the processing logic executed by each module, and does not mean that each module is independent of each other. The processing logic of two or more modules can be packaged into a large virtual module according to some actual needs.

[0066] As an implementation method, in an embodiment of the present application, the automatic fault discovery module provides an artificial intelligence fault discovery function, which is specifically used to train a preset machine learning model based on performance indicators and log records to obtain a target fault identification model, and use the target fault identification model to identify potential fault modes.

[0067] Specifically, the automatic fault discovery module can use performance indicators and log records as training sample data to train the neural network model set in the automatic fault discovery module, such as a random forest model, a decision tree model, a deep learning network model, etc. Specifically, an unsupervised training method can be used, and the neural network model can learn the correlation between the performance indicator data, the data in the log record data and the container image fault, and then judge whether the performance indicator of the current container image has a fault based on the correlation. In this way, the performance indicators obtained by monitoring can be mined and analyzed at a deeper level using machine learning methods, so as to discover and identify possible fault modes.

[0068] As an implementation method, a fault detection rule engine can be integrated into the automatic fault discovery module. Users or experts in professional fields can write corresponding fault detection rule engines based on business knowledge and historical experience, and then use the fault rule engine to conduct comprehensive analysis and judgment on monitoring data and alarm information to determine the fault type and location. The fault information is collated to generate a detailed fault report, which includes: specific fault type, location, impact range, handling suggestions, etc.

[0069] In some embodiments, the fault recovery module can quickly recover the target functional component with a fault based on the fault report, especially based on the processing suggestions. As an implementation mode, the fault recovery module is specifically used to:

[0070] If the failure type of the target container image is application unresponsiveness or memory leakage, the target container image is restored using the Pod restart record;

[0071] If the failure type of the target container image is update code, each Pod instance in the target container image is replaced by using the rolling update record;

[0072] If the failure type of the target container image is component replacement, the failover record is used to transfer the Pod instance running in the target container image;

[0073] If the failure type of the target container image is backup and recovery, a new backup container image is determined for the target container image using the backup and recovery record, and the data in the target container image is backed up and restored to the new backup container image.

[0074] Among them, if the failure type of the target container image is application unresponsiveness or memory leakage, it means that the failure of the target container image is only a simple failure. In the embodiment of the present application, the Pod restart record can be: recovery by restarting the Pod service instance. Specifically, the target container image can be controlled to restart the Pod service instance by calling the restart Pod API interface provided by Kubernetes, or the target container image can be triggered to restart the Pod service instance by an automated script.

[0075] If the target container image has a fault type of code update, it means that the target container image needs to be updated with code or configuration, specifically a version update. At this time, the rolling update record can be: using the rolling update function provided by Kubernetes, replace the old Pod service instances one by one, so as to ensure the normal operation of the functional components corresponding to the target container image during the update process.

[0076] If the failure type of the target container image is component replacement, it usually means that the load of the current target container image will not be able to support data access requirements or data processing requirements. At this time, load allocation is required and components with higher performance need to be replaced, such as databases or middleware. Failover filing can be: through Kubernetes' StatefulSet or Deployment in conjunction with the Service's load balancing to perform failover. Specifically, when a Pod fails, Kubernetes will automatically remove the failed Pod from the Service's Endpoint list, and forward the original target container image request to the healthy Pod service instance on other container images to which the functional component corresponding to the target container image belongs.

[0077] Among them, if the failure type of the target container image is backup and recovery, it means that there is a systematic failure in the target container image and it needs to be restored as a whole. At this time, the container image that was backed up for the target container image in advance can be retrieved, and then the data request for the target container image can be migrated to the backed-up container image to achieve rapid failure recovery without the user noticing. The specific backup and recovery record can be: combining Kubernetes storage volumes (Volume) and backup tools (such as Velero, etc.) to achieve data backup and recovery.

[0078] By using the embodiment of the present application, each functional component in the financial webpage system can be containerized and then uniformly managed and scheduled through container orchestration tools (such as Kubernetes), which can make each functional component more independent, portable and scalable, and facilitate fault location and isolation. Combined with customized monitoring scripts and tools, the financial webpage system can be monitored in real time, and fault recovery can be quickly achieved according to the set fault recovery record, greatly improving the accuracy and efficiency of fault discovery and recovery.

[0079] In the second aspect, based on the fault diagnosis and automatic recovery system provided in the first aspect, the present application provides a fault diagnosis and automatic recovery method, which is applied to any electronic device with a fault diagnosis and automatic recovery function, including but not limited to a personal mobile terminal, a computer or a server, etc. Figure 3 As shown, the method comprises the following steps:

[0080] S31, creating a container image for each functional component in the financial webpage system, and deploying each functional component into each container image. Each container image is deployed in a Kubernetes cluster, and each container image contains the operating environment that each functional component depends on.

[0081] S32. Deploy a monitoring service unit in the Kubernetes cluster, where the monitoring service unit is used to collect performance indicators and log records of each of the container images; and display the performance indicators and log records collected by the monitoring service unit through a visual chart display unit.

[0082] S33, setting fault detection rules in the monitoring service unit, if the performance indicators collected by the monitoring service unit meet the fault detection rules, determining the fault type of the target container image that meets the fault detection rules, and recovering the target container image according to the set fault recovery record. Wherein, the set fault recovery record includes: Pod restart record, rolling update record, failover record, backup and recovery record.

[0083] The specific description of each step can refer to the relevant description in the first aspect above, which will not be repeated here. The embodiment of the present application is selected to split the financial webpage system into various functional components, then build container images for different functional components, and then collect indicators for each container image respectively. In this way, the fault diagnosis and automatic recovery of each component in the financial webpage system can be covered, thereby effectively improving the fault recovery efficiency.

[0084] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0085] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0086] In a third aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present application when executed by the at least one processor.

[0087] The exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0088] The exemplary embodiments of the present application further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute the method according to the embodiment of the present application.

[0089] refer to Figure 4 , the structural block diagram of the electronic device 400 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0090] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0091] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0092] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the aforementioned fault diagnosis automatic recovery method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the aforementioned fault diagnosis automatic recovery method by any other appropriate means (e.g., by means of firmware).

[0093] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0094] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0098] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A fault diagnosis and automatic recovery system, characterized in that: The system includes: a containerized deployment module, a real-time monitoring module, an automatic fault discovery module, and a fault recovery module, wherein: The containerized deployment module is used to create a container image for each functional component in the financial webpage system, and deploy each functional component into each container image, wherein each container image is deployed in a Kubernetes cluster, and each container image contains the operating environment that each functional component depends on; The real-time monitoring module is used to deploy a monitoring service unit in the Kubernetes cluster, and the monitoring service unit is used to collect performance indicators and log records of each container image; and the performance indicators and log records collected by the monitoring service unit are displayed through a visual chart display unit; The automatic fault discovery module is used to set a fault detection rule in the monitoring service unit, and trigger the fault recovery module if the performance indicator collected by the monitoring service unit meets the fault detection rule; The fault recovery module is used to determine the fault type of the target container image that meets the fault detection rule, and recover the target container image according to the set fault recovery record, wherein the set fault recovery record includes: Pod restart record, rolling update record, failover record, backup and recovery record.

2. The system according to claim 1, characterized in that The automatic fault discovery module is also used to train a preset machine learning model based on the performance indicators and the log records to obtain a target fault identification model, and use the target fault identification model to identify potential fault modes.

3. The system according to claim 1, characterized in that The real-time monitoring module is also used to receive a monitoring script input by a user and execute the monitoring script, and collect performance indicators or log records specified in the monitoring script.

4. The system according to claim 1, characterized in that The containerized deployment module is also used to generate a Kubernetes deployment list according to the deployment status of each container image, and the Kubernetes deployment list includes the Pod templates and service resources of each functional component.

5. The system according to claim 1, characterized in that The fault recovery module is specifically used for: If the failure type of the target container image is application unresponsiveness or memory leakage, the target container image is restored using the Pod restart record; If the failure type of the target container image is update code, each Pod instance in the target container image is replaced by using the rolling update record; If the failure type of the target container image is component replacement, the failover record is used to transfer the Pod instance running in the target container image; If the failure type of the target container image is backup and recovery, a new backup container image is determined for the target container image using the backup and recovery record, and the data in the target container image is backed up and restored to the new backup container image.

6. The system according to claim 1, characterized in that The functional components include: financial gateway, redis cluster, index warehouse, message cluster, database read-write separation component, and application server cluster.

7. A fault diagnosis and automatic recovery method, characterized in that: The method comprises: Create a container image for each functional component in the financial webpage system, and deploy each functional component into each container image, wherein each container image is deployed in a Kubernetes cluster, and each container image contains the operating environment that each functional component depends on; Deploy a monitoring service unit in the Kubernetes cluster, wherein the monitoring service unit is used to collect performance indicators and log records of each of the container images; and display the performance indicators and log records collected by the monitoring service unit through a visual chart display unit; A fault detection rule is set in the monitoring service unit. If the performance indicators collected by the monitoring service unit meet the fault detection rule, the fault type of the target container image that meets the fault detection rule is determined, and the target container image is restored according to the set fault recovery record, wherein the set fault recovery record includes: Pod restart record, rolling update record, failover record, backup and recovery record.

8. The method according to claim 7, characterized in that The method further comprises: Based on the performance indicators and the log records, a preset machine learning model is trained to obtain a target fault identification model, and the target fault identification model is used to identify potential fault modes.

9. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program comprises instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 7 to 8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 7-8.

Images