Fault processing method, device, equipment, medium and program product

Abstract

The invention provides a fault processing method which can be applied to the distributed technical field and the financial science and technology field. The fault processing method comprises the steps that fault judgment is carried out on N host machines in a cluster according to a fault judgment rule, a fault host machine is determined, and N is an integer larger than 1; the identification information of the fault host machine is adjusted from first identification information to second identification information, the first identification information represents the priority of the host machine, and the second identification information represents the host machine which cannot be called; and adjusting the identification information of the host machines in the cluster based on the first identification information of each host machine in the cluster and a host machine number threshold value corresponding to the priority. The invention further provides a fault processing device, equipment, a medium and a program product.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed technology and the technical field of financial technology, and more particularly to a fault processing method, device, equipment, medium and program product. BACKGROUND

[0002] With the development of cloud platform technology, the cloud platform cluster of Internet financial enterprises is increasingly large. When a large number of faults occur in host computers, the related technology usually needs to manually set the fault host computer to be prohibited from scheduling, then estimate the resource occupation of each label in the cluster, and then adjust the identification information of the healthy host computer, so as to ensure that the system can normally start the related nodes. The whole emergency processing procedure is complex, needs manual intervention, is time-consuming and error-prone. SUMMARY

[0003] In view of the above problems, the present application provides a fault processing method, device, equipment, medium and program product.

[0004] According to a first aspect of the present application, a fault processing method is provided, comprising: performing fault judgment on N host computers in a cluster according to a fault judgment rule to determine a fault host computer, N being an integer greater than 1; adjusting the identification information of the fault host computer from first identification information to second identification information, the first identification information representing the priority of the host computer, and the second identification information representing an uncallable host computer; and adjusting the identification information of the host computers in the cluster based on the first identification information of each host computer in the cluster and the host computer quantity threshold corresponding to the priority.

[0005] According to an embodiment of the present application, the first identification information includes a first priority and a second priority, the first priority representing that the host computer is a core host computer, and the second priority representing that the host computer is a non-core host computer. Adjusting the identification information of the host computers in the cluster based on the first identification information of each host computer in the cluster and the host computer quantity threshold corresponding to the priority includes at least one of the following: in the case where the number of host computers corresponding to the first priority does not reach the corresponding quantity threshold, adjusting the identification information of a standby host computer from third identification information to the first priority, the third identification information representing that the host computer is a standby host computer; in the case where the number of host computers corresponding to the second priority does not reach the corresponding quantity threshold, and the number of host computers corresponding to the first priority reaches the corresponding quantity threshold, adjusting the identification information of the standby host computer from the third identification information to the second priority; in the case where the number of host computers corresponding to the first priority does not reach the corresponding quantity threshold and the number of standby host computers is zero: determining at least one target host computer from the host computers whose identification information is the second priority; adjusting the identification information of the target host computer from the second priority to the first priority.

[0006] According to an embodiment of the present application, the method further comprises: in a case where the failed host is a core host, performing software restart on the failed host, the software restart representing restarting services on the failed host; performing multi-dimensional verification on the failed host after the software restart, the multi-dimensional verification comprising at least one of hardware diagnosis, container startup test, and network connectivity check; in a case where the multi-dimensional verification on the failed host after the software restart passes, adjusting the identification information of the failed host from the second identification information to third identification information; in a case where the failed host is a non-core host, deleting a node corresponding to the failed host, the node representing a logical mapping of the failed host in the cluster control system.

[0007] According to an embodiment of the present application, the method further comprises: in a case where the multi-dimensional verification on the failed host after the software restart fails, performing hardware restart on the failed host, the hardware restart representing completely powering off and then powering on the hardware of the failed host; performing multi-dimensional verification on the failed host after the hardware restart; in a case where the multi-dimensional verification on the failed host after the hardware restart passes, adjusting the identification information of the failed host from the second identification information to third identification information; in a case where the multi-dimensional verification on the failed host after the hardware restart fails, reporting information of the failed host to the system.

[0008] According to an embodiment of the present application, the method further comprises, before the software restart on the failed host: issuing a shutdown instruction to the failed host, the shutdown instruction representing stopping running services on the failed host; in a case where the failed host does not respond to the shutdown instruction, issuing a hardware restart instruction to the failed host and performing multi-dimensional verification on the failed host after the hardware restart; in a case where the failed host does not respond to the hardware restart instruction, deleting a node corresponding to the failed host.

[0009] According to an embodiment of the present application, the method further comprises: in a case where the failed host is a core host, performing software restart on the failed host, the software restart representing restarting services on the failed host; performing multi-dimensional verification on the failed host after the software restart, the multi-dimensional verification comprising at least one of hardware diagnosis, container startup test, and network connectivity check; in a case where the multi-dimensional verification on the failed host after the software restart passes, adjusting the identification information of the failed host from the second identification information to third identification information; in a case where the failed host is a non-core host, deleting a node corresponding to the failed host, the node representing a logical mapping of the failed host in the cluster control system.

[0010] The second aspect of the present application provides a fault processing apparatus, comprising: a determination module configured to determine N host computers in a cluster according to a fault determination rule, determine a fault host computer, N being an integer greater than 1; a first adjustment module configured to adjust identification information of the fault host computer from first identification information to second identification information, the first identification information representing a priority of the host computer, and the second identification information representing an un-callable host computer; and a second adjustment module configured to adjust identification information of the host computers in the cluster based on the first identification information of each host computer in the cluster and a host computer quantity threshold corresponding to the priority.

[0011] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory configured to store one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0012] The fourth aspect of the present application further provides a computer-readable storage medium having stored thereon a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the method.

[0013] The fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the method.

[0014] According to the embodiments of the present application, by adjusting the identification information of the fault host computer to be unschedulable, the service processing failure caused by the service continuing to run on the fault host computer is avoided. The host computer quantity threshold corresponding to the priority represents the number of host computers required to ensure normal service, and therefore, after adjusting the identification information of the fault host computer, adjusting the identification information of each host computer in the cluster based on the host computer quantity threshold corresponding to the priority can ensure the stability of the service resource pool. BRIEF DESCRIPTION OF DRAWINGS

[0015] The above content of the present application and other purposes, features and advantages will be more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0016] Figure 1 An application scenario diagram of the fault processing method, apparatus, device, medium and program product according to the embodiments of the present application is schematically shown;

[0017] Figure 2 A flowchart of the fault processing method according to the embodiments of the present application is schematically shown;

[0018] Figure 3 A host computer identification information Venn diagram according to the embodiments of the present application is schematically shown;

[0019] Figure 4 a flowchart of a fault handling method according to another embodiment of the present application is shown schematically;

[0020] Figure 5 a block diagram of a fault handling apparatus according to an embodiment of the present application is shown schematically; and

[0021] Figure 6 a block diagram of an electronic device adapted to implement a fault handling method according to an embodiment of the present application is shown schematically. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present application, and is not intended to limit the scope of the present application. In the following detailed description of the embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "including" "comprising" and the like are meant to be inclusive, but not limiting to the components, steps, operations and / or features that were listed. It is noted that multiple recitations of a feature or component can or can not be literally the same.

[0024] All terms used herein including technical and scientific terms have the same meanings as those generally understood by those skilled in the art unless otherwise defined herein. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification, and should not be interpreted in an idealized or overly formal way.

[0025] In the case where expressions such as "at least one of A, B, and C, etc." are used, it is generally to be interpreted as including at least one of A, B, and C, etc. (e.g., "a system having at least one of A, B, and C" should include but not be limited to a system that has A alone, a system that has B alone, a system that has C alone, a system that has both A and B, a system that has both A and C, a system that has both B and C, and / or a system that has A, B, and C, etc.).

[0026] The terms involved herein are explained and described as follows.

[0027] A cloud platform cluster refers to a resource pool formed by a plurality of physical or virtual servers working in cooperation in a cloud computing environment.

[0028] The identification information of the host computer is important metadata for resource classification and scheduling in cloud computing and container orchestration systems, and the core function thereof is to identify physical or virtual host attributes through a key-value pair to achieve fine-grained resource management.

[0029] It should be noted that the fault method, device, equipment, medium and program product determined by the present application can be used in the fields of distributed technology and financial technology, and can also be used in various fields other than the fields of distributed technology and financial technology, such as the field of cloud computing. The application fields of the fault processing method, device, equipment, medium and program product provided by the embodiments of the present application are not limited.

[0030] In the technical solutions of the present application, the user information (including but not limited to user personal information, user image information, user device information such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0031] In the scenario of using personal information for automated decision-making, the fault method, device and system provided by the embodiments of the present application all provide corresponding operation portal for the user to choose to agree or refuse the automated decision-making result; if the user chooses to refuse, the expert decision-making process is entered. The expression "automated decision-making" here refers to the activity of automatically analyzing, evaluating the behavior habits, interests and hobbies or economic, health, credit status of individuals, etc. by computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by personnel who are engaged in a certain field of work, have special experience, knowledge and skills, and reach a certain professional level.

[0032] Figure 1 The application scenario diagram of the fault processing method, device, equipment, medium and program product according to the embodiments of the present application is schematically shown.

[0033] As Figure 1 shown, the application scenario 100 according to the embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used as a medium to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0034] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, etc.

[0036] The server 105 can be a server providing various services, such as a background management server providing support for websites browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as examples). The background management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data, etc. obtained or generated according to user requests) to the terminal device.

[0037] It should be noted that the fault processing method provided by the embodiments of the present application can generally be executed by the server 105. Correspondingly, the fault processing apparatus provided by the embodiments of the present application can generally be arranged in the server 105. The fault processing method provided by the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the fault processing apparatus provided by the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks, and servers in

[0039] The fault processing method according to the embodiments of the present application will be described in detail below based on the scenario described in Figure 1 Figures 2-4

[0040] Figure 2 The flowchart of the fault processing method according to the embodiments of the present application is schematically shown.

[0041] As​​Figure 2 As shown, the fault processing method of this embodiment includes operation S210 to operation S230, which can be performed by the server 105.

[0042] In operation S210, the N host computers in the cluster are judged for faults according to a fault judgment rule, and a fault host computer is determined, where N is an integer greater than 1.

[0043] For example, the fault judgment rule can be used to judge whether a host computer is a fault host computer, or to judge the specific fault type of a host computer. The cluster can refer to a cloud platform cluster.

[0044] Specifically, whether a host computer is a fault host computer can be judged by a heartbeat signal of the host computer or whether the host computer responds to a request, or the specific fault type of a host computer can be judged by analyzing the hardware state or network state of the host computer.

[0045] In operation S220, the identification information of the fault host computer is adjusted from first identification information to second identification information.

[0046] In the embodiments of the present application, the first identification information represents the priority of the host computer, and the second identification information represents an uncallable host computer.

[0047] The configurations of the host computers in the cluster can be different, and the priority of a host computer can be related to the configuration of the host computer. For example, a host computer with a relatively good configuration has a high priority, and a host computer with a relatively general configuration has a low priority. Core or important services such as real-time data calculation can run on host computers with high priority to meet the configuration requirements of core services. Non-core services such as offline data calculation do not have high requirements for stability and real-time performance, and such services can run on host computers with low priority. Therefore, the priority of a host computer is related to the configuration of the host computer, and it can also be understood that the priority of a host computer is related to the services or processes handled by the host computer.

[0048] In operation S230, the identification information of the host computers in the cluster is adjusted based on the first identification information of each host computer in the cluster and the host computer quantity threshold corresponding to the priority.

[0049] In the embodiments of the present application, the host computer quantity threshold corresponding to each priority is used to represent the number of host computers required at least for the priority to ensure that services can run smoothly. After the identification information of the fault host computer is adjusted, there can be a case where the number of host computers corresponding to the priority does not meet the host computer quantity threshold, in which case the identification information of each host computer in the cluster needs to be adjusted to support core services.

[0050] For example, after the identification information of the failed host is adjusted, there are 6 hosts with high priority, the quantity threshold corresponding to the high priority is 5, there are 8 hosts with low priority, and the quantity threshold corresponding to the low priority is 4. In this case, the identification information of one host with low priority can be adjusted to high priority to support core services.

[0051] It can be understood that, by adjusting the identification information of the failed host to be non-schedulable, the service processing failure caused by the service continuing to run on the failed host is avoided. The quantity threshold of the host corresponding to the priority indicates the number of hosts required to ensure normal service, and therefore, after the identification information of the failed host is adjusted, adjusting the identification information of each host in the cluster based on the quantity threshold of the host corresponding to the priority can ensure the stability of the service resource pool.

[0052] According to an embodiment of the present application, the first identification information includes a first priority and a second priority, the first priority indicates that the host is a core host, and the second priority indicates that the host is a non-core host. Based on the first identification information of each host in the cluster and the quantity threshold of the host corresponding to the priority, the adjustment of the identification information of the host in the cluster includes at least one of the following: in the case where the number of hosts corresponding to the first priority does not reach the corresponding quantity threshold, the identification information of the standby host is adjusted from the third identification information to the first priority, and the third identification information indicates that the host is a standby host; in the case where the number of hosts corresponding to the second priority does not reach the corresponding quantity threshold, and the number of hosts corresponding to the first priority reaches the corresponding quantity threshold, the identification information of the standby host is adjusted from the third identification information to the second priority; in the case where the number of hosts corresponding to the first priority does not reach the corresponding quantity threshold and the number of standby hosts is zero: at least one target host is determined from the hosts with the second priority; and the identification information of the target host is adjusted from the second priority to the first priority.

[0053] In the embodiments of the present application, the first identification information can be used to determine whether the host is a core host. If the identification information of the host is the first priority, it indicates that the host is a core host. If the identification information of the host is the second priority, it indicates that the host is a non-core host. The first priority and the second priority can each include multiple priorities. For example, the first priority includes priority A and priority B, both of which correspond to core hosts, and the second priority includes priority C, priority D, and priority E. Priority C, priority D, and priority E all correspond to non-core hosts. The host quantity threshold can be set for priority A, priority B, priority C, priority D, and priority E respectively.

[0054] The host quantity threshold corresponding to the priority represents the quantity of host required for guaranteeing service of each priority.

[0055] If the host quantity corresponding to the first priority does not reach the corresponding quantity threshold, it indicates that after the failure, the core host quantity is insufficient to support the core service, and the standby host can be adjusted to serve the core service. If the host quantity corresponding to the second priority does not reach the corresponding quantity threshold, and the host quantity corresponding to the first priority reaches the corresponding quantity threshold, it indicates that the core host quantity can support the core service, but the quantity of non-core host is insufficient to support the non-core service, and the standby host can be adjusted to serve the non-core service.

[0056] If the host quantity corresponding to the first priority does not reach the corresponding quantity threshold and the standby host quantity is zero, it indicates that the core host quantity is insufficient to support the core service, but there is no standby host to call, and at least one target host can be determined from the host with the second priority; the identification information of the target host is adjusted from the second priority to the first priority. The quantity of the target host can be determined according to the host quantity requirement of the first priority, that is, according to the difference between the host quantity corresponding to the first priority and the quantity threshold.

[0057] Figure 3 The host identification information Wean diagram according to the embodiment of the application is schematically shown as Figure 3 As shown in the figure, the identification information of the host includes first identification information, second identification information and third identification information, wherein the first identification information includes the first priority and the second priority.

[0058] For example, as shown in Table 1, it is assumed that the host quantity threshold corresponding to the first priority is 14, the host quantity threshold corresponding to the second priority is 8, and 6 faulty hosts are monitored, and the identification information of the host is the first priority. Then according to Table 1, after the identification information of the 6 faulty hosts is adjusted to the second identification information, the host quantity corresponding to the first priority is only 10, which does not meet the quantity threshold, and the identification information of the standby host is adjusted to the first priority, which still does not meet the quantity threshold, therefore, the identification information of the host of the second priority needs to be adjusted, and 2 hosts of the second priority can be selected as target hosts, and the identification information of the 2 hosts is adjusted from the second priority to the first priority.

[0059] Table 1: Host quantity table

[0060]

[0061] It can be understood that the embodiment of the application proposes a reasonable identification information adjustment method. When the number of core host computers is insufficient, the standby host computer is preferentially called to serve the core service. Compared with calling the non-core host computer, preferentially calling the standby host computer can ensure that the host computer is quickly put into operation, reduce the core resource vacancy time, and also can avoid the influence on the non-core service. Preferentially considering the number of core host computers can ensure the stability of the core service.

[0062] According to the embodiment of the application, the method further comprises: in the case that the fault host computer is a core host computer, performing software restart on the fault host computer, the software restart representing restarting the service on the fault host computer; performing multi-dimensional verification on the fault host computer after the software restart, the multi-dimensional verification comprising at least one of hardware diagnosis, container startup test and network connectivity check; in the case that the multi-dimensional verification on the fault host computer after the software restart is passed, adjusting the identification information of the fault host computer from the second identification information to the third identification information; in the case that the fault host computer is a non-core host computer, deleting a node corresponding to the fault host computer, the node representing the logical mapping of the fault host computer in the cluster control system.

[0063] In the embodiment of the application, the hardware diagnosis can refer to checking the port or disk state of the fault host computer. For example, when the port link state is normal and the result of the disk health state check is no hardware fault alarm, it can be considered that the hardware diagnosis is passed. The container startup test can refer to judging whether the container deployed on the host computer can be normally started, etc. For example, when the container can be normally started, the dependent resource connection is normal and the interface response is stable, it can be considered that the container startup test is passed. The network connectivity check can refer to checking the network link between the host computer and the cluster control node. For example, in the case that the connectivity is normal and the control plane protocol communication is normal, it can be considered that the network connectivity check is passed.

[0064] The multi-dimensional verification on the fault host computer after the software restart is passed, which indicates that the fault host computer has recovered to a normal host computer after the software restart, and the host computer can be adjusted to a standby host computer.

[0065] In addition, for the non-core host computer, since the non-core host computer usually has a certain replaceability and a relatively general configuration, considering the resources required for repairing it, the non-core host computer can be directly evicted to speed up the fault processing process.

[0066] It can be understood that the multi-dimensional verification used in the embodiment of the application can avoid the zombie node, i.e. the host computer that is logically online but actually unavailable. For the non-core host computer, directly evicting it can quickly release the resources occupied by it and speed up the fault processing process.

[0067] According to an embodiment of the present application, the method further comprises: in a case where the multi-dimensional verification of the software-restarted fault host fails, performing hardware restart on the fault host, the hardware restart representing powering off and then powering on the hardware of the fault host; performing multi-dimensional verification on the hardware-restarted fault host; in a case where the multi-dimensional verification of the hardware-restarted fault host passes, adjusting the identification information of the fault host from the second identification information to third identification information; and in a case where the multi-dimensional verification of the hardware-restarted fault host fails, reporting the information of the fault host to the system.

[0068] In the embodiment of the present application, if the multi-dimensional verification of the software-restarted fault host fails, it indicates that the fault of the fault host is likely not a software-level fault but a hardware-level fault, and the hardware restart needs to be performed on the fault host, that is, the fault host needs to be powered off and then powered on.

[0069] For example, when the hardware appears an exception, such as memory address conflict or motherboard firmware freezing, in this case, the software restart cannot fix the fault, and the hardware restart can be tried, and the multi-dimensional verification is performed on the restarted fault host.

[0070] If the verification passes, it can be considered that the fault host has been restored to a normal host after the hardware restart, and the fault host can be adjusted to a standby host. If the fault host still cannot be restored to a normal host after the hardware restart, the fault host can be reported to the system for manual intervention.

[0071] It can be understood that if the fault is caused by software level, such as service crash, only the service in the software layer needs to be restarted, and the hardware restart is not needed. The hardware restart is tried after the software restart fails, and the multi-dimensional verification is performed, which can avoid waste of resources, accurately handle the fault, and ensure the speed of handling the fault.

[0072] According to an embodiment of the present application, the method further comprises, before performing the software restart on the fault host: issuing a shutdown instruction to the fault host, the shutdown instruction representing stopping running services on the fault host; in a case where the fault host does not respond to the shutdown instruction, issuing a hardware restart instruction to the fault host and performing multi-dimensional verification on the hardware-restarted fault host; and in a case where the fault host does not respond to the hardware restart instruction, deleting a node corresponding to the fault host.

[0073] In the embodiments of the present application, before the software of the faulty host is restarted, there can be services running on the host, but since the host is faulty, the running services can be zombie processes that occupy resources, and the faulty state can last for a long time, which can cause data loss or damage, for example, data is not flushed to the disk due to sudden software crashes. Therefore, the services on the host need to be stopped first, and then the software and / or hardware of the host is restarted.

[0074] If the host does not respond to the stop command, it can be considered that the software layer of the host has lost the ability to process the command, and even if the software is restarted, it will fail. Therefore, the faulty host can be directly restarted.

[0075] It can be understood that before attempting to repair the fault, the services running on the host need to be stopped to avoid resource occupation and reduce the impact on data. If no response is received, it means that the host has lost the ability to process from the software layer, and the hardware can be directly restarted to reduce unnecessary fault processing procedures.

[0076] According to the embodiments of the present application, the N hosts in the cluster are judged for faults according to the fault judgment rule, and the faulty host is determined by: monitoring the heartbeat time of each host in the cluster, the heartbeat time representing the time when the host automatically updates the heartbeat signal; in the case that the time interval between the heartbeat time of the host and the current time is greater than a preset threshold, the host is determined as a faulty host.

[0077] In the embodiments of the present application, the heartbeat signal is a signal periodically sent to confirm that the connection is normal or the device is in a live state, and the heartbeat time is the time when the host automatically updates the heartbeat signal. The preset threshold can be set to a value less than the heartbeat signal sending period.

[0078] For example, if the host sends the heartbeat signal every second, the preset threshold can be set to ten seconds. If it is monitored that the time interval between the heartbeat time of a host and the current time is twelve seconds, it means that the host can be a faulty host.

[0079] It can be understood that whether the heartbeat time is updated for a long time can be used to determine whether the host is faulty, which can automatically monitor the faulty host without manual inspection, and can quickly find the faulty host for quick processing.

[0080] Figure 4 The flowchart of the fault processing method according to another embodiment of the present application is schematically shown.

[0081] As Figure 4As shown, the fault processing method of this embodiment includes operations S401-S415, which can be performed by the server 105 in the fault processing device.

[0082] In operation S401, a fault host is determined.

[0083] For example, the plurality of hosts included in the cluster can be judged for faults according to a fault judgment rule to determine the fault host. Specifically, it can be judged whether the time interval between the time when the host last sent a heartbeat signal and the current time is greater than a preset threshold. For example, the time interval between the time when the host last sent a heartbeat signal and the current time is 20 seconds, and the preset threshold is 10 seconds, so the host can be determined as a fault host.

[0084] In operation S402, the fault host is isolated.

[0085] For example, the identification information of the fault host can be adjusted, and the identification information of the fault host is adjusted from first identification information to second identification information. The first identification information represents the priority of the host, and the first identification information can be used to determine whether the host is a core host. The second identification information represents an uncallable host.

[0086] In operation S403, the identification information of the standby host in the cluster is adjusted.

[0087] After the fault host is isolated, the identification information of the standby host needs to be adjusted in order to ensure the stability of the resource pool.

[0088] For example, when the number of core hosts does not meet the number threshold, the identification information of the standby host is adjusted from third identification information to identification information corresponding to the core host. The third identification information represents that the host is a standby host.

[0089] In operation S404, it is judged whether the standby host is resource deficient.

[0090] After adjusting the identification information of the standby host, the core host still cannot meet the first number threshold (i.e., the number threshold of core hosts) or the number of non-core hosts does not meet the second number threshold (i.e., the number threshold of non-core hosts), which can be considered as resource deficiency of the standby host.

[0091] In operation S405, it is judged whether the fault host is a core host.

[0092] For example, according to the identification information of the fault host before it is isolated, it is judged whether the fault host is a core host.

[0093] In operation S406, the fault host is evicted.

[0094] For a non-core fault host machine, it can be directly evicted to quickly release the resources occupied by it, and the evicting of the fault host machine can be implemented by deleting the node of the host machine.

[0095] In operation S407, it is determined whether the fault host machine returns a response to the shutdown instruction.

[0096] Since the services on the fault host machine can still be running when the fault host machine fails, the shutdown instruction needs to be issued first, and then it is determined whether the fault host machine can respond to the shutdown instruction.

[0097] In operation S408, the fault host machine is software restarted.

[0098] If the fault host machine can respond to the shutdown instruction, it means that there can be a problem in the software layer, and in this case, the software restart can be tried first, that is, the software or the entire operating system running on the host machine is restarted through a software restart instruction.

[0099] In operation S409, it is determined whether the software restart is successful.

[0100] For example, whether the software restart is successful can be determined by determining whether the services or processes on the host machine are started.

[0101] In operation S410, the fault host machine is hardware restarted.

[0102] In the case that the software restart of the fault host machine fails or the fault host machine does not respond to the shutdown instruction, it means that there can be a problem in the hardware layer of the fault host machine, and the fault host machine needs to be hardware restarted.

[0103] In operation S411, it is determined whether the fault host machine passes the multi-dimensional test.

[0104] For example, the multi-dimensional test can include at least one of hardware diagnosis, container startup test, and network connectivity check.

[0105] In operation S412, the identification information of the fault host machine is adjusted.

[0106] If the fault host machine passes the multi-dimensional test, it means that the fault host machine has returned to a normal host machine after the software restart or the hardware restart, and therefore, the identification information of the fault host machine can be adjusted to third identification information, that is, the host machine is regarded as a standby host machine.

[0107] In operation S413, the identification information of the non-core host machine in the cluster is adjusted.

[0108] In the case of insufficient backup host resources, the identification information of the non-core host needs to be adjusted for relatively more important services.

[0109] In operation S414, it is determined whether the non-core host is resource deficient.

[0110] The non-core host resource deficiency can include one of the following: the core host still cannot meet the first quantity threshold after adjusting the identification information of the non-core host, and the number of non-core hosts does not meet the second quantity threshold.

[0111] In operation S415, it is reported to the system.

[0112] In the case of non-core host resource deficiency or failure host restart without passing multi-dimensional test, the abnormal condition is reported to the system for manual intervention.

[0113] Based on the above failure processing method, the application also provides a failure processing device. The following will be combined with Figure 5 The device is described in detail.

[0114] Figure 5 The structure block diagram of the failure processing device according to the embodiment of the application is schematically shown.

[0115] As Figure 5 shown, the failure processing device 500 of the embodiment includes a determination module 510, a first adjustment module 520, and a second adjustment module 530.

[0116] The determination module 510 is configured to determine the failure host according to the failure judgment rule for the N host computers in the cluster, and N is an integer greater than 1. In an embodiment, the determination module 510 can be configured to perform the operation S210 described above, and details are not repeated here.

[0117] The first adjustment module 520 is configured to adjust the identification information of the failure host from the first identification information to the second identification information, the first identification information representing the priority of the host computer, and the second identification information representing the uncallable host computer. In an embodiment, the first adjustment module 520 can be configured to perform the operation S220 described above, and details are not repeated here.

[0118] The second adjustment module 530 is configured to adjust the identification information of the host computer in the cluster based on the first identification information of each host computer in the cluster and the host computer quantity threshold corresponding to the priority. In an embodiment, the second adjustment module 530 can be configured to perform the operation S230 described above, and details are not repeated here.

[0119] According to an embodiment of the present application, the first identification information comprises a first priority and a second priority, the first priority represents that the host computer is a core host computer, and the second priority represents that the host computer is a non-core host computer. The second adjusting module 530 comprises at least one of the following: a first adjusting submodule, a second adjusting submodule, and a first processing module. The first adjusting submodule is configured to, in a case where the number of host computers corresponding to the first priority does not reach a corresponding number threshold, adjust the identification information of the standby host computer from third identification information to the first priority, the third identification information representing that the host computer is a standby host computer. The second adjusting submodule is configured to, in a case where the number of host computers corresponding to the second priority does not reach a corresponding number threshold, and the number of host computers corresponding to the first priority reaches a corresponding number threshold, adjust the identification information of the standby host computer from the third identification information to the second priority. The first processing module is configured to, in a case where the number of host computers corresponding to the first priority does not reach a corresponding number threshold and the number of standby host computers is zero: determine at least one target host computer from the host computers with the second priority, and adjust the identification information of the target host computer from the second priority to the first priority.

[0120] According to an embodiment of the present application, the fault processing apparatus 500 further comprises a software restart module, a first verification module, a third adjusting submodule, and a first deletion module. The software restart module is configured to, in a case where the fault host computer is a core host computer, perform software restart on the fault host computer, the software restart representing restarting a service on the fault host computer. The first verification module is configured to perform multi-dimensional verification on the fault host computer after the software restart, the multi-dimensional verification comprising at least one of hardware diagnosis, container startup test, and network connectivity check. The third adjusting submodule is configured to, in a case where the multi-dimensional verification on the fault host computer after the software restart passes, adjust the identification information of the fault host computer from the second identification information to the third identification information. The first deletion module is configured to, in a case where the fault host computer is a non-core host computer, delete a node corresponding to the fault host computer, the node representing a logical mapping of the fault host computer in the cluster control system.

[0121] According to an embodiment of the present application, the fault processing apparatus 500 further comprises a hardware restart module, a second verification module, a fourth adjusting submodule, and a reporting module. The hardware restart module is configured to, in a case where the multi-dimensional verification on the fault host computer after the software restart fails, perform hardware restart on the fault host computer, the hardware restart representing completely powering off and then powering on the hardware of the fault host computer. The second verification module is configured to perform multi-dimensional verification on the fault host computer after the hardware restart. The fourth adjusting submodule is configured to, in a case where the multi-dimensional verification on the fault host computer after the hardware restart passes, adjust the identification information of the fault host computer from the second identification information to the third identification information. The reporting module is configured to, in a case where the multi-dimensional verification on the fault host computer after the hardware restart fails, report information of the fault host computer to a system.

[0122] According to an embodiment of the present application, the fault processing apparatus 500 further comprises a second processing module, a third processing module and a second deleting module. The second processing module is configured to send a shutdown instruction to the fault host before software restarting the fault host, the shutdown instruction representing stopping running services on the fault host. The third processing module is configured to send a hardware restart instruction to the fault host and perform multi-dimensional verification on the fault host after hardware restarting in case that the fault host does not respond to the shutdown instruction. The second deleting module is configured to delete the node corresponding to the fault host in case that the fault host does not respond to the hardware restart instruction.

[0123] According to an embodiment of the present application, the determining module comprises a monitoring module and a judging module. The monitoring module is configured to monitor the heartbeat time of each host in the cluster, the heartbeat time representing the time of automatically updating when the host sends a heartbeat signal. The judging module is configured to judge the host as a fault host in case that the time interval between the heartbeat time of the host and the current time is greater than a preset threshold.

[0124] According to an embodiment of the present application, any of the determining module 510, the first adjusting module 520 and the second adjusting module 530 can be combined in one module for implementation, or any of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present application, at least one of the determining module 510, the first adjusting module 520 and the second adjusting module 530 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware that can be integrated or packaged with a circuit, or implemented in any one of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the determining module 510, the first adjusting module 520 and the second adjusting module 530 can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0125] Figure 6 A block diagram of an electronic device suitable for implementing the fault processing method according to an embodiment of the present application is schematically shown.

[0126] As Figure 6As shown, the electronic device 600 according to an embodiment of the present application includes a processor 601 which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. The processor 601 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application.

[0127] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 602 and / or the RAM 603. Note that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0128] According to embodiments of the present application, the electronic device 600 can also include an input / output (I / O) interface 605 which is also connected to the bus 604. The electronic device 600 can also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as necessary. A removable medium 611 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage section 608 as necessary.

[0129] The application further provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist independently without being assembled into the device / apparatus / system. The computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0130] According to the embodiments of the application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the application, the computer readable storage medium can include one or more of the above-described ROM 602 and / or RAM 603 and / or one or more memories other than the ROM 602 and the RAM 603.

[0131] The embodiments of the application also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the fault processing method provided by the embodiments of the application.

[0132] The above-described functions defined in the system / apparatus of the embodiments of the application are performed when the computer program is executed by the processor 601. According to the embodiments of the application, the above-described system, apparatus, module, unit, etc. can be implemented by computer program modules.

[0133] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 609 and / or installed from the detachable medium 611. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0134] In such embodiments, the computer program can be downloaded and installed from the network via the communication section 609, and / or installed from the removable media 611. When the computer program is executed by the processor 601, the above-described functions defined in the system of the embodiments of the present application are executed. According to the embodiments of the present application, the system, device, apparatus, module, unit, and the like described above can be realized by the computer program modules.

[0135] According to the embodiments of the present application, the program code for executing the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language, or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectural, functional, and operational scenarios of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks that are shown in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0137] Those skilled in the art can understand that the features described in various embodiments of the present application can be combined and / or integrated in various combinations, even if such combinations are not explicitly described in the present application. In particular, the features described in various embodiments of the present application can be combined and / or integrated in various combinations without departing from the spirit and teachings of the present application. All such combinations and / or integrations are within the scope of the present application.

Claims

1. A fault handling method, characterized in that, The method includes: Based on the fault diagnosis rules, fault diagnosis is performed on N host machines in the cluster to determine the faulty host machine, where N is an integer greater than 1; The identification information of the faulty host machine is adjusted from the first identification information to the second identification information, where the first identification information represents the priority of the host machine, and the second identification information represents that the host machine cannot be invoked; and Based on the first identification information of each host in the cluster and the host number threshold corresponding to the priority, the identification information of the host in the cluster is adjusted.

2. The method according to claim 1, characterized in that, The first identification information includes a first priority and a second priority. The first priority indicates that the host machine is a core host machine, and the second priority indicates that the host machine is a non-core host machine. Adjusting the identification information of the host machines in the cluster based on the first identification information of each host machine in the cluster and the host machine number threshold corresponding to the priority includes at least one of the following: If the number of host machines corresponding to the first priority does not reach the corresponding number threshold, the identification information of the backup host machine is adjusted from the third identification information to the first priority, whereby the third identification information indicates that the host machine is a backup host machine. If the number of host machines corresponding to the second priority does not reach the corresponding number threshold, but the number of host machines corresponding to the first priority reaches the corresponding number threshold, the identification information of the standby host machine will be adjusted from the third identification information to the second priority. When the number of hosts corresponding to the first priority does not reach the corresponding threshold and the number of standby hosts is zero: At least one target host is identified from the host machines whose identification information indicates the second priority. The identification information of the target host machine is adjusted from the second priority to the first priority.

3. The method according to claim 2, characterized in that, The method further includes: If the faulty host is a core host, a software reboot is performed on the faulty host, whereby the software reboot represents the restart of services on the faulty host. Perform multi-dimensional verification on the faulty host machine after software restart, including at least one of hardware diagnostics, container startup testing, and network connectivity checks. If the faulty host machine after the software restart passes multi-dimensional verification, the identification information of the faulty host machine will be adjusted from the second identification information to the third identification information. If the faulty host is a non-core host, delete the node corresponding to the faulty host. The node represents the logical mapping of the faulty host in the cluster control system.

4. The method according to claim 3, characterized in that, The method further includes: If the multi-dimensional verification of the faulty host machine after the software restart fails, the faulty host machine is hardware restarted. The hardware restart means that the hardware of the faulty host machine is completely powered off and then powered on again. Perform the aforementioned multi-dimensional verification on the faulty host machine after hardware reboot; If the multi-dimensional verification of the faulty host machine after the hardware restart is passed, the identification information of the faulty host machine will be adjusted from the second identification information to the third identification information. If the multi-dimensional verification of the faulty host machine after the hardware restart fails, the information of the faulty host machine will be reported to the system.

5. The method according to claim 4, characterized in that, The method further includes, before performing a software reboot on the faulty host machine: A shutdown command is issued to the faulty host machine, the shutdown command indicating that the services on the faulty host machine shall be stopped; If the faulty host does not respond to the shutdown command, a hardware restart command is issued to the faulty host and the faulty host after hardware restart is verified in multiple dimensions. If the faulty host does not respond to the hardware restart command, delete the node corresponding to the faulty host.

6. The method according to claim 1, characterized in that, The step of determining faulty host machines by analyzing the N host machines in the cluster according to fault judgment rules includes: Monitor the heartbeat time of each host machine in the cluster, where the heartbeat time represents the time when the host machine automatically updates when it sends a heartbeat signal; If the time interval between the host machine's heartbeat time and the current time is greater than a preset threshold, the host machine is determined to be a faulty host machine.

7. A fault handling device, characterized in that, The device includes: The determination module is used to determine the faulty host machine by judging the faults of N host machines in the cluster according to the fault judgment rules, where N is an integer greater than 1; A first adjustment module is used to adjust the identification information of the faulty host machine from a first identification information to a second identification information, wherein the first identification information represents the priority of the host machine, and the second identification information represents that the host machine cannot be invoked; and The second adjustment module is used to adjust the identification information of the host machines in the cluster based on the first identification information of each host machine in the cluster and the host machine number threshold corresponding to the priority.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Images