Service exception processing method, device and equipment and computer readable storage medium

Abstract

The invention discloses a service exception handling method, device and equipment and a computer readable storage medium, belongs to the field of cloud platforms, and aims to solve the problems that (1) a sentry service cannot deal with a fault of a container group where a service node is located and (2) the exception of the container group where the service node is located can be found in time through periodic monitoring. According to the invention, through the management node of the cloud platform, the container group where each service node of the target service is located is identified every preset period; and performing master-slave switching of the service nodes (taking one slave service node in a normal operation state as a new master service node, and taking other service nodes except the newly determined master service node as slave service nodes) when the target service cannot be normally provided due to the failure of the first container group. Through active switching and timely recovery of the target service, the reliability of the target service and the user experience are improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud platforms, and in particular to a service exception handling method, device, equipment and computer-readable storage medium. Background Art

[0002] High-availability distributed services are usually deployed in PODs (Plain Old Docker) of multiple servers on a cloud platform. Then, one of the service nodes of the high-availability distributed service is set as the master service node, and the remaining service nodes are set as slave service nodes. The master service node provides services, and the slave service nodes are used as redundant backups.

[0003] Although the Sentinel service can switch the master service node in some cases to ensure the normal operation of the high-availability distributed service, the master-slave switching through the Sentinel service can only cope with limited unexpected situations. The master-slave switching of the high-availability distributed service can only be achieved in a few unexpected situations, which reduces the reliability and user experience of the high-availability distributed service.

[0004] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. Summary of the invention

[0005] The purpose of the present invention is to provide a service exception processing method, device, equipment and computer-readable storage medium. In the present invention, the management node of the cloud platform is used to identify the container group where each service node of the target service is located at every preset period, and when the first container group where the main service node is located is not operating normally, the master-slave switching of the service node is performed (a slave service node in a normal operating state is used as a new main service node, and other service nodes except the newly determined main service node are used as slave service nodes). When the target service cannot be provided normally due to a failure of the first container group, the target service can be restored in time through active switching, thereby improving the reliability of the target service and user experience.

[0006] In order to solve the above technical problems, the present invention provides a service exception handling method, which is applied to a management node of a cloud platform, comprising:

[0007] At every preset period, identify the container group where each service node of the target service is located, wherein the target service is a highly available distributed service, and the service nodes include a single master service node and several slave service nodes;

[0008] Determine whether the first container group of the target service is in a normal operating state, wherein the first container group is: the container group where the primary service node is located;

[0009] If it is not in normal operation, a slave service node of the target service in normal operation is used as a new master service node, and other service nodes of the target service except the newly determined master service node are used as slave service nodes.

[0010] On the other hand, after determining whether the first container group of the target service is in a normal operating state, the service exception handling method further includes:

[0011] If it is in normal operation, determine whether the main service node in the first container group can operate normally;

[0012] If it cannot work normally, a slave service node of the target service in normal operation will be used as the new master service node, and other service nodes of the target service except the newly determined master service node will be used as slave service nodes.

[0013] On the other hand, the container groups where the service nodes of the target service are located include:

[0014] Determine the stateful replica set of the target service based on the custom resources related to the target service;

[0015] The container group where each service node of the target service is located and its container group status are determined from the stateful replica set of the target service, wherein the container group status includes a normal operating state and an abnormal operating state.

[0016] On the other hand, determining whether the primary service node in the first container group can work normally includes:

[0017] Sending an assistance determination request to the client, wherein the assistance determination request is used to request assistance in determining whether the primary service node in the first container group can work normally;

[0018] An assisted judgment result fed back by the client is received, wherein the assisted judgment result includes whether the primary service node in the first container group can work normally or cannot work normally.

[0019] On the other hand, after identifying the container group where each service node of the target service is located at each preset period, the service exception handling method further includes:

[0020] Determine whether the target service has a faulty container group, where the faulty container group is: a container group in an abnormally normal operating state;

[0021] If so, the fault information of the fault container group is pushed.

[0022] On the other hand, the target service includes a remote dictionary service, and the cloud platform includes a cloud platform to which a Kubernetes container orchestration engine is applied.

[0023] On the other hand, the target service is in sentinel mode;

[0024] After a slave service node in a normal operating state of the target service is used as a new master service node, and other service nodes of the target service except the newly determined master service node are used as slave service nodes, the service exception handling method further includes:

[0025] The network protocol address of the container group where the newly determined main service node is located is used as the network protocol address of the container group where the main service node is located in the sentinel service configuration file, where the sentinel configuration file refers to the configuration file of the sentinel service corresponding to the target service;

[0026] Control the restart of the sentinel service corresponding to the target service;

[0027] After the sentinel service corresponding to the target service is restarted, it is determined whether the primary network protocol address pointed to in the sentinel service is the preset initial address, wherein the primary network protocol address is: the network protocol address of the primary service node;

[0028] If it is a preset initial address, replace the primary network protocol address pointed to in the Sentinel service with the network protocol address of the container group where the primary service node is located in the Sentinel service configuration file;

[0029] If it is not the preset initial address, end.

[0030] In order to solve the above technical problems, the present invention also provides a service exception handling device, which is applied to a management node of a cloud platform, comprising:

[0031] A first identification module is used to identify, at every preset period, a container group where each service node of a target service is located, wherein the target service is a highly available distributed service, and the service nodes include a single master service node and a plurality of slave service nodes;

[0032] A first judgment module is used to judge whether the first container group of the target service is in a normal operating state. If it is not in a normal operating state, the first switching module is triggered, wherein the first container group is: the container group where the primary service node is located;

[0033] The first switching module is used to use a slave service node of the target service in a normal operating state as a new master service node, and use other service nodes of the target service except the newly determined master service node as slave service nodes.

[0034] In order to solve the above technical problems, the present invention also provides a service exception processing device, comprising:

[0035] Memory for storing computer programs;

[0036] The processor is used to implement the steps of the service exception handling method as described above when executing the computer program.

[0037] In order to solve the above technical problem, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the service exception handling method described above are implemented.

[0038] Beneficial effect: The present invention provides a method for handling service anomalies. Considering that (1) the sentinel service cannot cope with the situation of "failure of the container group where the service node is located" and (2) periodic monitoring can timely discover the anomaly of the container group where the service node is located, the present invention uses the management node of the cloud platform to identify the container group where each service node of the target service is located at every preset period, and when the first container group where the master service node is located is not operating normally, the master-slave switching of the service node is performed (a slave service node in normal operation is used as the new master service node, and other service nodes except the newly determined master service node are used as slave service nodes). When the target service cannot be provided normally due to the failure of the first container group, the target service can be restored in time through active switching, thereby improving the reliability of the target service and user experience.

[0039] The present invention also provides a service exception handling device, equipment and computer-readable storage medium, which have the same beneficial effects as the above service exception handling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the relevant technologies and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A flow chart of a service exception handling method provided by the present invention;

[0042] Figure 2 A structural diagram of a service exception handling system provided by the present invention;

[0043] Figure 3 A distribution diagram of a target service provided by the present invention;

[0044] Figure 4 A structural diagram of another service exception handling system provided by the present invention;

[0045] Figure 5 A schematic diagram of the structure of a service exception handling device provided by the present invention;

[0046] Figure 6 A schematic diagram of the structure of a service exception handling device provided by the present invention;

[0047] Figure 7 A schematic diagram of the structure of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0048] The core of the present invention is to provide a service exception processing method, device, equipment and computer-readable storage medium. In the present invention, the management node of the cloud platform is used to identify the container group where each service node of the target service is located at every preset period, and when the first container group where the main service node is located is not operating normally, the master-slave switching of the service node is performed (a slave service node in normal operation is used as the new main service node, and other service nodes except the newly determined main service node are used as slave service nodes). When the target service cannot be provided normally due to the failure of the first container group, the target service can be restored in time through active switching, thereby improving the reliability of the target service and user experience.

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Please refer to Figure 1 , Figure 1 A flow chart of a service exception handling method provided by the present invention is provided. The service exception handling method is applied to a management node of a cloud platform, and includes:

[0051] S101: every preset period, identifying the container group where each service node of the target service is located, wherein the target service is a highly available distributed service, and the service nodes include a single master service node and a plurality of slave service nodes;

[0052] Specifically, taking into account the technical problems in the above background technology, and considering that (1) the sentinel service cannot cope with the situation of "failure of the container group where the service node is located" and (2) periodic monitoring can timely discover the abnormality of the container group where the service node is located, the embodiment of the present invention intends to periodically monitor the "failure of the container group where the service node is located" of the target service, so as to timely discover the "failure of the container group where the service node is located" and actively switch the master and slave service nodes, thereby making up for the shortcomings of the sentinel service and improving the reliability of high-availability distributed services. Therefore, the service exception handling method in this step can be applied to the management node of the cloud platform, and at every preset period, the container group where each service node of the target service is located is identified, and used as the data basis for subsequent steps.

[0053] Specifically, in order to better illustrate the embodiments of the present invention, please refer to Figure 2 , Figure 2 A schematic diagram of the structure of a service exception handling system provided by the present invention, from Figure 2 It can be seen intuitively on the right that the target service nodes (that is, the service nodes of the target service) are distributed on the three server nodes of the cloud platform, that is, the first to third server nodes are each equipped with a target service node, one of which can be designated as the main service node. Correspondingly, when the sentinel mode is applied, a number (odd number) of sentinel service nodes (service nodes of the sentinel service) can also be set in the cloud platform. The sentinel service nodes can usually be distributed on the server nodes where the target service nodes are set.

[0054] Among them, the preset period can be set independently, for example, it can be set according to the interruption time requirement of the business layer to meet the interruption time requirement of the business layer, for example, it can be 10 seconds, etc., and the embodiment of the present invention is not limited here.

[0055] S102: Determine whether the first container group of the target service is in a normal operating state, wherein the first container group is: the container group where the primary service node is located;

[0056] Specifically, since the failure of the first container group will directly cause the primary service node of the target service to be unable to provide services, thereby causing service interruption, this step focuses on the first container group, that is, it can be determined whether the first container group of the target service is in a normal operating state, so as to trigger the actions of subsequent steps according to the determination result.

[0057] Among them, for any container group, it may be in an abnormal operating state due to various reasons, such as the server node where the container group is located is powered off or disconnected from the network (due to abnormal unplugging of the network cable or network card failure, etc.), etc., which is not limited in the embodiments of the present invention.

[0058] S103: If it is not in a normal operating state, a slave service node of the target service in a normal operating state is used as a new master service node, and other service nodes of the target service except the newly determined master service node are used as slave service nodes.

[0059] Specifically, in order to enable the target service due to the "failure of the first container group" to provide services normally, in this step, when the first container group is not in a normal operating state, a slave service node of the target service in a normal operating state can be used as a new master service node, and other service nodes of the target service except the newly determined master service node can be used as slave service nodes, for example, Figure 2 In the embodiment, it is assumed that the target service node in the first server node serves as the main service node and is in an abnormal operating state. At this time, if the target service nodes in the second server node and the third server node are both operating normally, a slave service node can be selected as the main service node for the target service according to a preset selection rule. For example, the target service node in the server node with a smaller serial number can be used as the main service node for the target service, and then the other service nodes of the target service can be used as "slave nodes of the newly determined main service node", etc. The embodiment of the present invention is not limited here.

[0060] Among them, when determining whether the service node is in a normal operating state, the operating state of the container group where the service node is located can be used as the operating state of the service node, or the actual operating state of the service node can be directly determined by other means, which is not limited in this embodiment of the present invention.

[0061] The present invention provides a method for handling service anomalies. Considering that (1) the sentinel service cannot cope with the situation of "failure of the container group where the service node is located" and (2) periodic monitoring can timely discover the anomaly of the container group where the service node is located, the present invention uses a management node of a cloud platform to identify the container group where each service node of the target service is located at every preset period, and when the first container group where the master service node is located is not operating normally, the master-slave switching of the service node is performed (a slave service node in normal operation is used as the new master service node, and other service nodes except the newly determined master service node are used as slave service nodes). When the target service cannot be provided normally due to the failure of the first container group, the target service can be restored in time through active switching, thereby improving the reliability of the target service and user experience.

[0062] Based on the above embodiments:

[0063] As an optional embodiment, after determining whether the first container group of the target service is in a normal operating state, the service exception handling method further includes:

[0064] If it is in normal operation, determine whether the main service node in the first container group can operate normally;

[0065] If it cannot work normally, a slave service node of the target service in normal operation will be used as the new master service node, and other service nodes of the target service except the newly determined master service node will be used as slave service nodes.

[0066] Specifically, considering that in some cases (for example, the container group network is disconnected), the container group can still display a normal operating state, but the container group is no longer accessible. If a master service node is provided inside the container group, this situation will still cause the target service to be unable to provide services. Therefore, in order to restore the normal operation of the target service in time under this situation, in the embodiment of the present invention, even when the first container group is in a normal operating state, it is still possible to continue to determine whether the master service node in the first container group can operate normally. If it cannot operate normally, the master-slave service node switching work can be performed, that is, a slave service node in a normal operating state of the target service is used as a new master service node, and other service nodes of the target service except the newly determined master service node are used as slave service nodes, so as to restore the normal operation of the target service.

[0067] There may be many reasons for the container group network disconnection, such as network configuration errors within the container group, container group port conflicts, and traffic anomalies, which are not limited in the embodiments of the present invention.

[0068] As an optional embodiment, identifying the container group where each service node of the target service is located includes:

[0069] Determine the stateful replica set of the target service based on the custom resources associated with the target service;

[0070] The container group where each service node of the target service is located and its container group status are determined from the stateful replica set of the target service, wherein the container group status includes a normal operating state and an abnormal operating state.

[0071] Specifically, considering that the stateful replica set (Statefulset resource) of the target service can efficiently and accurately query the POD where each service node of the target service is located and the status of the POD, and the stateful replica set of the target service can be determined through the CR (Custom Resource) related to the target service, the stateful replica set of the target service can be determined according to the custom resource related to the target service in an embodiment of the present invention, and then the container group where each service node of the target service is located and its container group status can be determined from the stateful replica set of the target service, so as to use it as the data basis for subsequent steps.

[0072] Specifically, in addition to the container group status, it is also possible to determine from the stateful replica set whether the service node in each container group where the service node of the target service is located is a master node or a slave node.

[0073] The CR of the target service may include characteristic information of each service node of the target service, such as the service node name, the server node name, CPU resource range, memory resource range, etc., which is not limited in the embodiment of the present invention.

[0074] Of course, in addition to this method, there may be other methods for identifying the container group where each service node of the target service is located, which is not limited in the embodiment of the present invention.

[0075] Specifically, in Figure 2 In the cloud platform, the management node can determine the POD where each service node is located based on the custom resources.

[0076] As an optional embodiment, determining whether the primary service node in the first container group can work normally includes:

[0077] Sending an assistance determination request to the client, wherein the assistance determination request is used to request assistance in determining whether the primary service node in the first container group can work normally;

[0078] An assisted judgment result fed back by the client is received, wherein the assisted judgment result includes whether the primary service node in the first container group can work normally or cannot work normally.

[0079] Specifically, considering that whether the primary service node in the first container group can work normally can be judged by "whether the client can use the target service normally", in order to make a judgment efficiently and accurately, in an embodiment of the present invention, the client can be asked to assist in the judgment, that is, an assistance judgment request can be sent to the client, so that the client can "detect whether the target service can be used normally" to assist in judging whether the primary service node in the first container group can work normally or not, and then the assistance judgment result is fed back to the management node of the cloud platform. For the management node of the cloud platform, it can receive the assistance judgment result fed back by the client, so as to efficiently and accurately determine "whether the primary service node in the first container group can work normally".

[0080] Of course, in addition to this specific method, other methods may be used to determine whether the primary service node in the first container group can work normally, which is not limited in the embodiment of the present invention.

[0081] As an optional embodiment, after identifying the container group where each service node of the target service is located at every preset period, the service exception handling method further includes:

[0082] Determine whether the target service has a faulty container group, where the faulty container group is: a container group in an abnormally normal operating state;

[0083] If it exists, the fault information of the failed container group is pushed.

[0084] Specifically, considering that even if the service node in the faulty container group is a slave service node, fault recovery should be performed in a timely manner to maintain high availability of the target service. Therefore, in order to facilitate the staff to promptly learn of the failure of the container group and carry out maintenance, in the embodiment of the present invention, when it is determined that the target service has a faulty container group, the fault information of the faulty container group can be pushed.

[0085] The fault information may include the server node where the faulty container group is located and the POD name, etc., which is not limited in the embodiment of the present invention.

[0086] As an optional embodiment, the target service includes a remote dictionary service, and the cloud platform includes a cloud platform to which a Kubernetes container orchestration engine is applied.

[0087] Specifically, Redis (Remote Dictionary Service) is an open source, memory-based, persistent, log-based, Key-Value database written in ANSI (a character encoding standard) C language, supporting the network, and providing APIs (Application Programming Interface) in multiple languages. Redis data is stored in memory. It has a very fast read and write speed and can handle more than 100,000 read and write operations per second.

[0088] Specifically, cloud platforms based on K8S (Kubernetes, the Kubernetes container orchestration engine) are highly mature and reliable.

[0089] Of course, in addition to Redis, the target service can also be of many other types. In addition to the cloud platform using the Kubernetes container orchestration engine, the cloud platform can also be of many other types, which is not limited in the embodiments of the present invention.

[0090] Specifically, Figure 2 and Figure 4 The additional coordinator is a function module in the operator, the management software of the cloud platform using K8S. Its English name is ExtraReconciler. The update function in ExtraReconciler can be used to switch the master and slave service nodes of the target service. The embodiment of the present invention can enhance the capabilities of the operator.

[0091] As an optional embodiment, the target service is in sentinel mode;

[0092] After a slave service node in a normal operating state of the target service is used as a new master service node, and other service nodes of the target service except the newly determined master service node are used as slave service nodes, the service exception handling method further includes:

[0093] The network protocol address of the container group where the newly determined main service node is located is used as the network protocol address of the container group where the main service node is located in the sentinel service configuration file, where the sentinel configuration file refers to the configuration file of the sentinel service corresponding to the target service;

[0094] Control the restart of the sentinel service corresponding to the target service;

[0095] After the sentinel service corresponding to the target service is restarted, it is determined whether the primary network protocol address pointed to in the sentinel service is the preset initial address, wherein the primary network protocol address is: the network protocol address of the primary service node;

[0096] If it is a preset initial address, replace the primary network protocol address pointed to in the Sentinel service with the network protocol address of the container group where the primary service node is located in the Sentinel service configuration file;

[0097] If it is not the preset initial address, end.

[0098] Specifically, in order to better illustrate the embodiments of the present invention, please refer to Figure 3 and Figure 4 , Figure 3 A distribution diagram of a target service provided by the present invention, Figure 4 A schematic diagram of the structure of another service exception handling system provided by the present invention, Figure 3 The relationship between the Sentinel service, the client, and the target service is shown in Figure 1. The client can obtain the address of the target service's primary service node through the Sentinel service to access the target service, while the Sentinel service can monitor and failover the target service. Figure 3 The sentinel service in may include three service nodes, and the target service may also include three service nodes.

[0099] Specifically, considering that in the sentinel mode, after the target service is switched from the master to the slave service node, on the one hand, the sentinel service needs to be restarted, and on the other hand, the IP address of the new master service node of the target service needs to be updated to the sentinel service, and considering that the sentinel service will rebuild the POD where the sentinel service is located after the restart, the sentinel service may not be able to automatically point to the "network protocol address of the container group where the master service node is located in the sentinel service configuration file" due to unexpected reasons during the reconstruction process, and display the preset initial address (for example, 127.0.0.1). Therefore, in the embodiment of the present invention, it is possible to "use the network protocol address of the container group where the newly determined master service node is located as the sentinel service configuration file" If it is the preset initial address, it means that it cannot automatically point to the “network protocol address of the container group where the main service node is located in the sentinel service configuration file” due to unexpected reasons. Therefore, in this case, in an embodiment of the present invention, the main network protocol address pointed to in the sentinel service can be replaced with the network protocol address of the container group where the main service node is located in the sentinel service configuration file, thereby ensuring the normal execution of the sentinel service, and ensuring that the client can use the target service normally, further improving the reliability of the target service.

[0100] Specifically, it is worth mentioning that the sentinel mode itself does not have the ability to quickly select a new master service node when the master service node of the target service is suddenly disconnected, and the management node of the cloud platform usually manages the target service only by adding and configuring service nodes. Therefore, in the embodiment of the present invention, the master-slave service node switching of the target service in unexpected situations such as POD failure can be added to the management node of the cloud platform.

[0101] Additionally, as an optional embodiment, the target service is in sentinel mode;

[0102] The service exception handling method also includes:

[0103] When a second container group that has resumed normal operation appears in the cloud platform, it is determined whether the target service in the cloud platform has two primary service nodes at the same time, wherein the second container group refers to the container group where the service node of the target service is located;

[0104] If both exist, the service node in the second container group that has recovered to normal operation will be used as the slave service node of the target service;

[0105] When a third container group that has resumed normal operation appears in the cloud platform, it is determined whether two primary service nodes exist at the same time for the target sentinel service in the cloud platform, wherein the third container group refers to the container group where the service node of the target sentinel service is located, and the target sentinel service refers to the sentinel service corresponding to the target service;

[0106] If both exist, the service node in the third container group that has recovered to normal operation will be used as the slave service node of the target sentinel service.

[0107] Specifically, considering that a faulty POD may resume normal operation after being discovered and repaired, and for a faulty POD, when it resumes normal operation, it will resume the state before the fault. If a master service node is set in the faulty POD before the fault, then whether it is the recovery of the container group of the master service or the recovery of the container group of the sentinel service, there will be multiple master service nodes in the service, resulting in the inability to provide the service normally. Therefore, in order to improve the reliability of the target service, in an embodiment of the present invention, when a second container group that has resumed normal operation appears in the cloud platform, it can be determined whether there are two master service nodes for the target service in the cloud platform at the same time. If they exist at the same time, the service node in the second container group that has resumed normal operation will be used as the slave service node of the target service; for the sentinel service, when a third container group that has resumed normal operation appears in the cloud platform, it can be determined whether there are two master service nodes for the target sentinel service in the cloud platform at the same time. If they exist at the same time, the service node in the third container group that has resumed normal operation will be used as the slave service node of the target sentinel service; in an embodiment of the present invention, the reliability of the target service can be further improved by coordinating the master and slave nodes of the target service and the sentinel service in the faulty POD recovery scenario.

[0108] Among them, the reason why the service node in the POD that has resumed normal operation is used as a slave service node is that the data in the POD that has resumed normal operation is relatively lagging. Of course, the service node in the POD that has resumed normal operation can also be used as the main service node through data synchronization, and the embodiment of the present invention does not limit this.

[0109] For details, please refer to Figure 4 , assuming that the target service node in the first server node is the current primary service node, and when the POD where the target service node of the second server node is located resumes operation, the target service has two primary service nodes, Figure 4The additional coordinator can decide on the target service node in the first server node and the second server node, and select one of them as the slave service node (for example, select the target service node in the second server node that will resume normal operation as the slave service node), and can reset the Sentinel service according to the decided master service node when the master service node changes, including setting the Sentinel service configuration file and restarting the Sentinel service.

[0110] Please refer to Figure 5 , Figure 5 A schematic diagram of the structure of a service exception handling device provided by the present invention, wherein the service exception handling device is applied to a management node of a cloud platform, and comprises:

[0111] A first identification module 51 is used to identify the container group where each service node of the target service is located at every preset period, wherein the target service is a high-availability distributed service, and the service node includes a single master service node and a plurality of slave service nodes;

[0112] A first judgment module 52 is used to judge whether the first container group of the target service is in a normal operating state. If it is not in a normal operating state, the first switching module 53 is triggered, wherein the first container group is: the container group where the primary service node is located;

[0113] The first switching module 53 is used to use a slave service node of the target service in a normal operating state as a new master service node, and use other service nodes of the target service except the newly determined master service node as slave service nodes.

[0114] Based on the above embodiments:

[0115] As an optional embodiment, the service exception handling device further includes:

[0116] A second judgment module is used to judge whether the main service node in the first container group can work normally if it is in a normal operating state, and trigger the first switching module if it cannot work normally;

[0117] The first switching module is used to use a slave service node of the target service in a normal operating state as a new master service node, and use other service nodes of the target service except the newly determined master service node as slave service nodes.

[0118] As an optional embodiment, the second judgment module includes:

[0119] A first sending module is used to send an assistance determination request to the client, wherein the assistance determination request is used to request assistance in determining whether the primary service node in the first container group can work normally;

[0120] The first receiving module is configured to receive an assisted judgment result fed back by the client, wherein the assisted judgment result includes whether the primary service node in the first container group can work normally or cannot work normally.

[0121] As an optional embodiment, the service exception handling device further includes:

[0122] The third judgment module is used to judge whether there is a faulty container group in the target service, and if so, trigger the first push module, wherein the faulty container group is: a container group in an abnormal operating state;

[0123] The first pushing module is used to push the fault information of the faulty container group.

[0124] As an optional embodiment, the target service is in sentinel mode;

[0125] The service exception handling device also includes:

[0126] The first action module is used to use the network protocol address of the container group where the newly determined main service node is located as the network protocol address of the container group where the main service node is located in the sentinel service configuration file, wherein the sentinel configuration file refers to the configuration file of the sentinel service corresponding to the target service;

[0127] The second action module is used to control the restart of the sentinel service corresponding to the target service;

[0128] The fourth judgment module is used to judge whether the main network protocol address pointed to in the sentinel service is a preset initial address after the sentinel service corresponding to the target service is restarted, and if it is the preset initial address, the third action module is triggered, and if it is not the preset initial address, the process ends, wherein the main network protocol address is: the network protocol address of the main service node;

[0129] The third action module is used to replace the main network protocol address pointed to in the sentinel service with the network protocol address of the container group where the main service node is located in the sentinel service configuration file.

[0130] For an introduction to the service exception handling device provided in the embodiment of the present invention, please refer to the embodiment of the service exception handling method described above, and the embodiment of the present invention will not be described in detail here.

[0131] Please refer to Figure 6 , Figure 6 A schematic diagram of the structure of a service exception handling device provided by the present invention, the service exception handling device comprises:

[0132] A memory 61, used for storing computer programs;

[0133] The processor 62 is used to implement the steps of the service exception handling method in the above-mentioned embodiment when executing the computer program.

[0134] For an introduction to the service exception handling device provided in the embodiment of the present invention, please refer to the aforementioned embodiment of the service exception handling method, and the embodiment of the present invention will not be described in detail here.

[0135] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer-readable storage medium provided by the present invention. A computer program 72 is stored on the computer-readable storage medium 71. When the computer program 72 is executed by the processor, the steps of the service exception handling method in the aforementioned embodiment are implemented.

[0136] For an introduction to the computer-readable storage medium provided in an embodiment of the present invention, please refer to the aforementioned embodiment of the service exception handling method, and the embodiment of the present invention will not be described in detail here.

[0137] The present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the service exception handling method in the above-mentioned embodiment when executed by a processor.

[0138] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the aforementioned embodiment of the service exception handling method, and the embodiment of the present invention will not be described in detail here.

[0139] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the element.

[0140] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A service exception handling method, characterized in that: Management nodes used in cloud platforms include: At every preset period, identify the container group where each service node of the target service is located, wherein the target service is a highly available distributed service, and the service nodes include a single master service node and several slave service nodes; Determine whether the first container group of the target service is in a normal operating state, wherein the first container group is: the container group where the primary service node is located; If it is not in normal operation, a slave service node of the target service in normal operation is used as a new master service node, and other service nodes of the target service except the newly determined master service node are used as slave service nodes.

2. The service exception handling method according to claim 1, characterized in that: After determining whether the first container group of the target service is in a normal operating state, the service exception handling method further includes: If it is in normal operation, determine whether the main service node in the first container group can operate normally; If it cannot work normally, a slave service node of the target service in normal operation will be used as the new master service node, and other service nodes of the target service except the newly determined master service node will be used as slave service nodes.

3. The service exception handling method according to claim 2, characterized in that: Identify the container groups where each service node of the target service is located, including: Determine the stateful replica set of the target service based on the custom resources associated with the target service; The container group where each service node of the target service is located and its container group status are determined from the stateful replica set of the target service, wherein the container group status includes a normal operating state and an abnormal operating state.

4. The service exception handling method according to claim 2, characterized in that: Determining whether the primary service node in the first container group can work normally includes: Sending an assistance determination request to the client, wherein the assistance determination request is used to request assistance in determining whether the primary service node in the first container group can work normally; An assisted judgment result fed back by the client is received, wherein the assisted judgment result includes whether the primary service node in the first container group can work normally or cannot work normally.

5. The service exception handling method according to claim 1, characterized in that: After identifying the container group where each service node of the target service is located at each preset period, the service exception handling method further includes: Determine whether the target service has a faulty container group, wherein the faulty container group is: a container group in an abnormally normal operating state; If so, the fault information of the fault container group is pushed.

6. The service exception handling method according to claim 1, characterized in that: The target service includes a remote dictionary service, and the cloud platform includes a cloud platform to which a Kubernetes container orchestration engine is applied.

7. The service exception handling method according to claim 1, characterized in that: The target service is in sentinel mode; After a slave service node in a normal operating state of the target service is used as a new master service node, and other service nodes of the target service except the newly determined master service node are used as slave service nodes, the service exception handling method further includes: The network protocol address of the container group where the newly determined main service node is located is used as the network protocol address of the container group where the main service node is located in the sentinel service configuration file, where the sentinel configuration file refers to the configuration file of the sentinel service corresponding to the target service; Control the restart of the sentinel service corresponding to the target service; After the sentinel service corresponding to the target service is restarted, it is determined whether the primary network protocol address pointed to in the sentinel service is the preset initial address, wherein the primary network protocol address is: the network protocol address of the primary service node; If it is a preset initial address, replace the primary network protocol address pointed to in the Sentinel service with the network protocol address of the container group where the primary service node is located in the Sentinel service configuration file; If it is not the preset initial address, end.

8. A service exception handling device, characterized in that: Management nodes used in cloud platforms include: A first identification module is used to identify, at every preset period, a container group where each service node of a target service is located, wherein the target service is a highly available distributed service, and the service nodes include a single master service node and a plurality of slave service nodes; A first judgment module is used to judge whether the first container group of the target service is in a normal operating state. If it is not in a normal operating state, the first switching module is triggered, wherein the first container group is: the container group where the primary service node is located; The first switching module is used to use a slave service node of the target service in a normal operating state as a new master service node, and use other service nodes of the target service except the newly determined master service node as slave service nodes.

9. A service exception processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the service exception handling method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the service exception handling method according to any one of claims 1 to 7 are implemented.

Images