Service exception processing method, device and equipment and computer readable storage medium

Abstract

The application discloses a service exception processing method and device, equipment and computer readable storage medium, belong to cloud platform field, take into account (1) sentry service cannot cope with'service node is in container group's failure' situation (2) periodic monitoring can discover the exception of service node is in container group in time, therefore, in the application, through the management node of cloud platform, every interval preset period of time is identified the container group that each service node of target service is in, and when the first container group that the main service node is in does not normally operate, the master-slave switching of service node is carried out (one from service node in normal operation state is regarded as new main service node, and other service nodes except newly determined main service node are regarded as slave service node), can provide target service normally when because the first container group fails, through active switching, target service is recovered in time, the reliability of target service and user experience are improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud platforms, and in particular to a service exception handling method, apparatus, device, and computer-readable storage medium. Background Art

[0002] High-availability distributed services are usually deployed in PODs (Plain Old Docker) on multiple servers in a cloud platform. One of the service nodes in the high-availability distributed service is then set as the master service node, and the remaining service nodes are set as slave service nodes. The master service node provides services, and the slave service nodes serve as redundant backups.

[0003] Although the Sentinel service can switch the master service node in some cases to ensure the normal operation of the high-availability distributed service, the master-slave switching through the Sentinel service can only cope with limited unexpected situations. The master-slave switching of the high-availability distributed service can only be achieved in a few unexpected situations, which reduces the reliability and user experience of the high-availability distributed service.

[0004] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. Summary of the Invention

[0005] The present invention aims to provide a service exception handling method, apparatus, device, and computer-readable storage medium. In the present invention, a cloud platform management node identifies, at preset intervals, the container groups containing each service node of a target service. When a first container group containing a master service node fails to operate normally, a master-slave switching of the service nodes is performed (a slave service node in a normally operating state is designated as the new master service node, and all service nodes other than the newly designated master service node are designated as slave service nodes). This method enables timely restoration of the target service through proactive switching when the target service cannot be provided normally due to a failure of the first container group, thereby improving the reliability of the target service and user experience.

[0006] To solve the above technical problems, the present invention provides a service exception handling method, which is applied to the management node of the cloud platform, comprising:

[0007] At every preset period, identify the container group where each service node of the target service is located, where the target service is a highly available distributed service, and the service nodes include a single master service node and several slave service nodes;

[0008] Determine whether the first container group of the target service is in a normal operating state, where the first container group is: the container group where the primary service node is located;

[0009] If not, a slave service node in normal operation state of the target service is taken as a new master service node, and other service nodes of the target service except the newly determined master service node are taken as slave service nodes.

[0010] On the other hand, after judging whether the first container group of the target service is in normal operation state, the service exception processing method further comprises:

[0011] If in normal operation state, whether the master service node in the first container group can work normally is judged.

[0012] If not, a slave service node in normal operation state of the target service is taken as a new master service node, and other service nodes of the target service except the newly determined master service node are taken as slave service nodes.

[0013] On the other hand, identifying the container groups in which the service nodes of the target service are located comprises:

[0014] Determining the stateful replica set of the target service according to the custom resource related to the target service;

[0015] Determining the container groups in which the service nodes of the target service are located and their container group states from the stateful replica set of the target service, wherein the container group states include normal operation state and non-normal operation state.

[0016] On the other hand, whether the master service node in the first container group can work normally comprises:

[0017] Issuing a request for assistance in judgment to a client, wherein the request for assistance in judgment is used to request assistance in judging whether the master service node in the first container group can work normally;

[0018] Receiving the assistance in judgment result fed back by the client, wherein the assistance in judgment result includes that the master service node in the first container group can work normally or not.

[0019] On the other hand, after identifying the container groups in which the service nodes of the target service are located every preset period, the service exception processing method further comprises:

[0020] Judging whether the target service has a faulty container group, wherein the faulty container group is a container group in non-normal operation state;

[0021] If yes, the fault information of the faulty container group is pushed.

[0022] On the other hand, the target service comprises a remote dictionary service, and the cloud platform comprises a cloud platform to which a Kubernete container orchestration engine is applied.

[0023] In another aspect, the target service is in a sentinel mode;

[0024] After one of the service nodes in which the target service is in a normal running state is taken as a new master service node and other service nodes of the target service except the newly determined master service node are taken as slave service nodes, the service exception processing method further comprises:

[0025] The network protocol address of the container group in which the newly determined master service node is located is taken as the network protocol address of the container group in which the master service node is located in the sentinel service configuration file, wherein the sentinel configuration file refers to the configuration file of the sentinel service corresponding to the target service;

[0026] The sentinel service corresponding to the target service is restarted;

[0027] After the sentinel service corresponding to the target service is restarted, it is judged whether the master network protocol address pointed to in the sentinel service is a preset initial address, wherein the master network protocol address is the network protocol address of the master service node;

[0028] If the master network protocol address is the preset initial address, the master network protocol address pointed to in the sentinel service is replaced with the network protocol address of the container group in which the master service node is located in the sentinel service configuration file;

[0029] If the master network protocol address is not the preset initial address, the process is ended.

[0030] To solve the above technical problems, the application further provides a service exception processing device applied to a management node of a cloud platform, comprising:

[0031] A first identification module is configured to identify a container group in which each service node of a target service is located every preset period, wherein the target service belongs to a high-availability distributed service, and the service nodes include a single master service node and a plurality of slave service nodes;

[0032] A first judgment module is configured to judge whether a first container group of the target service is in a normal running state, and if not, trigger a first switching module, wherein the first container group is a container group in which the master service node is located;

[0033] The first switching module is configured to take one of the service nodes in which the target service is in a normal running state as a new master service node, and take other service nodes of the target service except the newly determined master service node as slave service nodes.

[0034] To solve the above technical problems, the application further provides a service exception processing device, comprising:

[0035] A memory is configured to store a computer program;

[0036] A processor is configured to implement the steps of the service exception processing method when executing the computer program.

[0037] To solve the above technical problems, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is configured to implement the steps of the service exception processing method when executed by a processor.

[0038] Beneficial effects: The application provides a service exception processing method, which considers that (1) the sentinel service cannot cope with the failure of the container group where the service node is located, and (2) the periodic monitoring can discover the exception of the container group where the service node is located in time. Therefore, in the application, the management node of the cloud platform identifies the container groups where the service nodes of the target service are located every preset period, and when the first container group where the master service node is located does not operate normally, the master-slave switching of the service nodes is performed (one slave service node in a normal operating state is used as a new master service node, and the other service nodes except the newly determined master service node are used as slave service nodes). In this way, when the target service cannot be normally provided due to the failure of the first container group, the target service can be recovered in time through active switching, and the reliability of the target service and the user experience are improved.

[0039] The application further provides a service exception processing device, equipment and computer readable storage medium, which have the same beneficial effects as the service exception processing method. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the application, the related technologies and the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 A flowchart of a service exception processing method provided by the application;

[0042] Figure 2 A structural diagram of a service exception processing system provided by the application;

[0043] Figure 3 A distribution diagram of a target service provided by the application;

[0044] Figure 4 A structural diagram of another service exception processing system provided by the application;

[0045] Figure 5 A structural diagram of a service exception processing device provided by the application;

[0046] Figure 6 A structural schematic diagram of a service exception processing device provided by the present application is shown in the figure.

[0047] Figure 7 A structural schematic diagram of a computer readable storage medium provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0048] The core of the present application is to provide a service exception processing method, device, equipment and computer readable storage medium, in the present application, through the management node of the cloud platform, each container group in which the service nodes of the target service are identified every preset period, and when the first container group in which the master service node is not normally running, the master-slave switching of the service node is performed (one of the slave service nodes in the normal running state is taken as a new master service node, and the other service nodes except the newly determined master service node are taken as slave service nodes), which can actively switch and restore the target service in time when the target service cannot be normally provided due to the failure of the first container group, and improve the reliability of the target service and the user experience.

[0049] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0050] Please refer to Figure 1 , Figure 1 A flowchart of a service exception processing method provided by the present application is shown in the figure, the service exception processing method is applied to the management node of the cloud platform, and includes:

[0051] S101: every preset period, identify the container group in which the service nodes of the target service are located, wherein the target service belongs to a high-availability distributed service, and the service nodes include a single master service node and a plurality of slave service nodes;

[0052] Specifically, considering the technical problems in the background art above, and in combination with the consideration that (1) the sentinel service cannot cope with the case of "failure of the container group where the service node is located" and (2) periodic monitoring can timely discover the abnormality of the container group where the service node is located, in the embodiment of the present application, the "failure of the container group where the service node is located" of the target service is periodically monitored, so as to timely discover the "failure of the container group where the service node is located" and actively switch the master and slave service nodes, thereby making up for the deficiency of the sentinel service and improving the reliability of the high-availability distributed service. Therefore, the service exception processing method in this step can be applied to the management node of the cloud platform, and every preset period, the container groups where the service nodes of the target service are located are identified and used as the data basis for the subsequent steps.

[0053] Specifically, in order to better illustrate the embodiment of the present application, please refer to Figure 2 , Figure 2 The structure diagram of a service exception processing system provided by the present application is shown in Figure 2 From the right side, it can be seen that the target service nodes (i.e. the service nodes of the target service) are distributed on three server nodes of the cloud platform, i.e. the first to third server nodes each have a target service node, and one of the target service nodes can be designated as a master service node. Correspondingly, when the sentinel mode is applied, a plurality of (odd number) sentinel service nodes (service nodes of the sentinel service) can also be set in the cloud platform, and the sentinel service nodes can usually be distributed on the server nodes where the target service nodes are located.

[0054] The preset period can be set autonomously, for example, it can be set according to the interruption time requirement of the business layer to meet the interruption time requirement of the business layer, for example, it can be 10 seconds, etc. The embodiment of the present application does not limit this.

[0055] S102: determining whether the first container group of the target service is in a normal running state, wherein the first container group is the container group where the master service node is located;

[0056] Specifically, since the failure of the first container group will directly cause the master service node of the target service to be unable to provide services, thereby causing business interruption, in this step, the first container group is focused on, i.e. whether the first container group of the target service is in a normal running state is determined, so as to trigger the action of the subsequent step according to the determination result.

[0057] For any container group, it can be in an abnormal running state due to various reasons, for example, the server node where the container group is located is powered off or disconnected from the network (due to the abnormal pulling out of the network cable or the failure of the network card, etc.), etc. The embodiment of the present application does not limit this.

[0058] S103: If not in normal operation state, one of the service nodes in normal operation state of the target service is taken as a new master service node, and other service nodes of the target service except the newly determined master service node are taken as slave service nodes.

[0059] Specifically, in order to make the target service normally provide services due to the "first container group failure", in this step, when the first container group is not in normal operation state, one of the service nodes in normal operation state of the target service is taken as a new master service node, and other service nodes of the target service except the newly determined master service node are taken as slave service nodes, for example, in Figure 2 In the embodiment, it is assumed that the target service node in the first server node is taken as a master service node and is in an abnormal operation state, at this time, if the target service nodes in the second server node and the third server node are in normal operation state, one of the slave service nodes can be selected as a master service node of the target service according to a preset selection rule, for example, the target service node in the server node with a smaller serial number can be taken as a master service node of the target service, and then each of the other service nodes of the target service is taken as a slave node of the newly determined master service node, etc., which is not limited in the embodiment.

[0060] In the embodiment, when it is determined whether the service node is in normal operation state, the running state of the container group where the service node is located can be taken as the running state of the service node, or the real running state of the service node can be directly determined by other means, which is not limited in the embodiment.

[0061] The application provides a service exception processing method, considering that (1) the sentinel service cannot cope with the "failure of the container group where the service node is located" and (2) the periodic monitoring can timely find the abnormality of the container group where the service node is located, therefore, in the application, the management node of the cloud platform identifies the container groups where each service node of the target service is located every preset period, and when the first container group where the master service node is located is not in normal operation state, the master-slave switching of the service node is performed (one of the service nodes in normal operation state of the target service is taken as a new master service node, and other service nodes except the newly determined master service node are taken as slave service nodes), which can timely restore the target service by actively switching when the target service cannot normally provide services due to the failure of the first container group, and improves the reliability of the target service and the user experience.

[0062] On the basis of the above embodiment:

[0063] As an optional embodiment, after it is judged whether the first container group of the target service is in normal operation state, the service exception processing method further comprises:

[0064] If in normal operation state, it is judged whether the master service node in the first container group can work normally;

[0065] If not, a slave service node in normal operation state of the target service is taken as a new master service node, and other service nodes of the target service except the newly determined master service node are taken as slave service nodes.

[0066] Specifically, considering that in some cases (for example, container group network disconnection), the container group can still show a normal operation state, but the container group cannot be accessed inside, if the master service node is set inside the container group, the target service cannot be provided in this case, therefore, in order to recover the normal operation of the target service in time in this case, even if the first container group is in a normal operation state, it can be judged whether the master service node in the first container group can work normally, if not, the switching of master and slave service nodes can be performed, that is, a slave service node in normal operation state of the target service is taken as a new master service node, and other service nodes of the target service except the newly determined master service node are taken as slave service nodes, so as to recover the normal operation of the target service.

[0067] Among them, the reasons for the disconnection of the container group network can be various, for example, can include network configuration error in the container group, container group port conflict and traffic anomaly, etc., which are not limited in the embodiment of the application.

[0068] As an optional embodiment, identifying the container groups in which the service nodes of the target service are located comprises:

[0069] Determining the stateful replica set of the target service according to the custom resource related to the target service;

[0070] Determining the container groups in which the service nodes of the target service are located and the container group states thereof from the stateful replica set of the target service, wherein the container group states include normal operation state and non-normal operation state.

[0071] Specifically, considering that the PODs and the states of the PODs in which the service nodes of the target service are located can be efficiently and accurately queried through the stateful replica set (Statefulset resource) of the target service, and the stateful replica set of the target service can be determined through the CR (Custom Resource, custom resource) related to the target service, therefore, in the embodiment of the application, the stateful replica set of the target service can be determined according to the custom resource related to the target service, and then the container groups in which the service nodes of the target service are located and the container group states thereof are determined from the stateful replica set of the target service, so as to be taken as the data basis of the subsequent steps.

[0072] Specifically, in addition to the container group state, it can also be determined from the stateful replica set that the service nodes in each container group where the service nodes of the target service are located are master nodes or slave nodes.

[0073] In the CR of the target service, the characteristic information of each service node of the target service can be included, such as the service node name, the server node name, the CPU resource range, the memory resource range, etc., which is not limited in the embodiment of the application.

[0074] Of course, in addition to this way, there can be other ways to identify the container group where each service node of the target service is located, which is not limited in the embodiment of the application.

[0075] Specifically, in Figure 2 In this way, the management node of the cloud platform can determine the POD where each service node is located according to the custom resource.

[0076] As an optional embodiment, determining whether the master service node in the first container group can work normally includes:

[0077] Sending a request for assistance in judgment to the client, wherein the request for assistance in judgment is used to request assistance in determining whether the master service node in the first container group can work normally;

[0078] Receiving the assistance in judgment result fed back by the client, wherein the assistance in judgment result includes that the master service node in the first container group can work normally or cannot work normally.

[0079] Specifically, considering that whether the client can normally use the target service can be used to determine whether the master service node in the first container group can work normally, in order to efficiently and accurately determine, the client can be invited to assist in the judgment in the embodiment of the application, that is, a request for assistance in judgment can be sent to the client, so that the client can assist in determining whether the master service node in the first container group can work normally or cannot work normally by detecting whether it can normally use the target service, and then feed back the assistance in judgment result to the management node of the cloud platform. For the management node of the cloud platform, the assistance in judgment result fed back by the client can be received, so as to efficiently and accurately determine whether the master service node in the first container group can work normally.

[0080] Of course, in addition to this specific way, other ways can also be used to determine whether the master service node in the first container group can work normally, which is not limited in the embodiment of the application.

[0081] As an optional embodiment, after identifying the container group where each service node of the target service is located every preset period, the service exception processing method further includes:

[0082] determining whether the target service has a faulty container group, wherein the faulty container group is a container group in an abnormal operating state;

[0083] If yes, push the fault information of the faulty container group.

[0084] Specifically, considering that the service node in the faulty container group should be recovered in time even if it is a slave service node, in order to maintain the high availability of the target service, in order to facilitate the staff to learn the fault of the container group in time and carry out maintenance, the embodiment of the application can push the fault information of the faulty container group when it is determined that the target service has a faulty container group.

[0085] The fault information can include the server node where the faulty container group is located and the POD name, and the embodiment of the application does not limit this.

[0086] As an optional embodiment, the target service includes a remote dictionary service, and the cloud platform includes a cloud platform on which a Kubernetes container orchestration engine is applied.

[0087] Specifically, Redis (remote dictionary service) is an open source Key-Value database written in ANSI C language, supporting network, based on memory and persistent log, and providing APIs in multiple languages. The data of Redis exists in the memory. Its read-write speed is very fast, and it can handle more than 100,000 read-write operations per second.

[0088] Specifically, the cloud platform based on K8S (Kubernetes container orchestration engine) has high maturity and strong reliability.

[0089] Of course, in addition to Redis, the target service can also be of other types, and in addition to the cloud platform on which the Kubernetes container orchestration engine is applied, the cloud platform can also be of other types, and the embodiment of the application does not limit this.

[0090] Specifically, Figure 2 and Figure 4 The additional reconciler in the application is a function module in the management software operator of the cloud platform applying K8S, and the English name is ExtraReconciler. The update function in ExtraReconciler can be used to switch the master-slave service nodes of the target service, and the embodiment of the application can enhance the capability of the operator.

[0091] As an optional embodiment, the target service is in a sentinel mode.

[0092] After the target service is in normal operation state with one of the service nodes as a new master service node and other service nodes of the target service except the newly determined master service node as slave service nodes, the service exception processing method further comprises:

[0093] The network protocol address of the container group in which the newly determined master service node is located is taken as the network protocol address of the container group in which the master service node is located in the sentinel service configuration file, wherein the sentinel configuration file refers to the configuration file of the sentinel service corresponding to the target service;

[0094] The sentinel service corresponding to the target service is controlled to restart;

[0095] After the sentinel service corresponding to the target service is restarted, it is judged whether the master network protocol address pointed to in the sentinel service is a preset initial address, wherein the master network protocol address is the network protocol address of the master service node;

[0096] If the master network protocol address is the preset initial address, the master network protocol address pointed to in the sentinel service is replaced with the network protocol address of the container group in which the master service node is located in the sentinel service configuration file;

[0097] If the master network protocol address is not the preset initial address, the process ends.

[0098] Specifically, in order to better illustrate the embodiments of the present application, please refer to Figure 3 and Figure 4 , Figure 3 a distribution diagram of a target service provided by the present application, Figure 4 a structure diagram of another service exception processing system provided by the present application, Figure 3 The relationship between the sentinel service and the client and the target service is shown in the figure, the client can know the address of the master service node of the target service through the sentinel service to access the target service, and the sentinel service can monitor and failover the target service, Figure 3 The sentinel service in the figure can include three service nodes, and the target service can also include three service nodes.

[0099] Specifically, considering that in the sentinel mode, after the master-slave service node switching of the target service, on the one hand, the sentinel service needs to be restarted, and on the other hand, the IP address of the new master service node of the target service needs to be updated to the sentinel service, and considering that the sentinel service will rebuild the POD where the sentinel service is located after being restarted, the sentinel service may not be able to automatically point to the network protocol address of the container group where the master service node in the sentinel service configuration file is located due to unexpected reasons during the rebuilding process, and a preset initial address (for example, it can be 127.0.0.1) is displayed. Therefore, in the embodiment of the application, the network protocol address of the container group where the newly determined master service node is located can be determined as the network protocol address of the container group where the master service node in the sentinel service configuration file is located, and after the sentinel service is restarted, it is actively judged whether the master network protocol address pointed to in the sentinel service is the preset initial address. If it is the preset initial address, it means that it cannot automatically point to the network protocol address of the container group where the master service node in the sentinel service configuration file is located due to unexpected reasons. Therefore, in the embodiment of the application, the master network protocol address pointed to in the sentinel service can be replaced with the network protocol address of the container group where the master service node in the sentinel service configuration file is located in this case, so as to ensure the normal execution of the sentinel service, also ensure that the client can normally use the target service, and further improve the reliability of the target service.

[0100] Specifically, it is worth mentioning that the sentinel mode itself has no ability to quickly choose a new master service node in the case of sudden disconnection of the master service node of the target service, and the management node of the cloud platform usually only adds and configures the service node for the management of the target service. Therefore, in the embodiment of the application, the management node of the cloud platform can add the master-slave service node switching of the target service in the case of POD failure and other unexpected situations.

[0101] In addition, as an optional embodiment, the target service is in a sentinel mode;

[0102] The service exception processing method further comprises:

[0103] When a second container group that resumes the normal running state appears in the cloud platform, it is judged whether the target service in the cloud platform simultaneously exists two master service nodes, wherein the second container group refers to a container group where a service node of the target service is located;

[0104] If so, the service node in the second container group that resumes the normal running state is taken as a slave service node of the target service;

[0105] When the third container group that resumes the normal running state appears in the cloud platform, it is judged whether the target sentinel service in the cloud platform simultaneously exists two master service nodes, wherein the third container group refers to the container group in which the service node of the target sentinel service is located, and the target sentinel service refers to the sentinel service corresponding to the target service;

[0106] If they exist simultaneously, the service node in the third container group that resumes the normal running state is taken as the slave service node of the target sentinel service.

[0107] Specifically, considering that the faulty POD may resume the normal running state after being repaired after being discovered, and for the faulty POD, when it resumes the normal running state, it will restore the state before the fault, if the faulty POD has a master service node before the fault, then whether the container group of the master service is restored or the container group of the sentinel service is restored, multiple master service nodes will exist in the service, which causes the service to be unable to normally provide, therefore, in order to improve the reliability of the target service, in the embodiment of the application, when the second container group that resumes the normal running state appears in the cloud platform, it is judged whether the target service in the cloud platform simultaneously exists two master service nodes, if they exist simultaneously, the service node in the second container group that resumes the normal running state is taken as the slave service node of the target service; for the sentinel service, when the third container group that resumes the normal running state appears in the cloud platform, it is judged whether the target sentinel service in the cloud platform simultaneously exists two master service nodes, if they exist simultaneously, the service node in the third container group that resumes the normal running state is taken as the slave service node of the target sentinel service; in the embodiment of the application, the master-slave node coordination of the target service and the sentinel service in the faulty POD recovery scenario can be used to further improve the reliability of the target service.

[0108] Wherein, the reason why the service node in the POD that resumes the normal running state is taken as the slave service node is that the data in the POD that resumes the normal running state is relatively lagging, of course, the service node in the POD that resumes the normal running state can also be taken as the master service node through data synchronization, which is not limited in the embodiment of the application.

[0109] Specifically, please refer to Figure 4 , it is assumed that the target service node in the first server node is the current master service node, and when the target service node in the second server node resumes the running state, the target service has two master service nodes, Figure 4The additional coordinator in the cloud platform can make a decision on the first server node and the target service node in the second server node, select one of them as a slave service node (for example, select the target service node in the second server node which will be restored to normal operation as a slave service node), and reset the sentinel service according to the decision of the master service node, including setting the sentinel service configuration file and restarting the sentinel service, etc.

[0110] Please refer to Figure 5 , Figure 5 A structural schematic diagram of a service exception processing device provided by the application is provided, which is applied to a management node of a cloud platform and includes:

[0111] The first identification module 51 is configured to identify a container group where each service node of a target service is located every preset period, wherein the target service belongs to a high-availability distributed service, and the service node includes a single master service node and a plurality of slave service nodes.

[0112] The first judgment module 52 is configured to judge whether a first container group of the target service is in a normal operation state, and if not, trigger the first switching module 53, wherein the first container group is a container group where the master service node is located.

[0113] The first switching module 53 is configured to select a slave service node of the target service in a normal operation state as a new master service node, and select other service nodes of the target service except the newly determined master service node as slave service nodes.

[0114] Based on the above embodiment:

[0115] As an optional embodiment, the service exception processing device further includes:

[0116] The second judgment module is configured to, if in the normal operation state, judge whether the master service node in the first container group can work normally, and if not, trigger the first switching module.

[0117] The first switching module is configured to select a slave service node of the target service in a normal operation state as a new master service node, and select other service nodes of the target service except the newly determined master service node as slave service nodes.

[0118] As an optional embodiment, the second judgment module includes:

[0119] The first sending module is configured to send an assistance judgment request to a client, wherein the assistance judgment request is used to request assistance in judging whether the master service node in the first container group can work normally.

[0120] The first receiving module is configured to receive a client feedback assistance judgment result, wherein the assistance judgment result includes whether the master service node in the first container group can work normally or not.

[0121] As an optional embodiment, the service exception processing apparatus further includes:

[0122] The third judging module is configured to judge whether the target service has a fault container group, and if so, trigger the first pushing module, wherein the fault container group is a container group in an abnormal operating state.

[0123] The first pushing module is configured to push fault information of the fault container group.

[0124] As an optional embodiment, the target service is in a sentinel mode.

[0125] The service exception processing apparatus further includes:

[0126] The first action module is configured to set the network protocol address of the container group in which the newly determined master service node is located as the network protocol address of the container group in which the master service node is located in the sentinel service configuration file, wherein the sentinel configuration file refers to a configuration file of a sentinel service corresponding to the target service.

[0127] The second action module is configured to control the sentinel service corresponding to the target service to restart.

[0128] The fourth judging module is configured to judge whether the master network protocol address pointed to in the sentinel service is a preset initial address after the sentinel service corresponding to the target service restarts, and if so, trigger the third action module, and if not, end, wherein the master network protocol address is the network protocol address of the master service node.

[0129] The third action module is configured to replace the master network protocol address pointed to in the sentinel service with the network protocol address of the container group in which the master service node is located in the sentinel service configuration file.

[0130] The service exception processing apparatus provided by the embodiment of the present application is introduced in the foregoing embodiments of the service exception processing method, and thus will not be described here.

[0131] Please refer to Figure 6 , Figure 6 The service exception processing apparatus provided by the embodiment of the present application is introduced in the foregoing embodiments of the service exception processing method, and thus will not be described here.

[0132] The memory 61 is configured to store a computer program.

[0133] The processor 62 is configured to execute the computer program to realize the steps of the service exception processing method in the foregoing embodiments.

[0134] The service exception processing device provided by the embodiments of the present application is described in the foregoing embodiments of the service exception processing method, and will not be described here again.

[0135] Please refer to Figure 7 , Figure 7 A structural schematic diagram of a computer readable storage medium provided by the present application is shown in the figure, and the computer readable storage medium 71 stores a computer program 72, and the computer program 72 is executed by a processor to realize the steps of the service exception processing method in the foregoing embodiments.

[0136] The computer readable storage medium provided by the embodiments of the present application is described in the foregoing embodiments of the service exception processing method, and will not be described here again.

[0137] The present application further provides a computer program product, comprising computer programs / instructions, which are executed by a processor to realize the steps of the service exception processing method in the foregoing embodiments.

[0138] The computer program product provided by the embodiments of the present application is described in the foregoing embodiments of the service exception processing method, and will not be described here again.

[0139] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part. It should be further explained that, in the specification, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or sequence between the entities or operations. Moreover, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to the process, method, article or device. Without more limitations, the element defined by the statement “including a…” does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0140] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for handling service exceptions, characterized in that: Management nodes used in cloud platforms include: At every preset period, identify the container group where each service node of the target service is located, where the target service is a highly available distributed service, and the service nodes include a single master service node and several slave service nodes; Determine whether the first container group of the target service is in a normal operating state, where the first container group is: the container group where the primary service node is located; If it is not in normal operation, a slave service node of the target service that is in normal operation will be used as the new master service node, and all other service nodes of the target service except the newly determined master service node will be used as slave service nodes; The target service is in sentinel mode; Use the network protocol address of the container group where the newly determined primary service node is located as the network protocol address of the container group where the primary service node is located in the sentinel service configuration file, where the sentinel service configuration file refers to the configuration file of the sentinel service corresponding to the target service; Control the restart of the sentinel service corresponding to the target service; After the sentinel service corresponding to the target service is restarted, determine whether the primary network protocol address pointed to in the sentinel service is the preset initial address, where the primary network protocol address is: the network protocol address of the primary service node; If it is a preset initial address, replace the primary network protocol address pointed to in the Sentinel service with the network protocol address of the container group where the primary service node is located in the Sentinel service configuration file; If it is not the preset initial address, then end.

2. The service exception handling method according to claim 1, wherein: After determining whether the first container group of the target service is in a normal operating state, the service exception handling method further includes: If it is in normal operation, determine whether the main service node in the first container group can operate normally; If it fails to work properly, a slave service node of the target service that is in normal operation will be used as the new master service node, and other service nodes of the target service except the newly determined master service node will be used as slave service nodes.

3. The service exception handling method according to claim 2, wherein: Identify the container groups where each service node of the target service is located, including: Determine the stateful replica set of the target service based on the custom resources associated with the target service; The container group where each service node of the target service is located and its container group status are determined from the stateful replica set of the target service, where the container group status includes a normal operating state and an abnormal operating state.

4. The service exception handling method according to claim 2, wherein: Determining whether the primary service node in the first container group can operate normally includes: Sending a determination assistance request to the client, wherein the determination assistance request is used to request assistance in determining whether the primary service node in the first container group can operate normally; An assisted judgment result fed back by the client is received, where the assisted judgment result includes whether the primary service node in the first container group can work normally or cannot work normally.

5. The service exception handling method according to claim 1, wherein: After identifying the container group where each service node of the target service is located at each preset period, the service exception handling method further includes: Determine whether the target service has a faulty container group, where a faulty container group is a container group that is in an abnormal operating state; If so, the fault information of the faulty container group is pushed.

6. The service exception handling method according to claim 1, wherein: The target service includes a remote dictionary service, and the cloud platform includes a cloud platform using a Kubernetes container orchestration engine.

7. A service exception handling device, characterized in that: Management nodes used in cloud platforms include: A first identification module is configured to identify, at a preset period, a container group where each service node of a target service is located, wherein the target service is a highly available distributed service, and the service nodes include a single master service node and a plurality of slave service nodes; A first determination module is configured to determine whether a first container group of a target service is in a normal operating state, and if not, trigger a first switching module, wherein the first container group is the container group where the primary service node is located; A first switching module is configured to use a slave service node in a normal operating state of the target service as a new master service node, and use other service nodes of the target service except the newly determined master service node as slave service nodes; The target service is in sentinel mode; The service exception handling device also includes: The first action module is configured to use the network protocol address of the container group where the newly determined primary service node is located as the network protocol address of the container group where the primary service node is located in the sentinel service configuration file, wherein the sentinel configuration file refers to the configuration file of the sentinel service corresponding to the target service; The second action module is used to control the restart of the sentinel service corresponding to the target service; The fourth judgment module is used to determine whether the primary network protocol address pointed to in the sentinel service is a preset initial address after the sentinel service corresponding to the target service is restarted. If it is the preset initial address, the third action module is triggered; if it is not the preset initial address, the process ends, where the primary network protocol address is: the network protocol address of the primary service node; The third action module is used to replace the main network protocol address pointed to in the sentinel service with the network protocol address of the container group where the main service node is located in the sentinel service configuration file.

8. A service exception handling device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the service exception handling method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the service exception handling method according to any one of claims 1 to 6.

Images