Business application elegant offline and process self-healing method, electronic equipment and storage medium

Abstract

The invention discloses a business application elegant offline and process self-healing method, electronic equipment and a storage medium aiming at the problems of order loss and process abnormity caused by abnormal exit of a core application, and relates to the field of cloud computation.The method comprises the steps that a business application node cluster is deployed in Kubernetes, and the business application node cluster comprises a master node and N slave nodes; determining the master-slave relationship of the nodes by using the scrambling of the distributed lock; the master node is responsible for monitoring the online state of the service application node and informing other slave nodes in time when an abnormal condition is found; if the main node exits abnormally, triggering the application to be offline gracefully by using a life cycle management mechanism of Kubernetes; and the slave node takes over the role of the master node when the master node is abnormal and continues to execute the original role task. The abnormal process is automatically processed through a closed-loop management mechanism, and the effects that the process can be self-healed, production order operation is not affected, and a user does not have abnormal perception are achieved.

Description

Technical Field

[0001] This invention relates to the field of cloud computing, specifically to a method for gracefully shutting down business applications and self-healing processes, an electronic device, and a storage medium. Background Technology

[0002] Kubernetes, often abbreviated as K8s, is an open-source container orchestration platform. It is used to automate the deployment, scaling, and management of containerized applications. Kubernetes makes it easy to manage large numbers of containers, ensuring their efficient operation and dynamic scaling as needed.

[0003] Redis: An open-source, ANSI C-language-based, network-enabled, in-memory or persistent log-structured key-value database.

[0004] PaaS: PaaS (Platform as a Service) is a cloud computing service model that provides developers with a platform that allows them to build, deploy, and manage applications in a cloud environment without having to worry about the management and maintenance of the underlying infrastructure (such as servers, storage, and networks).

[0005] SaaS (Software as a Service) is a cloud computing service model. In the SaaS model, software applications are hosted in the cloud by a service provider and made available to users via the internet. Users do not need to install and maintain the software locally; they only need an internet connection to access and use the applications they need.

[0006] Graceful shutdown, also known as lossless shutdown, refers to executing a series of pre-defined, orderly steps when stopping an application to ensure that ongoing user operations are not affected, no critical data is lost, and the impact on other services that depend on it is minimized.

[0007] In the cloud computing field, cloud-native architecture has become the mainstream approach for building and running applications. It emphasizes leveraging the elasticity, scalability, and automated management features of cloud services to improve application deployment efficiency and operational reliability. Furthermore, various applications provide container lifecycle management capabilities through a unified platform. Business provisioning systems exist within digital solutions across various sectors of society; process orchestration and business logic execution are typically supported by business provisioning IT systems.

[0008] Currently, most business activation and orchestration systems are designed around middleware such as relational databases, caching services, and message queues to handle various work order processes and intermediate state-driven events. The overall process is: the application side consumes events, executes business logic, receives a receipt, and triggers the next step.

[0009] Existing technical solutions only address data transaction isolation, caching service atomicity, and message queue consumption offset commit strategies, aiming to minimize message loss or duplicate message consumption based on business requirements. For example:

[0010] (1) Use persistent row data in the database for event logging:

[0011] Mainstream event scheduling frameworks for dependency-based database models, such as Quartz, offer powerful capabilities for planning and triggering scheduled tasks, as well as flexible configurability. Business systems based on the Quartz scheduling framework typically employ a customized process and event model. All event reception, execution, rollback, and termination are recorded in the database as event instance data. When a work order enters the next stage, the process status is updated, and the application then identifies the process status and performs business processing in a thread.

[0012] (2) Utilizing message queue features:

[0013] While global scheduling frameworks like Quartz and XXL-JOB offer rich task scheduling rules and high reliability, their scheduling models have a learning curve and relatively complex configurations. Many lightweight process-oriented applications utilize message queues for event-driven operations. Different message topics are typically designed for consumption and processing at different stages, triggering business logic processing at each stage. This is equivalent to hosting process events to the message queue. To avoid end-to-end message loss or duplicate executions that could hinder analysis, messages are usually persisted. Almost all message queue technologies offer effective strategies for addressing message loss and duplicate message consumption issues.

[0014] Relying on high-availability middleware is an effective way to improve the overall availability of business system architecture. However, in daily system operation, a large proportion of production failures originate from the business applications themselves. The solutions mentioned above cannot solve the problem of event loss caused by abnormal application exits or restarts.

[0015] Currently, most large-scale business systems on the market tend to use cloud-native architectures. These architectures offer numerous features to provide the flexibility of containerized deployment, rich elastic scaling strategies, fine-grained control of network traffic between Pods, and improved server node resource utilization. However, this also brings the following problems to the applications deployed on the architecture platform:

[0016] (1) Compared to traditional virtual machine deployment, Pod scheduling in containerized deployment becomes more frequent. For example, dynamic scaling strategies based on performance monitoring, affinity and anti-affinity strategies based on Node and Pod objects, and load balancing scheduling strategies involving Node nodes are all designed to provide flexibility in container scheduling. However, this increases the possibility of application Pods being shut down or restarted. For ordinary web applications, the problem can be solved through gateways and Ingress health check mechanisms. However, for work order flow applications, business logic processing is usually performed on the backend. When a running Pod is forcibly scheduled and restarted, it often leads to the interruption of business logic that has not been completed by the thread, resulting in work order blockage. Generally, only operation and maintenance methods and external monitoring methods can be used to handle this, and the lost business process cannot be automatically closed.

[0017] (2) If the abnormal Kubernetes scheduling means cause the abnormal interruption of the application life cycle, such as node failure, storage failure, resource over-provisioning, K8S control plane failure, etc., the Pod running on certain nodes will be unexpectedly terminated. At this time, the terminated Pod has some or even a large number of business events in its memory. The loss of these events will lead to a large number of production order blockages. This situation is unacceptable for many production core business work order flow systems.

[0018] (3) The clustering of general business applications is stateless, which makes deployment and maintenance simple, horizontal scaling more flexible, easier to migrate and elastically deploy, and better able to adapt to constantly changing business needs and complex operating environments. However, this also limits their lifecycle, which only has the lifecycle of system integration and cannot be effectively associated with events lost in memory when the application exits or is scheduled to restart, making them untraceable. Summary of the Invention

[0019] This invention addresses the shortcomings of existing technologies by providing a method, electronic device, and storage medium for graceful business application shutdown and process self-healing. It primarily solves problems faced by business activation and process orchestration systems, namely, order loss and process anomalies caused by the abnormal exit of core applications, which previously required manual intervention for data analysis and processing for repair. This invention automatically handles abnormal processes through a closed-loop management mechanism, achieving self-healing processes, uninterrupted production order operation, and seamless user experience.

[0020] To achieve the above objectives, the present invention adopts the following technical solution:

[0021] A method for gracefully shutting down business applications and self-healing processes is implemented on Kubernetes. A business application node cluster is deployed on Kubernetes, which includes 1 master node and N slave nodes. The master-slave relationship of the nodes is confirmed by contention using a distributed lock.

[0022] The master node is responsible for monitoring the online status of business application nodes and promptly notifying other slave nodes if any abnormalities are detected.

[0023] If the master node exits abnormally, the application is gracefully shut down using Kubernetes' lifecycle management mechanism; when the master node fails, the slave node takes over its role and continues to execute the original role's tasks.

[0024] To optimize the above technical solution, the specific measures also include:

[0025] Furthermore, the master-slave relationship of the nodes used to confirm contention using distributed locks is specifically as follows:

[0026] Use Redis as a distributed caching service to store application heartbeats and message backups;

[0027] When a business application node starts up, it registers its unique ID and node information with a specific hash in Redis. The master-slave relationship of the node is confirmed by contention for a distributed lock. The node that wins the lock becomes the master node, and the node that wins the lock becomes the slave node. At the same time, when a business application node starts up, a daemon thread is created to send heartbeats at regular intervals and update the heartbeat sending time in the hash. The heartbeat represents the online status of the business application node.

[0028] After a business application node consumes a message and reads an event, it immediately writes the message ID or event ID into a list queue in Redis that is bound to the node's unique ID. This list queue represents the message backup queue of the current business application node. After the event ends, the event ID is removed from the message backup queue.

[0029] Furthermore, the master node is responsible for monitoring the online status of the business application nodes, specifically as follows:

[0030] The master node periodically checks whether the heartbeats of the business application nodes in the hash have timed out. If the heartbeat of a slave node times out, it considers that the slave node has gone offline. Based on the unique ID of the slave node, it checks the message backup queue of the slave node. If the queue depth of the backup queue is greater than 0, it means that these slave nodes have failed to gracefully go offline. At this time, the master node reorganizes the event IDs of the message backup queue and then distributes them to the message consumption source of other currently online slave nodes to complete the event redistribution.

[0031] Furthermore, the specific steps for triggering graceful application shutdown using Kubernetes' lifecycle management mechanism are as follows:

[0032] At each stage of the application lifecycle, Kubernetes provides operations that actively trigger interaction with Pods to achieve pre-offline operations before application process restart or exit, and automatically trigger the offline operation after the execution timeout.

[0033] The pre-offline operation specifically involves:

[0034] Shutting off incoming traffic includes closing the traffic consumer instance and ending the thread object that receives traffic, notifying the process processing thread that it is about to go offline, giving events that are being processed in the application memory a certain amount of time, and trying to complete the execution of all events before the container process completely exits.

[0035] Furthermore, the slave node taking over the role of the master node and continuing to execute the original role's tasks when the master node encounters an anomaly specifically involves:

[0036] When the master node exits abnormally, its lock will not be renewed. When the lock expires, a slave node will acquire the lock and become the master node. It will then execute the tasks of the original master node. The new master node will reorganize the event IDs in the message backup queue and distribute them to the message consumption sources of other currently online slave nodes to complete event redistribution and continue to execute the events that were not completed when the node was scheduled to go offline.

[0037] Furthermore, the method also includes: using message queues to achieve asynchronous communication and event-driven data synchronization between business applications.

[0038] Furthermore, the method also includes:

[0039] Employ continuous integration and continuous delivery pipelines to enable business application image building and business application upgrades.

[0040] Furthermore, the method also includes: using a service mesh to implement global service governance, traffic management, and secure connectivity in the control plane.

[0041] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the graceful shutdown of business applications and the process self-healing method described above.

[0042] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute the graceful shutdown and process self-healing methods for business applications as described above.

[0043] The beneficial effects of this invention are as follows: This invention utilizes distributed caching for backup of events executed in application memory, featuring multiple caching mechanisms. For stateless business applications, a heartbeat registration mechanism and a master-slave relationship mechanism are introduced to complete backup, backup deletion, and backup recovery of offline nodes in a closed loop. Through a closed-loop management mechanism, abnormal processes are automatically handled, achieving self-healing processes, unaffected production order operation, and seamless user experience. Attached Figure Description

[0044] Figure 1 This is an overall flowchart of the graceful shutdown and process self-healing method for business applications proposed in this invention.

[0045] Figure 2 This is an architecture diagram of the service activation system proposed in this invention. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0047] Example 1

[0048] This invention proposes a method for graceful shutdown of business applications and self-healing of processes, such as... Figure 1 As shown, the implementation is based on Kubernetes, with a cluster of business application nodes deployed on Kubernetes (K8S) to ensure system scalability and load capacity, as well as a more comprehensive container lifecycle management strategy. For systems with complex microservice architectures, a service mesh (such as Istio or Linkerd) can be introduced to achieve global service governance, traffic management, and secure connections in the control plane. The core business applications in this embodiment include a unified order receiving application, a work order decomposition application, a work order interaction application, and an order confirmation application.

[0049] The business application node cluster consists of 1 master node and N slave nodes; the master-slave relationship of the nodes is confirmed using a distributed lock contention mechanism; the specific process of confirming the master-slave relationship of the nodes using a distributed lock contention mechanism is as follows:

[0050] Use Redis as a distributed caching service to store application heartbeats and message backups;

[0051] The business application integrates an SDK. When a business application node starts up, it registers its unique ID and node information with a specific hash in Redis. The master-slave relationship of the node is confirmed by the contention of a distributed lock. The node that wins the lock becomes the master node, and the node that wins the lock becomes the slave node. At the same time, when a business application node starts up, a daemon thread is created to send heartbeats at regular intervals and update the heartbeat sending time in the hash. The heartbeat represents the online status of the business application node.

[0052] After a business application node consumes a message and reads an event, it immediately writes the message ID or event ID to a list queue in Redis that is bound to the node's unique ID. This list queue represents the message backup queue of the current business application node. After the event ends, the event ID is removed from the message backup queue. The entire process is wrapped in Lua scripts to ensure the atomicity of the overall logic.

[0053] The master node is responsible for monitoring the online status of the business application nodes and promptly notifying other slave nodes of any anomalies. The specific process by which the master node monitors the online status of the business application nodes is as follows:

[0054] The master node periodically checks whether the heartbeats of the business application nodes in the hash have timed out. If the heartbeat of a slave node times out, it considers that the slave node has gone offline. Based on the unique ID of the slave node, it checks the message backup queue of the slave node. If the queue depth of the backup queue is greater than 0, it means that these slave nodes have failed to gracefully go offline. At this time, the master node reorganizes the event IDs of the message backup queue and then distributes them to the message consumption source of other currently online slave nodes to complete the event redistribution.

[0055] If the master node exits abnormally, the application is gracefully decommissioned using Kubernetes' lifecycle management mechanisms (such as the PreStop hook). Slave nodes take over the master node's role when it fails, continuing to execute the tasks of their original roles. The specific process of triggering graceful application decommissioning using Kubernetes' lifecycle management mechanisms is as follows:

[0056] At various stages of the application lifecycle, such as Pending, Running, Successful, and Failed, Kubernetes provides operations (Hooks) to actively trigger interactions with Pods, enabling pre-shutdown operations before application process restarts or exits. This achieves automatic triggering of shutdown operations after a timeout. The application provides internal pre-shutdown APIs such as / preShutDown, which enable functions such as traffic shutdown and setting internal thread exit flags.

[0057] Deploy an application cluster based on Kubernetes, configure the PreStop hook method to trigger pre-shutdown API calls, and configure the graceful shutdown tolerance time.

[0058] lifecycle:

[0059] preStop:

[0060] httpGet:

[0061] path: / preShutDown

[0062] port:8080

[0063] scheme:HTTP

[0064] TerminationGracePeriodSeconds:30

[0065] The specific process of the pre-offline operation is as follows:

[0066] Shutting off incoming traffic includes closing the traffic consumer instance and ending the thread object that receives traffic, notifying the process processing thread that it is about to go offline, giving events that are being processed in the application memory a certain amount of time, and trying to complete the execution of all events before the container process completely exits.

[0067] The specific process by which a slave node takes over the role of the master node and continues to execute the original role's tasks when the master node encounters an error is as follows:

[0068] When the master node exits abnormally, its lock will not be renewed. When the lock expires, a slave node will acquire the lock and become the master node. It will then execute the tasks of the original master node. The new master node will reorganize the event IDs in the message backup queue and distribute them to the message consumption sources of other currently online slave nodes to complete event redistribution and continue to execute the events that were not completed when the node was scheduled to go offline.

[0069] To achieve asynchronous communication and event-driven data synchronization between business applications, this embodiment can adopt an event-driven architecture, which can be implemented using message queues (such as RabbitMQ or Kafka). In this way, this application can easily push backup events of offline nodes to the message source to achieve real-time data synchronization and fault tolerance.

[0070] It is necessary to ensure that the business applications of this application remain compatible with the latest version of the cloud-native ecosystem. To this end, this application may adopt a continuous integration and continuous delivery (CI / CD) pipeline to achieve one-click image building and application upgrades.

[0071] Example 2

[0072] This invention proposes a service activation system that applies the method described in Embodiment 1. The system architecture diagram is as follows. Figure 2As shown, the core business applications in the system include a unified order receiving application, a work order decomposition application, a work order interaction application, and an order confirmation application. These applications act as schedulers and participants in the business activation process, exchanging events with each other via message queues to complete the entire lifecycle of a business activation order. All these applications have improved overall application availability using the method described in Example 1, ensuring no order events are lost even in various failure scenarios, and immediately restoring all in-transit business after application startup.

[0073] Example 3

[0074] This invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the graceful shutdown and process self-healing method for business applications as described in Embodiment 1.

[0075] Example 4

[0076] This invention proposes a computer-readable storage medium storing a computer program that enables a computer to execute the graceful shutdown and process self-healing method for business applications as described in Embodiment 1.

[0077] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0078] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0079] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for gracefully shutting down business applications and achieving process self-healing, implemented on Kubernetes, wherein a cluster of business application nodes is deployed on Kubernetes, characterized by: The business application node cluster includes 1 master node and N slave nodes; the master-slave relationship of the nodes is confirmed by contention using a distributed lock; The master node is responsible for monitoring the online status of business application nodes and promptly notifying other slave nodes if any abnormalities are detected. If the master node exits abnormally, the application is gracefully shut down using Kubernetes' lifecycle management mechanism; when the master node fails, the slave node takes over its role and continues to execute the original role's tasks.

2. The method for gracefully decommissioning business applications and self-healing processes as described in claim 1, characterized in that, The specific method for confirming the master-slave relationship of nodes using distributed locks is as follows: Use Redis as a distributed caching service to store application heartbeats and message backups; When a business application node starts up, it registers its unique ID and node information with a specific hash in Redis. The master-slave relationship of the node is confirmed by contention for a distributed lock. The node that wins the lock becomes the master node, and the node that wins the lock becomes the slave node. At the same time, when a business application node starts up, a daemon thread is created to send heartbeats at regular intervals and update the heartbeat sending time in the hash. The heartbeat represents the online status of the business application node. After a business application node consumes a message and reads an event, it immediately writes the message ID or event ID into a list queue in Redis that is bound to the node's unique ID. This list queue represents the message backup queue of the current business application node. After the event ends, the event ID is removed from the message backup queue.

3. The method for gracefully decommissioning business applications and self-healing processes as described in claim 1, characterized in that, The master node is responsible for monitoring the online status of the business application nodes, specifically as follows: The master node periodically checks whether the heartbeats of the business application nodes in the hash have timed out. If the heartbeat of a slave node times out, it considers that the slave node has gone offline. Based on the unique ID of the slave node, it checks the message backup queue of the slave node. If the queue depth of the backup queue is greater than 0, it means that these slave nodes have failed to gracefully go offline. At this time, the master node reorganizes the event IDs of the message backup queue and then distributes them to the message consumption source of other currently online slave nodes to complete the event redistribution.

4. The method for gracefully decommissioning business applications and self-healing processes as described in claim 1, characterized in that, The specific steps for triggering graceful application shutdown using Kubernetes' lifecycle management mechanism are as follows: At each stage of the application lifecycle, Kubernetes provides operations that actively trigger interaction with Pods to achieve pre-offline operations before application process restart or exit, and automatically trigger the offline operation after the execution timeout. The pre-offline operation specifically involves: Shutting off incoming traffic includes closing the traffic consumer instance and ending the thread object that receives traffic, notifying the process processing thread that it is about to go offline, giving events that are being processed in the application memory a certain amount of time, and trying to complete the execution of all events before the container process completely exits.

5. The method for gracefully decommissioning business applications and self-healing processes as described in claim 1, characterized in that, The slave node takes over the role of the master node and continues to perform the original role's tasks when the master node encounters an anomaly, specifically as follows: When the master node exits abnormally, its lock will not be renewed. When the lock expires, a slave node will acquire the lock and become the master node. It will then execute the tasks of the original master node. The new master node will reorganize the event IDs in the message backup queue and distribute them to the message consumption sources of other currently online slave nodes to complete event redistribution and continue to execute the events that were not completed when the node was scheduled to go offline.

6. The method for gracefully decommissioning business applications and self-healing processes as described in claim 1, characterized in that, The method also includes using message queues to achieve asynchronous communication and event-driven data synchronization between business applications.

7. The method for gracefully decommissioning business applications and self-healing processes as described in claim 1, characterized in that, The method further includes: Employ continuous integration and continuous delivery pipelines to enable business application image building and business application upgrades.

8. The method for gracefully decommissioning business applications and self-healing processes as described in claim 1, characterized in that, The method also includes using a service mesh to achieve global service governance, traffic management, and secure connectivity in the control plane.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the graceful shutdown and process self-healing method for business applications as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to execute the graceful shutdown and process self-healing method for business applications as described in any one of claims 1-8.

Images