Service recovery method and device, electronic equipment and storage medium
By monitoring the Kubernetes cluster status in real time and generating service recovery instructions, quickly finding and restoring the target service backup set, the problem of large data recovery resource occupancy in the existing technology is solved, and service recovery efficiency and reliability are improved.
Patent Information
- Application Number
- CN202411142165.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-20
AI Technical Summary
In Kubernetes clusters, existing data recovery methods usually require a large amount of system resources, causing the system to slow down or stop responding, and fail to quickly and effectively restore services in emergencies.
By monitoring the running status of the Kubernetes cluster in real time, different service recovery instructions are generated to quickly find backup sets that can be restored for service, reduce the number of backup sets for query, and improve the efficiency of service recovery.
It realizes the rapid search and recovery of the target service backup set when an exception occurs in Kubernetes cluster, reducing the system resource usage required for service recovery, and improving recovery efficiency and recovery integrity and reliability.
Smart Images

Figure CN118939480A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of databases, and specifically to a service recovery method, device, electronic device, and storage medium. Background Art
[0002] Kubernetes (k8s) is an open-source container orchestration engine from Google, used for automated deployment, scaling, and maintenance of container clusters. This open-source platform enables rapid and effective response to user needs, rapid and predictable deployment of user applications, rapid application expansion, and seamless integration of new application features. Furthermore, it conserves resources, optimizes hardware usage, and provides a complete open-source solution for container orchestration and management. Due to these advantages, Kubernetes has become the de facto standard for container orchestration. An increasing number of systems are using the Kubernetes platform to scale or run their own applications, improving resource utilization and productivity.
[0003] Although applications can be deployed in a high-availability manner within a Kubernetes cluster, emergencies such as earthquakes, fires, or human error in the data center can still cause application unavailability or even data loss. Therefore, data backup is often necessary so that it can be used for recovery in the event of an emergency. However, existing data recovery methods often consume a large amount of system resources, causing the system to slow down or even stop responding. Summary of the Invention
[0004] The present application provides a service recovery method, device, electronic device and storage medium, which generates different service recovery instructions to quickly find backup sets that can be used for service recovery, reduce the number of backup sets queried, and improve the efficiency of service recovery.
[0005] In a first aspect, the present application provides a service recovery method, applied to a client, the method comprising:
[0006] Monitor the running status of the k8s cluster in real time to determine whether the k8s cluster has any abnormalities;
[0007] If an exception occurs in the k8s cluster, the target exception that occurs in the k8s cluster is determined, and a service recovery instruction corresponding to the target exception is generated. Different exceptions that occur in the k8s cluster correspond to different service recovery instructions;
[0008] Determine target service backup sets for service recovery corresponding to different service recovery instructions according to the backup set parameters of the different service recovery instructions and the multiple service backup sets, where the multiple service backup sets are backed up by the k8s cluster according to the service backup strategy;
[0009] Perform service recovery according to the target service backup set to obtain the restored target service.
[0010] In some possible embodiments, before monitoring the operating status of the k8s cluster in real time to determine whether an abnormality occurs in the k8s cluster, the method further includes:
[0011] Obtain a service backup strategy for backing up the k8s cluster;
[0012] Determining the service to be backed up and the first storage location according to the service backup strategy;
[0013] Backing up the service to be backed up to obtain multiple service backup sets, and storing the multiple service backup sets in the first storage location.
[0014] In some possible embodiments, the cluster monitoring component is used to monitor the operating status of the k8s cluster in real time to determine whether the k8s cluster is abnormal, including:
[0015] Deploy the k8s cluster in the client;
[0016] Deploy a status monitoring component in the client, wherein the status monitoring component includes a cluster monitoring component and a client monitoring component;
[0017] Using the cluster monitoring component to monitor the cluster operation indicators of the k8s cluster, and using the client monitoring component to monitor the client operation indicators of the client;
[0018] According to the cluster operation indicators and the client operation indicators, determine whether the k8s cluster is abnormal.
[0019] In some possible embodiments, judging whether an abnormality occurs in the k8s cluster according to the cluster operation indicator and the client operation indicator includes:
[0020] Obtaining a preset operation indicator training data set, wherein the operation indicator training data set includes cluster operation indicator training data, client operation indicator training data, and cluster anomaly judgment results;
[0021] Using the preset operating indicator training data set to train the initial cluster anomaly recognition model to determine the cluster anomaly prediction result;
[0022] Calculating the difference between the cluster abnormality judgment result and the cluster abnormality prediction result;
[0023] Adjusting the model parameters of the initial cluster anomaly recognition model according to the difference until the difference meets a preset training condition, stopping adjusting the model parameters, and obtaining a trained cluster anomaly recognition model;
[0024] According to the cluster anomaly identification model, the cluster operation indicators and the client operation indicators, it is determined whether the k8s cluster has an abnormality.
[0025] In some possible embodiments, the k8s cluster includes a cluster master node and a cluster slave node; if an exception occurs in the k8s cluster, a target exception that occurs in the k8s cluster is determined, and a service recovery instruction corresponding to the target exception is generated. Different exceptions that occur in the k8s cluster correspond to different service recovery instructions, including:
[0026] If an abnormality occurs in the k8s cluster, it is determined that the abnormality is an abnormality in the cluster operation indicator or an abnormality in the client operation indicator;
[0027] If the exception is an abnormality in the client operation indicator, generating a first service recovery instruction;
[0028] If the abnormality is an abnormality of the cluster operation indicator, then the abnormality is determined to be an abnormality of the cluster master node or a cluster slave node;
[0029] If the cluster master node is abnormal, a second service recovery instruction is generated;
[0030] If the cluster slave node is abnormal, a third service recovery instruction is generated.
[0031] In some possible embodiments, according to the different service recovery instructions and the backup set parameters of the multiple service backup sets, the target service backup sets for service recovery corresponding to the different service recovery instructions include:
[0032] Determining, according to the different service recovery instructions, a backup set type for service recovery, wherein the backup set type includes a full backup or an incremental backup;
[0033] Determining a first service backup set in the plurality of service backup sets according to the backup set type;
[0034] Obtaining backup set parameters of the first service backup set;
[0035] determining, based on the backup set parameters corresponding to the respective first service backup sets, service recovery weights corresponding to the respective first service backup sets, to obtain a plurality of service recovery weights;
[0036] The target service backup set is determined in the first service backup set according to the multiple service recovery weights.
[0037] In some possible embodiments, the method further includes:
[0038] The target service is verified to confirm that the target service is restored correctly.
[0039] In a second aspect, an embodiment of the present application provides a service recovery device, the service recovery device comprising:
[0040] The cluster monitoring module is used to monitor the operating status of the k8s cluster in real time to determine whether the k8s cluster has any abnormalities;
[0041] A service recovery instruction generation module is used to determine the target exception of the k8s cluster if an exception occurs in the k8s cluster, and generate a service recovery instruction corresponding to the target exception. Different exceptions in the k8s cluster correspond to different service recovery instructions;
[0042] A target service backup set determination module is used to determine target service backup sets for service recovery corresponding to different service recovery instructions according to the different service recovery instructions and backup set parameters of multiple service backup sets, where the multiple service backup sets are backed up by the k8s cluster according to the service backup strategy;
[0043] The service recovery module is used to perform service recovery according to the target service backup set to obtain the recovered target service.
[0044] In a third aspect, the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the service recovery method as described in any one of the first aspects.
[0045] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the steps in the service recovery method as described in any one of the first aspects.
[0046] In an embodiment of the present application, a service recovery method, device, electronic device and storage medium are provided, including: real-time monitoring of the operating status of the k8s cluster to determine whether an abnormality occurs in the k8s cluster; if an abnormality occurs in the k8s cluster, determining the target abnormality that occurs in the k8s cluster, and generating a service recovery instruction corresponding to the target abnormality. Different abnormalities that occur in the k8s cluster correspond to different service recovery instructions; according to different service recovery instructions and backup set parameters of multiple service backup sets, determining the target service backup set that can perform service recovery corresponding to different service recovery instructions, and multiple service backup sets are backed up for the k8s cluster according to the service backup strategy; performing service recovery according to the target service backup set to obtain the target service after recovery. The present application generates different service recovery instructions for different types of abnormalities that occur in the cluster, selects different service backup sets for service recovery according to different service recovery instructions, reduces the number of service backup sets that need to be queried, and improves the efficiency of service recovery. At the same time, combined with
[0047] Selecting the target service backup set based on the backup set parameters of the service backup set can improve the integrity and reliability after service recovery. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0049] Figure 1 This is a schematic diagram of a service recovery operating system provided by an embodiment of the present application;
[0050] Figure 2 This is a flow chart of an embodiment of the service recovery method provided in the embodiments of the present application;
[0051] Figure 3 A schematic diagram of a flow chart of an embodiment of the present application for determining whether a k8s cluster is abnormal;
[0052] Figure 4 A schematic diagram of a flow chart of an embodiment of verifying a service recovery instruction provided in an embodiment of the present application;
[0053] Figure 5 A flowchart of an embodiment of the present application for selecting different service backup sets according to different service recovery instructions;
[0054] Figure 6 A schematic diagram of an embodiment of a service recovery operation device provided in an embodiment of the present application;
[0055] Figure 7 A schematic structural diagram of an electronic device according to an embodiment of the present application is shown;
[0056] Figure 8 This is a schematic diagram of the terminal device structure provided by an embodiment of the present application;
[0057] Figure 9 This is a schematic diagram of a server structure provided in an embodiment of the present application. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.
[0059] In the following description, the specific embodiments of the present application will be described with reference to steps and symbols performed by one or more computers, unless otherwise stated. Therefore, these steps and operations will be mentioned several times as being performed by a computer, and the computer execution referred to herein includes the operation of a computer processing unit by an electronic signal representing data in a structured form. This operation converts the data or maintains it at a location in the computer's memory system, which can be reconfigured or otherwise change the operation of the computer in a manner familiar to testers in the field. The data structure in which the data is maintained is a physical location in the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the above text, which does not represent a limitation, and testers in the field will understand that the various steps and operations described below can also be implemented in hardware.
[0060] As used herein, the terms "module" or "unit" may be considered software objects executed on the computing system. The various components, modules, engines, and services described herein may be considered implementation objects on the computing system. While the devices and methods described herein are preferably implemented in software, they may also be implemented in hardware and are within the scope of protection of this application.
[0061] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.
[0062] Please refer to Figure 1 ,This application also provides a service to restore the operating system, such as Figure 1 As shown, the service recovery operating system includes a computing device 100, and the service recovery operation apparatus provided by the present application is integrated into the computing device 100. In the embodiment of the present application, the computing device 100 can be a terminal device or a server.
[0063] In the embodiment of the present application, when the computing device 100 is a server, the server can be an independent server or a server network or server cluster composed of servers. For example, the server described in the embodiment of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of multiple servers. Among them, the cloud server is composed of a large number of computers or network servers based on cloud computing. In the embodiment of the present application, communication between the server and the client can be achieved through any communication method, including but not limited to mobile communications based on the Third Generation Partnership Project (3GPP), Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMAX), or computer network communications based on the TCP / IP Protocol Suite (TCP / IP) and User Datagram Protocol (UDP).
[0064] It is understood that when the computing device 100 used in the embodiments of the present application is a terminal device, the terminal device can be a device that includes both receiving hardware and transmitting hardware, that is, a device that has receiving and transmitting hardware capable of performing two-way communication over a two-way communication link. Such terminal devices may include: cellular or other communication devices that have a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display. The specific computing device 100 can be a desktop terminal or a mobile terminal, and the computing device 100 can be a mobile phone, a tablet computer, a laptop computer, etc.
[0065] The terminal devices involved in the embodiments of the present application may also refer to devices that provide voice and / or data connectivity to users, handheld devices with wireless connection capabilities, or other processing devices connected to a wireless modem. For example, mobile phones (or "cellular" phones) and computers with mobile terminals, for example, can be portable, pocket-sized, handheld, computer-built-in, or vehicle-mounted mobile devices that exchange voice and / or data with a wireless access network. For example, personal communication service (PCS) phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), and other devices.
[0066] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer computing devices shown in , or computing device network connection relationships, such as Figure 1 Only one computing device is shown. It is understandable that the service recovery operating system may also include one or more other computing devices, or / and one or more other computing devices connected to the computing device 100 through a network, which is not specifically limited here.
[0067] In addition, if Figure 1 As shown, the service recovery system may further include a memory 200 for storing cluster configuration data and the like.
[0068] In the embodiment of the present application, the memory 200 can be a cloud storage. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.
[0069] Currently, storage systems utilize a method for creating logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a specific storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID) of the data entity. The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can provide access to the data based on the storage location information of each object.
[0070] The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.
[0071] It should be noted that Figure 1 The scenario diagram of the service recovery system shown is only an example. The service recovery operating system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the service recovery operating system and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.
[0072] The solutions provided in the embodiments of the present application involve technologies such as artificial intelligence (AI), computer vision (CV), and machine learning (ML), and are specifically described through the following embodiments:
[0073] AI refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.
[0074] AI technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.
[0075] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, which uses cameras and computers to replace the human eye in identifying, tracking, and measuring objects. This involves further image processing, which transforms the computer's image into an image more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include adversarial perturbation generation, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.
[0076] The service recovery method, device, electronic device and storage medium provided in the embodiments of the present application are described in detail below. It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.
[0077] like Figure 2 FIG. 1 is a flow chart of an embodiment of a service recovery method provided in an embodiment of the present application. Specifically, the service recovery method may include the following steps:
[0078] 201. Monitor the running status of the k8s cluster in real time to determine whether there is any abnormality in the k8s cluster.
[0079] The service recovery method provided in this application is mainly used to recover services in the cluster. Before recovering the services, it is necessary to back up the services first so that the backed-up services can be used for recovery. Therefore, before determining whether the k8s cluster has an abnormality, the service recovery method also includes:
[0080] Obtain a service backup strategy for backing up the k8s cluster; determine the service to be backed up and the first storage location based on the service backup strategy; back up the service to be backed up, obtain multiple service backup sets, and store the multiple service backup sets in the first storage location.
[0081] Specifically, in an embodiment of the present application, it is first necessary to deploy a k8s cluster in an electronic device, and the k8s cluster includes multiple different services; then it is necessary to obtain a preset service backup strategy. In this application, the service backup strategy can be read from the hardware storage of the preset electronic device, or it can be obtained from a cloud storage such as a server, which is not limited here. According to the service backup strategy, the corresponding backup content can be determined, including but not limited to: key information such as service configuration, container image, and persistent data; it also includes the service to be backed up and the first storage location. The service backup strategy can be used to back up the service to be backed up, obtain multiple service backup sets, and store the multiple service backup sets in the first storage location. The first storage location in this application can also be hardware storage such as a local disk, or cloud storage, etc. The specific backup process can refer to the existing technology and is not limited here.
[0082] In the above embodiment, after determining the service to be backed up and the first storage location according to the service backup policy, it is also necessary to obtain a namespace query command for the k8s cluster and run the namespace query command to determine whether the namespace in the k8s cluster is normal. Only when the namespace is normal, the namespace in which the first storage location is located is determined, and the backed-up backup set is stored in the first storage location under the namespace.
[0083] At the same time, in an embodiment of the present application, it is also necessary to continuously monitor the operation of the entire k8s cluster, determine whether the k8s cluster is abnormal, and automatically trigger the service recovery mechanism to perform service recovery after determining that the k8s cluster is abnormal. In some embodiments, the status monitoring component can be redeployed on the client where the k8s cluster is deployed, and the status monitoring component is used to monitor the operating status of the k8s cluster in real time. In an embodiment of the present application, monitoring the operating status of the k8s cluster includes monitoring the operating status of the ks8 cluster itself, and also includes monitoring the client operating status of the client (or electronic device) where the k8s cluster is deployed. Therefore, the status monitoring component actually includes a cluster monitoring component for monitoring the cluster operating indicators of the k8s cluster; and also includes a client monitoring component for monitoring the client operating indicators of the client. And to determine whether the k8s cluster is abnormal, it is necessary to make a comprehensive judgment based on the cluster operating indicators and the client operating indicators.
[0084] like Figure 3 As shown, it is a flow chart of an embodiment of determining whether a k8s cluster is abnormal according to an embodiment of the present application; judging whether a k8s cluster is abnormal according to cluster operation indicators and client operation indicators may include the following steps:
[0085] 301. Obtain a preset operating indicator training data set.
[0086] 302. Use the preset operation indicator training data set to train the initial cluster anomaly recognition model to determine the cluster anomaly prediction result.
[0087] 303. Calculate the difference between the cluster anomaly judgment result and the cluster anomaly prediction result.
[0088] 304. Adjust the model parameters of the initial cluster anomaly recognition model according to the difference until the difference meets the preset training condition, stop adjusting the model parameters, and obtain a trained cluster anomaly recognition model.
[0089] 305. Determine whether the k8s cluster has any abnormalities based on the cluster anomaly identification model, cluster operation indicators, and client operation indicators.
[0090] In some embodiments, a neural network model can be used to determine whether a Kubernetes cluster has experienced an anomaly. Specifically, an initial cluster anomaly detection model must first be trained to obtain a trained cluster anomaly detection model. Therefore, a training set must be obtained, namely, a preset operational indicator training dataset. This operational indicator training dataset includes cluster operational indicator training data, client operational indicator training data, and cluster anomaly detection results, such as whether a cluster anomaly has occurred and, if so, whether the anomaly is a client anomaly or a cluster anomaly. The initial cluster anomaly detection model is then trained using the data in the operational indicator training dataset to determine a cluster anomaly prediction result. The difference between the cluster anomaly prediction result and the actual cluster anomaly detection result is calculated. Based on this difference, the model parameters of the initial cluster anomaly detection model can be continuously adjusted until the final calculated difference meets preset training conditions, such as if the difference is no greater than a preset difference threshold or if the training has reached a preset number of times. At this point, model parameter adjustment can be stopped, resulting in a trained cluster anomaly detection model that meets the requirements. The trained cluster anomaly detection model can then identify cluster operational indicators and client operational indicators, thereby determining whether a Kubernetes cluster anomaly has occurred and what type of anomaly it has occurred. The specific model training process and the process of using the model to determine whether a cluster is abnormal can refer to the existing technology and are not limited in this application.
[0091] 202. If an exception occurs in the k8s cluster, different service recovery instructions will be generated according to the different exceptions that occur in the k8s cluster.
[0092] 203. According to different service recovery instructions, target service backup sets corresponding to different service recovery instructions and capable of performing service recovery are determined in multiple service backup sets. The multiple service backup sets are obtained by backing up the k8s cluster according to the service backup strategy.
[0093] 204. Perform service recovery based on the target service backup set to obtain the recovered target service.
[0094] In an embodiment of the present application, if an abnormality is detected in the k8s cluster, different service recovery instructions can be generated according to the different abnormalities that occur in the k8s cluster. Generally, one service recovery instruction corresponds to one abnormality. According to different service recovery instructions, a corresponding target service backup set for service recovery can be determined from multiple service backup sets, and service recovery can be performed according to the target service backup set to obtain the recovered target service.
[0095] A service recovery method is provided in an embodiment of the present application, including: real-time monitoring of the operating status of the k8s cluster to determine whether an abnormality occurs in the k8s cluster; if an abnormality occurs in the k8s cluster, determining the target abnormality that occurs in the k8s cluster, and generating a service recovery instruction corresponding to the target abnormality, and different abnormalities that occur in the k8s cluster correspond to different service recovery instructions; according to different service recovery instructions and the backup set parameters of multiple service backup sets, determining the target service backup set that can perform service recovery corresponding to the different service recovery instructions, and the multiple service backup sets are backed up by the k8s cluster according to the service backup strategy; performing service recovery according to the target service backup set to obtain the target service after recovery. The present application generates different service recovery instructions for different types of abnormalities that occur in the cluster, selects different service backup sets for service recovery according to different service recovery instructions, reduces the number of service backup sets that need to be queried, and improves the efficiency of service recovery. At the same time, combining the backup set parameters of the service backup set to select the target service backup set can improve the integrity and reliability after service recovery.
[0096] In an embodiment of the present application, a k8s cluster typically has a cluster master node and multiple cluster slave nodes, and the multiple cluster slave nodes are respectively connected to the cluster master node; at this time, if an exception occurs in the k8s cluster, different service recovery instructions are generated according to different exceptions that occur in the k8s cluster, including the following aspects:
[0097] If an abnormality occurs in the k8s cluster, it is determined that the abnormality is a cluster operation indicator abnormality or a client operation indicator abnormality; if the abnormality is a client operation indicator abnormality, a first service recovery instruction is generated; if the abnormality is a cluster operation indicator abnormality, it is determined that the abnormality is a cluster master node abnormality or a cluster slave node abnormality; if the abnormality is the cluster master node, a second service recovery instruction is generated; if the abnormality is the cluster slave node, a third service recovery instruction is generated.
[0098] In the above embodiment, the k8s cluster abnormality includes the abnormality of the k8s cluster itself and the abnormality of the client that deploys the k8s cluster; wherein the client operation indicators include client operation data such as CPU usage, memory usage, network bandwidth, and service response time. If these data are abnormal, it can be determined that the client is abnormal, that is, the client operation indicator is abnormal, and the first service recovery instruction is generated at this time. Similarly, for the cluster master node and the cluster slave node, the master node operation indicators of the cluster master node and the slave node operation indicators of the slave node can be monitored respectively to determine whether the cluster master node or the cluster slave node has an abnormality. The master node operation indicators and slave node operation indicators that need to be monitored can be adjusted according to actual conditions and are not limited here. In this application, different service recovery instructions are generated for different types of abnormalities, namely the first service recovery instruction, the second service recovery instruction, and the third service recovery instruction. Due to the different types of abnormalities, the services that need to be restored are also different. Therefore, different service backup sets need to be selected for service recovery according to different service recovery instructions.
[0099] In the above embodiment, different service recovery instructions are generated according to different exceptions, and the service recovery instructions need to be verified later. Only when the verification is passed can the service be restored according to the service recovery instructions. Figure 4 As shown, verification of the service recovery instruction may include:
[0100] 401. Perform instruction format verification on the service recovery instruction to determine whether the service recovery instruction meets the instruction format of the k8s cluster.
[0101] 402. If the service recovery instruction passes the instruction format check, the service recovery instruction is decoded to determine an abnormality flag in the service recovery instruction.
[0102] 403. Determine whether the exception identifier matches a pre-stored target exception identifier; if the two match, determine that the service recovery instruction passes verification.
[0103] Specifically, it's necessary to first verify that the format of the service recovery instruction matches the format of instructions in the current Kubernetes cluster to avoid receiving malicious instructions or instructions from other clusters. Only when the formats are consistent can the service grayscale instruction be confirmed to be the instruction for service recovery in the current Kubernetes cluster. Generally, both the client and the cluster master and slave nodes have corresponding unique identifiers. When generating a service recovery instruction, this unique identifier must be added to the instruction to determine whether the client or node has experienced an anomaly, requiring service recovery. Adding this unique identifier to the service recovery instruction typically generates a new anomaly identifier; different nodes or clients have different corresponding anomaly identifiers. When generating a service recovery instruction including the anomaly identifier, the generated anomaly identifier must also be stored for subsequent verification of the service recovery instruction. After the service recovery instruction passes the instruction format verification, it is necessary to decode the service recovery instruction, determine the anomaly identifier within the service recovery instruction, and compare the parsed anomaly identifier with the stored target anomaly identifier. If the two match, it can be further confirmed that the current service recovery instruction indicates an anomaly in the current Kubernetes cluster or a client deployed by the current Kubernetes cluster, requiring service recovery. In this application, the instruction format can be set according to actual needs without any limitation; at the same time, the specific process of generating exception identification and comparing exception identification can also refer to the existing technology and is not limited here.
[0104] After the service recovery instruction is verified, the service can be restored according to the service recovery instruction; before the actual service recovery, a target service backup set needs to be selected from multiple service backup sets that have been backed up, and then the target service backup set is used to restore the service. Figure 5 FIG. 1 is a flow chart of an embodiment of selecting different service backup sets according to different service recovery instructions provided by an embodiment of the present application. In some embodiments, determining target service backup sets corresponding to different service recovery instructions for service recovery from multiple service backup sets may include:
[0105] 501. Determine the type of backup set for service recovery according to different service recovery instructions, where the backup set type includes a full backup or an incremental backup.
[0106] 502. Determine a first service backup set from multiple service backup sets according to the backup set type.
[0107] Specifically, if an abnormality occurs in the client operation indicator of the cluster and a first data recovery instruction is generated, it can be determined that an abnormality occurs in the client first. Secondly, generally speaking, when an abnormality occurs in the client, the cluster deployed in the client will most likely also occur abnormally, so it is necessary to restore the original data in the client or cluster as much as possible. Generally speaking, full data is a back-up of all data, and incremental data is a back-up of only changed data; if you want to ensure that all data of the client or cluster is restored to its original state, it is usually necessary to use the full backup data for recovery. As for the nodes in the cluster, different nodes usually run different services, and the services between different nodes are usually independent; if an abnormality occurs in a node, only the service on the node can be restored, and there is no need to use the data obtained from the full backup for data recovery, only the data obtained from the incremental backup can be used for recovery. Therefore, in the embodiment of the present application, different service recovery instructions represent different abnormality types, and according to different service recovery instructions, the type of backup set to be restored can also be determined; specifically, whether to use the backup set obtained from the full backup for service recovery, or to use the backup set obtained from the incremental backup for service recovery.
[0108] In this application, backup set types include full backup and incremental backup. The backup set types of different backup sets are determined when backing up according to the service backup policy; therefore, the first service backup set can be determined from multiple service backup sets based on the backup set type.
[0109] 503. Obtain backup set parameters of the first service backup set.
[0110] 504. Determine the service recovery weight corresponding to each of the first service backup sets according to the backup set parameters corresponding to each of the first service backup sets, and obtain a plurality of service recovery weights.
[0111] 505. Determine a target service backup set in the first service backup set according to the multiple service recovery weights.
[0112] In the aforementioned embodiment, the determination of the first service backup set has already screened the backup sets, reducing the number of backup sets for subsequent service recovery; at the same time, further screening is required to determine the only target service backup set. Specifically, the backup set parameters of the first service backup set can be obtained separately, including but not limited to parameters such as backup time and backup data size; each first service backup set corresponds to a backup set parameter. According to the backup set parameters corresponding to each first service backup set, a service recovery weight can be determined, which represents the reliability of service recovery using the service backup set. Therefore, after determining the service recovery weight corresponding to each first service backup set, the target service backup set that can perform service recovery and corresponds to the service recovery instruction can be determined in the first service backup set based on multiple service recovery weights. In the embodiment of the present application, the larger the service recovery weight, the higher the reliability of service recovery using the service backup set; therefore, the service backup set with the largest service recovery weight can be selected as the target service backup set. The specific method for determining the service backup weight based on the backup set parameters can refer to the existing technology and is not limited here.
[0113] After the target service backup set is selected, the service can be restored according to the selected target service backup set. In the process of service recovery, it is first necessary to determine the second storage location where the restored service is located and the namespace where the second storage location is located; subsequently, it is necessary to restore the service in the second storage location or store the restored service in the second storage location. In some embodiments, service recovery can be performed in the second storage location, and before that, the cluster needs to be configured according to the target service backup set; that is, after reverse configuration according to the configuration and status at the time of service backup, service recovery is performed to ensure that the restored service is consistent with the backup. At the same time, after the service recovery is completed, the target service needs to be verified to determine that the target service has been restored correctly. In an embodiment of the present application, actual business scenarios can be simulated to verify whether the restored service can operate normally and meet business needs.
[0114] In order to facilitate better implementation of the service recovery method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above service recovery method. The meaning of the nouns is the same as that in the above service recovery method. For specific implementation details, please refer to the description in the method embodiment. Figure 6 FIG. 1 is a schematic diagram of an embodiment of a service recovery operation device provided in an embodiment of the present application, which may include:
[0115] The cluster monitoring module 601 is used to monitor the operating status of the k8s cluster in real time to determine whether there is any abnormality in the k8s cluster.
[0116] The service recovery instruction generation module 602 is used to determine the target exception of the k8s cluster if an exception occurs in the k8s cluster, and generate a service recovery instruction corresponding to the target exception. Different exceptions in the k8s cluster correspond to different service recovery instructions.
[0117] The target service backup set determination module 603 is used to determine the target service backup set for service recovery corresponding to different service recovery instructions based on different service recovery instructions and the backup set parameters of multiple service backup sets. The multiple service backup sets are obtained by backing up the k8s cluster according to the service backup strategy.
[0118] The service recovery module 604 is used to perform service recovery according to the target service backup set to obtain the recovered target service.
[0119] In an embodiment of the present application, a service recovery device is provided, which implements the following functions, including: real-time monitoring of the operating status of the k8s cluster to determine whether an abnormality occurs in the k8s cluster; if an abnormality occurs in the k8s cluster, the target abnormality that occurs in the k8s cluster is determined, and a service recovery instruction corresponding to the target abnormality is generated. Different abnormalities that occur in the k8s cluster correspond to different service recovery instructions; according to different service recovery instructions and the backup set parameters of multiple service backup sets, the target service backup set corresponding to the different service recovery instructions for service recovery is determined, and the multiple service backup sets are backed up for the k8s cluster according to the service backup strategy; service recovery is performed according to the target service backup set to obtain the target service after recovery. The present application generates different service recovery instructions for different types of abnormalities that occur in the cluster, selects different service backup sets for service recovery according to different service recovery instructions, reduces the number of service backup sets that need to be queried, and improves the efficiency of service recovery. At the same time, the target service backup set is selected in combination with the backup set parameters of the service backup set, which can improve the integrity and reliability of the service after recovery.
[0120] In some implementations of the present application, the cluster monitoring module 601 may be specifically used to:
[0121] Deploy the k8s cluster in the client;
[0122] Deploy a status monitoring component in the client, wherein the status monitoring component includes a cluster monitoring component and a client monitoring component;
[0123] Using the cluster monitoring component to monitor the cluster operation indicators of the k8s cluster, and using the client monitoring component to monitor the client operation indicators of the client;
[0124] According to the cluster operation indicators and the client operation indicators, determine whether the k8s cluster is abnormal.
[0125] In some implementations of the present application, the cluster monitoring module 601 may be specifically used to:
[0126] Obtaining a preset operation indicator training data set, wherein the operation indicator training data set includes cluster operation indicator training data, client operation indicator training data, and cluster anomaly judgment results;
[0127] Using the preset operating indicator training data set to train the initial cluster anomaly recognition model to determine the cluster anomaly prediction result;
[0128] Calculating the difference between the cluster abnormality judgment result and the cluster abnormality prediction result;
[0129] Adjusting the model parameters of the initial cluster anomaly recognition model according to the difference until the difference meets a preset training condition, stopping adjusting the model parameters, and obtaining a trained cluster anomaly recognition model;
[0130] According to the cluster anomaly identification model, the cluster operation indicators and the client operation indicators, it is determined whether the k8s cluster has an abnormality.
[0131] In some embodiments of the present application, the k8s cluster includes a cluster master node and a cluster slave node, and the service recovery instruction generation module 602 can be specifically used to:
[0132] If an abnormality occurs in the k8s cluster, it is determined that the abnormality is an abnormality in the cluster operation indicator or an abnormality in the client operation indicator;
[0133] If the exception is an abnormality in the client operation indicator, generating a first service recovery instruction;
[0134] If the abnormality is an abnormality of the cluster operation indicator, then the abnormality is determined to be an abnormality of the cluster master node or a cluster slave node;
[0135] If the cluster master node is abnormal, a second service recovery instruction is generated;
[0136] If the cluster slave node is abnormal, a third service recovery instruction is generated.
[0137] In some implementations of the present application, the target service backup set determination module 603 may be specifically configured to:
[0138] Determining, according to the different service recovery instructions, a backup set type for service recovery, wherein the backup set type includes a full backup or an incremental backup;
[0139] Determining a first service backup set in the plurality of service backup sets according to the backup set type;
[0140] Obtaining backup set parameters of the first service backup set;
[0141] determining, based on the backup set parameters corresponding to the respective first service backup sets, service recovery weights corresponding to the respective first service backup sets, to obtain a plurality of service recovery weights;
[0142] The target service backup set is determined in the first service backup set according to the multiple service recovery weights.
[0143] The present application also provides an electronic device, such as Figure 7 , which shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:
[0144] The electronic device may include one or more processing core processors 701, one or more computer-readable storage media memories 702, a power supply 703, an input unit 704 and other components. Those skilled in the art will understand that Figure 7 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.
[0145] The processor 701 is the control center of the electronic device. It connects the various parts of the entire electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 702 and calling data stored in the memory 702, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operation of storage media, user interface, and application programs, and the modem processor mainly handles wireless communications. It is understood that the above-mentioned modem processor may not be integrated into the processor 701.
[0146] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area, wherein the program storage area may store operating storage media, applications required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.
[0147] The electronic device also includes a power supply 703 for supplying power to various components. Preferably, the power supply 703 can be logically connected to the processor 701 via a power management storage medium, thereby implementing functions such as charging, discharging, and power consumption management through the power management storage medium. The power supply 703 can also include one or more DC or AC power supplies, a recharge storage medium, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.
[0148] The electronic device may further include an input unit 704, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0149] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically, in this embodiment, the processor 701 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 702 according to the following instructions, and the processor 701 will run the application programs stored in the memory 702, thereby implementing the steps in the above-mentioned service recovery method embodiment.
[0150] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0151] To this end, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. The computer program is loaded by a processor to execute the steps in any one of the service recovery methods provided in the embodiments of the present application.
[0152] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.
[0153] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Since the computer program stored in the computer-readable storage medium can execute the steps of any of the service recovery methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the service recovery methods provided in the embodiments of the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.
[0154] When the computing device in the embodiment of the present application is a terminal device, the embodiment of the present application also provides a terminal device, such as Figure 8 For ease of explanation, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal device can be any terminal device including a mobile phone, tablet computer, personal digital assistant (PDA), point of sales (POS), car computer, etc., taking the mobile phone as an example:
[0155] Figure 8 The block diagram shows a partial structure of a mobile phone related to the terminal device provided in the embodiment of the present application. Figure 8 The mobile phone includes components such as a radio frequency (RF) circuit 810, a memory 820, an input unit 830, a display unit 840, a sensor 850, an audio circuit 860, a wireless fidelity (WiFi) module 870, a processor 880, and a power supply 890. Those skilled in the art will understand that Figure 8 The mobile phone structure shown in the figure does not constitute a limitation to the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0156] The following combination Figure 8 A detailed introduction to the various components of a mobile phone:
[0157] The RF circuit 810 can be used to receive and send signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to the processor 880 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 810 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 810 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.
[0158] The memory 820 can be used to store software programs and modules. The processor 880 executes the various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 820. The memory 820 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 820 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.
[0159] The input unit 830 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 830 may include a touch panel 831 and other input devices 832. The touch panel 831, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel 831) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 831 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 880, and can receive commands sent by the processor 880 and execute them. In addition, the touch panel 831 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 831, the input unit 830 may further include other input devices 832. Specifically, the other input devices 832 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick.
[0160] The display unit 840 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 840 may include a display panel 841. Optionally, the display panel 841 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 831 may cover the display panel 841. When the touch panel 831 detects a touch operation on or near it, it is transmitted to the processor 880 to determine the type of touch event. Subsequently, the processor 880 provides corresponding visual output on the display panel 841 according to the type of touch event. Although in Figure 8 In the embodiment, the touch panel 831 and the display panel 841 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 831 and the display panel 841 can be integrated to realize the input and output functions of the mobile phone.
[0161] The mobile phone may also include at least one sensor 850, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 841 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 841 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.
[0162] Audio circuit 860, speaker 861, and microphone 862 provide an audio interface between the user and the phone. Audio circuit 860 converts received audio data into electrical signals and transmits them to speaker 861, which then converts them into sound signals for output. Microphone 862, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 860 and converted into audio data. The audio data is then processed by processor 880 and transmitted to, for example, another phone via RF circuit 810, or stored in memory 820 for further processing.
[0163] Wi-Fi is a short-range wireless transmission technology. Mobile phones can help users send and receive emails, browse the web, and access streaming media through the Wi-Fi module 870. It provides users with wireless broadband Internet access. Figure 8 A Wi-Fi module 870 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.
[0164] Processor 880 is the control center of the mobile phone, connecting all parts of the mobile phone using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 820 and accessing data stored in memory 820, it performs various functions of the mobile phone and processes data, thereby providing overall monitoring of the mobile phone. Optionally, processor 880 may include one or more processing units; alternatively, processor 880 may integrate an application processor and a modem processor, with the application processor primarily handling the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 880.
[0165] The mobile phone also includes a power supply 890 (such as a battery) for powering each component. Optionally, the power supply can be logically connected to the processor 880 through a power management system, thereby managing charging, discharging, and power consumption management through the power management system. Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here. In an embodiment of the present application, the processor 880 included in the mobile phone also has a function of controlling the execution of the service recovery method process performed by the service recovery device.
[0166] The present application also provides a server. Figure 9 , Figure 9 This is a schematic diagram of a server structure provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 922 (for example, one or more processors) and memories 932, and one or more storage media 930 (for example, one or more mass storage devices) for storing application programs 942 or data 944. The memories 932 and storage media 930 may be short-term storage or persistent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 922 may be configured to communicate with the storage medium 930 to execute a series of instruction operations in the storage medium 930 on the server 900.
[0167] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems 941, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.
[0168] The steps in the service recovery method in the above embodiment can be based on the Figure 9 The structure of the server 900 is shown.
[0169] For example, the CPU 922 performs the following operations by calling instructions in the memory 932:
[0170] Monitor the running status of the k8s cluster in real time to determine whether the k8s cluster has any abnormalities;
[0171] If an exception occurs in the k8s cluster, the target exception that occurs in the k8s cluster is determined, and a service recovery instruction corresponding to the target exception is generated. Different exceptions that occur in the k8s cluster correspond to different service recovery instructions;
[0172] Determine target service backup sets for service recovery corresponding to different service recovery instructions according to the backup set parameters of the different service recovery instructions and the multiple service backup sets, where the multiple service backup sets are backed up by the k8s cluster according to the service backup strategy;
[0173] Perform service recovery according to the target service backup set to obtain the restored target service.
[0174] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0175] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0176] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0177] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.
[0178] In addition, the functional modules in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into a module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium.
[0179] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0180] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive Solid State Disk (SSD)), etc.
[0181] The above is a detailed introduction to the technical solutions provided in the embodiments of the present application. Specific examples are used in the embodiments of the present application to illustrate the principles and implementation methods of the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.
Claims
1. A service recovery method, It is characterized in that Applied to a client, the method comprises: Monitor the running status of the k8s cluster in real time to determine whether the k8s cluster is abnormal; If an exception occurs in the k8s cluster, the target exception that occurs in the k8s cluster is determined, and a service recovery instruction corresponding to the target exception is generated. Different exceptions that occur in the k8s cluster correspond to different service recovery instructions; According to the different service recovery instructions and the backup set parameters of the multiple service backup sets, determine the target service backup sets for service recovery corresponding to the different service recovery instructions, where the multiple service backup sets are backed up by the k8s cluster according to the service backup strategy; The service is restored according to the target service backup set to obtain the restored target service.
2. The service recovery method according to claim 1, It is characterized in that Before real-time monitoring of the running status of the k8s cluster to determine whether the k8s cluster is abnormal, the method further includes: Obtain a service backup strategy for backing up the k8s cluster; Determine the service to be backed up and the first storage location according to the service backup strategy; The service to be backed up is backed up to obtain a plurality of service backup sets, and the plurality of service backup sets are stored in the first storage location.
3. The service recovery method according to claim 1, It is characterized in that The cluster monitoring component is used to monitor the running status of the k8s cluster in real time to determine whether the k8s cluster is abnormal, including: Deploy the k8s cluster in the client; Deploy a status monitoring component in the client, wherein the status monitoring component includes a cluster monitoring component and a client monitoring component; Using the cluster monitoring component to monitor the cluster operation indicators of the k8s cluster, and using the client monitoring component to monitor the client operation indicators of the client; According to the cluster operation indicators and the client operation indicators, determine whether the k8s cluster is abnormal.
4. The service recovery method according to claim 3, It is characterized in that The step of judging whether the k8s cluster is abnormal according to the cluster operation index and the client operation index includes: Obtaining a preset operation indicator training data set, wherein the operation indicator training data set includes cluster operation indicator training data, client operation indicator training data, and cluster abnormality judgment results; Using the preset operation indicator training data set to train the initial cluster anomaly recognition model to determine the cluster anomaly prediction result; Calculating the difference between the cluster abnormality judgment result and the cluster abnormality prediction result; Adjusting the model parameters of the initial cluster anomaly recognition model according to the difference until the difference meets the preset training condition, stopping adjusting the model parameters, and obtaining a trained cluster anomaly recognition model; According to the cluster anomaly identification model, the cluster operation index and the client operation index, it is determined whether the k8s cluster has an abnormality.
5. The service recovery method according to claim 4, It is characterized in that The k8s cluster includes a cluster master node and a cluster slave node; if an exception occurs in the k8s cluster, a target exception in the k8s cluster is determined, and a service recovery instruction corresponding to the target exception is generated. Different exceptions in the k8s cluster correspond to different service recovery instructions, including: If the k8s cluster is abnormal, it is determined that the abnormality is an abnormality in the cluster operation index or an abnormality in the client operation index; If the exception is an abnormality in the client operation indicator, generating a first service recovery instruction; If the abnormality is an abnormality of the cluster operation indicator, then the abnormality is determined to be an abnormality of the cluster master node or a cluster slave node; If the cluster master node is abnormal, a second service recovery instruction is generated; If the cluster slave node is abnormal, a third service recovery instruction is generated.
6. The service recovery method according to claim 5, It is characterized in that According to the different service recovery instructions and the backup set parameters of the multiple service backup sets, the target service backup sets for service recovery corresponding to the different service recovery instructions include: Determine the type of backup set for service recovery according to the different service recovery instructions, where the backup set type includes a full backup or an incremental backup; Determine a first service backup set in the plurality of service backup sets according to the backup set type; Obtaining backup set parameters of the first service backup set; Determine, according to the backup set parameters corresponding to each of the first service backup sets, the service recovery weights corresponding to each of the first service backup sets, to obtain a plurality of service recovery weights; The target service backup set is determined in the first service backup set according to the multiple service recovery weights.
7. The service recovery method according to claim 1, It is characterized in that The method further comprises: The target service is verified to confirm that the target service is restored correctly.
8. A service recovery device, It is characterized in that The service recovery device comprises: The cluster monitoring module is used to monitor the running status of the k8s cluster in real time to determine whether the k8s cluster has any abnormalities; A service recovery instruction generation module is used to determine the target exception of the k8s cluster if an exception occurs in the k8s cluster, and generate a service recovery instruction corresponding to the target exception. Different exceptions occurring in the k8s cluster correspond to different service recovery instructions; A target service backup set determination module is used to determine target service backup sets for service recovery corresponding to different service recovery instructions according to the different service recovery instructions and backup set parameters of multiple service backup sets, wherein the multiple service backup sets are obtained by backing up the k8s cluster according to the service backup strategy; The service recovery module is used to perform service recovery according to the target service backup set to obtain the restored target service.
9. An electronic device, It is characterized in that The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps in the service recovery method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the service recovery method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Data recovery method, apparatus and system based on distributed system
CN107451013A
Server operation state alarm method and device, server and storage medium
CN111459770A
Data Backup Method and System, and Related Device
US20240054054A1
Cited By
Data backup and recovery method and device, electronic equipment and storage medium
CN121144109A
