A node management method and device, electronic equipment and storage medium

Abstract

The application provides a node management method and device, electronic equipment and storage medium, and belongs to the technical field of big data. In the running process of a job management node, the running state of the job management node is listened to. When it is listened that the job management node exits the running due to a fault, a target snapshot stored in the running process of the job management node is acquired, wherein the target snapshot is used for storing intermediate data in the running process of the job management node; and the job management node is restarted based on the target snapshot. That is, through the application, high-availability deployment of Flink on kubernetes can be realized by running only one job management node, without simultaneously running multiple job management nodes, thereby saving server resources.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a node management method, apparatus, electronic device and storage medium. Background Technology

[0002] Flink (a distributed streaming data stream engine), as a next-generation big data processing engine, has become a highly sought-after technology framework due to its powerful computing capabilities and advanced design concepts, and is valued by many large internet companies. Kubernetes (a container orchestration engine) is a popular container orchestration tool. Combined with Docker (an application container engine), it can provide powerful cluster resource management functions, becoming one of the main solutions for container clouds.

[0003] Currently, a high-availability deployment scheme for Flink on Kubernetes is to start multiple Flink JobManagers (job management nodes) simultaneously on Kubernetes, and switch to the backup JobManager when the primary JobManager fails, based on the Kubernetes configmap (configuration dictionary).

[0004] However, the above solution requires multiple JobManagers to be started simultaneously on Kubernetes to achieve high availability of Flink on Kubernetes, which consumes a lot of resources. Summary of the Invention

[0005] The purpose of this application is to provide a node management method, apparatus, electronic device, and storage medium to solve the problem that current methods require multiple JobManagers to be started simultaneously on Kubernetes to achieve high availability deployment of Flink on Kubernetes, which consumes a lot of resources. The specific technical solution is as follows:

[0006] Firstly, a node management method is provided, the method comprising:

[0007] During the operation of the job management node, monitor the running status of the job management node;

[0008] When the job management node is detected to have exited operation due to a fault, a target snapshot stored during the operation of the job management node is obtained, wherein the target snapshot is used to store intermediate data during the operation of the job management node;

[0009] The job management node is restarted based on the target snapshot.

[0010] Optionally, the step of detecting that the job management node has exited operation due to a fault includes:

[0011] When the exit code corresponding to the job management node is detected and it is determined that the exit code is the first value, it is determined that the job management node has exited operation due to a fault.

[0012] Optionally, the method further includes:

[0013] When the container to which the job management node belongs starts, determine whether the container's startup is a fault recovery restart;

[0014] If so, obtain the target snapshot stored during the operation of the job management node;

[0015] The job management node is restarted based on the target snapshot.

[0016] Optionally, the method further includes:

[0017] When the container starts, record the number of times the container starts;

[0018] Determining whether the container's current startup is a fault recovery restart includes:

[0019] Obtain the number of times the container has been started;

[0020] When the number of startups is determined to be greater than a preset threshold, the current startup of the container is determined to be a fault recovery startup.

[0021] Optionally, obtaining the target snapshot stored during the operation of the job management node includes:

[0022] The multiple snapshots stored during the operation of the job management node are sorted according to their creation time.

[0023] The last snapshot in the list is identified as the target snapshot.

[0024] Optionally, obtaining the target snapshot stored during the operation of the job management node includes:

[0025] Obtain multiple snapshots stored during the operation of the job management node;

[0026] Send the identification information of multiple snapshots to a preset object;

[0027] The snapshot corresponding to the identification information returned by the preset object is determined as the target snapshot.

[0028] Optionally, the method further includes:

[0029] Delete all snapshots except the target snapshot.

[0030] Secondly, a node management device is provided, the device comprising:

[0031] The monitoring module is used to monitor the running status of the job management node during its operation.

[0032] The acquisition module is used to acquire a target snapshot stored during the operation of the job management node when it is detected that the job management node has exited the operation due to a fault. The target snapshot is used to store intermediate data during the operation of the job management node.

[0033] The restart module is used to restart the job management node based on the target snapshot.

[0034] Optionally, the apparatus further includes a determining module, the determining module being configured to:

[0035] When the exit code corresponding to the job management node is detected and it is determined that the exit code is the first value, it is determined that the job management node has exited operation due to a fault.

[0036] Optionally, the determining module is further configured to:

[0037] When the container to which the job management node belongs starts, determine whether the container's startup is a fault recovery restart;

[0038] If so, obtain the target snapshot stored during the operation of the job management node;

[0039] The job management node is restarted based on the target snapshot.

[0040] Optionally, the device further includes a recording module, the recording module being used for:

[0041] When the container starts, record the number of times the container starts;

[0042] The determining module is further configured to:

[0043] Obtain the number of times the container has been started;

[0044] When the number of startups is determined to be greater than a preset threshold, the current startup of the container is determined to be a fault recovery startup.

[0045] Optionally, the acquisition module is further configured to:

[0046] The multiple snapshots stored during the operation of the job management node are sorted according to their creation time.

[0047] The last snapshot in the list is identified as the target snapshot.

[0048] Optionally, the acquisition module is further configured to:

[0049] Obtain multiple snapshots stored during the operation of the job management node;

[0050] Send the identification information of multiple snapshots to a preset object;

[0051] The snapshot corresponding to the identification information returned by the preset object is determined as the target snapshot.

[0052] Optionally, the device further includes a deletion module, the deletion module being configured to:

[0053] Delete all snapshots except the target snapshot.

[0054] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0055] Memory, used to store computer programs;

[0056] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0057] Fourthly, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0058] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute any of the node management methods described above.

[0059] Beneficial effects of the embodiments in this application:

[0060] This application provides a node management method, apparatus, electronic device, and storage medium. During the operation of a job management node, the application monitors its running status. When a job management node is detected to have exited operation due to a fault, a target snapshot stored during the job management node's operation is obtained, and the job management node is restarted based on this target snapshot. In other words, this application achieves high-availability deployment of Flink on Kubernetes by running only one job management node, eliminating the need to run multiple job management nodes simultaneously and saving server resources.

[0061] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A flowchart illustrating a node management method provided in an embodiment of this application;

[0064] Figure 2 A flowchart illustrating a node management method provided in another embodiment of this application;

[0065] Figure 3 A flowchart illustrating a node management method provided in another embodiment of this application;

[0066] Figure 4 This is a schematic diagram of the structure of a node management device provided in an embodiment of this application;

[0067] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] Currently, a high-availability deployment scheme for Flink on Kubernetes is to start multiple Flink JobManagers (job management nodes) simultaneously on Kubernetes, and switch to the backup JobManager when the primary JobManager fails, based on the Kubernetes configmap (configuration dictionary).

[0070] However, this solution requires multiple JobManagers to be started simultaneously on Kubernetes to achieve high availability of Flink on Kubernetes, which consumes a lot of resources.

[0071] Therefore, this application provides a node management method.

[0072] The following will describe in detail a node management method provided in the embodiments of this application, with reference to specific implementation methods. Figure 1 As shown, the specific steps are as follows:

[0073] S101 monitors the running status of the job management node during its operation.

[0074] The node management method provided in this application is used to achieve highly available deployment of Flink on Kubernetes. Here, the job management node refers to the job management node in Flink, and its running status includes normal operation, normal exit, and failure exit.

[0075] In this embodiment of the application, a monitoring process for the job management node is started at the same time as the job management node is started, so as to monitor the running status of the job management node.

[0076] S102, when it is detected that the job management node has exited the operation due to a fault, obtain the target snapshot stored during the operation of the job management node, wherein the snapshot is used to store intermediate data during the operation of the job management node.

[0077] S103, Restart the job management node based on the target snapshot.

[0078] In this embodiment, a fault tolerance mechanism is initiated simultaneously with the startup of the job management node. Based on this mechanism, snapshots of the job management node are continuously created at regular intervals during its operation. These snapshots are used to store intermediate data generated during the job management node's execution. Furthermore, these snapshots can be stored in the job management node's memory, or in a local file system or a distributed file system.

[0079] Based on this, when the job management node is detected to have exited operation due to a fault, the target snapshot is obtained from all snapshots stored during the operation of the job management node, and the intermediate data stored in the target snapshot is used to restart the job management node.

[0080] In this embodiment, during the operation of the job management node, its running status is monitored. When the job management node is detected to have exited operation due to a failure, a target snapshot stored during the operation of the job management node is obtained, and the job management node is restarted based on the target snapshot. That is, this application enables high-availability deployment of Flink on Kubernetes by running only one job management node, eliminating the need to run multiple job management nodes simultaneously and saving server resources.

[0081] In another embodiment of this application, S102 may include the following steps:

[0082] When the exit code corresponding to the job management node is detected and it is determined that the exit code is the first value, it is determined that the job management node has exited operation due to a fault.

[0083] In this embodiment, the exit code of a job management node is zero when it exits normally, and it exits due to a fault when the exit code is any other value (hereinafter referred to as the first value, which is a non-zero value). Therefore, when the exit code corresponding to the job management node is detected and it is determined that the exit code is the first value, it can be determined that the job management node has exited operation due to a fault.

[0084] In yet another embodiment of this application, as Figure 2 As shown, the method may also include the following steps:

[0085] S201, when the container belonging to the job management node starts, determine whether the container's current startup is a fault recovery restart;

[0086] S202, if so, then obtain the target snapshot stored during the operation of the job management node;

[0087] S203, restart the job management node based on the target snapshot.

[0088] In this embodiment, a container refers to a Pod in Kubernetes. The job management node runs within this container. During runtime, the container is started first, and the job management node is started after the container starts. Based on the Kubernetes Pod high availability mechanism, the container will automatically restart when it fails (i.e., Pod Crash), i.e., a fault recovery restart.

[0089] When a container exits due to a failure, the job management node running in that container will also inevitably exit due to a failure. Therefore, when it is determined whether the current startup of the container is a failure recovery restart, it means that the startup of the job management node is also a restart after exiting due to a failure. Therefore, we can directly execute the steps of obtaining the target snapshot stored during the operation of the job management node and restarting the job management node based on the target snapshot.

[0090] This solution allows for the direct determination of a restart method for the job management node when the current container startup is identified as a fault recovery restart. This eliminates the need to rely on exit codes to determine whether the job management node has exited due to a fault, thus enabling a rapid determination of the restart plan and improving restart efficiency.

[0091] In another embodiment of this application, S201 may include the following steps:

[0092] Step 1: Obtain the number of times the container has been started;

[0093] Step 2: When the number of startups exceeds a preset threshold, determine that the current startup of the container is a fault recovery startup.

[0094] In this embodiment, the number of times the container starts is recorded via an external interface each time it starts. When the number of starts is 1, it indicates that the container is starting for the first time; when the number of starts is greater than 1, it indicates that the container is starting after a failure. Therefore, when the container starts, the number of starts recorded via the external interface can be used to determine whether the container is starting after a failure.

[0095] In another embodiment of this application, a target snapshot can be obtained through the following steps:

[0096] Step 1: Sort the multiple snapshots stored during the operation of the job management node according to their creation time.

[0097] Step two: Select the last snapshot as the target snapshot.

[0098] In this embodiment, the most recently created snapshot can be determined as the target snapshot. During the operation of the job management node, snapshots of the job management node are continuously created through a fault-tolerance mechanism, and the creation time of each snapshot is recorded. According to the order of creation time, the multiple snapshots stored during the operation of the job management node are sorted, and the last snapshot in the order is the most recently created snapshot, and therefore, it is determined as the target snapshot.

[0099] This solution allows for the rapid determination of target snapshots, improving the efficiency of job management node restarts. Furthermore, restarting job management nodes using the latest created snapshot avoids the duplication of already processed data, saving computing resources.

[0100] In yet another embodiment of this application, a target snapshot can also be obtained through the following steps:

[0101] S301, retrieve multiple snapshots stored during the operation of the job management node;

[0102] S302, send the identification information of multiple snapshots to the preset object;

[0103] S303, determine the snapshot corresponding to the identification information returned by the preset object as the target snapshot.

[0104] In this embodiment, the preset object refers to a user-end device, such as a computer or mobile phone. By sending the identification information of multiple snapshots stored during the operation of the job management node to the preset object, the user can select one based on the identification information received by the preset object. After the user confirms the selection, the preset object returns the selected identification information, and the snapshot corresponding to that identification information is determined as the target snapshot. This solution allows the target snapshot to be determined according to user needs, that is, the job management node can be restarted according to user requirements, thereby improving the user experience.

[0105] In yet another embodiment of this application, the method may further include the following steps:

[0106] Delete all snapshots except the target snapshot.

[0107] In this embodiment of the application, after determining the target snapshot, other snapshots besides the target snapshot can be deleted, thereby saving storage resources.

[0108] In this embodiment, during the operation of the job management node, its running status is monitored. When the job management node is detected to have exited operation due to a failure, a target snapshot stored during the operation of the job management node is obtained, and the job management node is restarted based on the target snapshot. That is, this application enables high-availability deployment of Flink on Kubernetes by running only one job management node, eliminating the need to run multiple job management nodes simultaneously and saving server resources.

[0109] Based on the same technical concept, embodiments of this application also provide a node management device, such as... Figure 4 As shown, the device includes:

[0110] The monitoring module 401 is used to monitor the running status of the job management node during its operation.

[0111] The acquisition module 402 is used to acquire a target snapshot stored during the operation of the job management node when it is detected that the job management node has exited the operation due to a fault. The target snapshot is used to store intermediate data during the operation of the job management node.

[0112] The restart module 403 is used to restart the job management node based on the target snapshot.

[0113] Optionally, the apparatus further includes a determining module, the determining module being configured to:

[0114] When the exit code corresponding to the job management node is detected and it is determined that the exit code is the first value, it is determined that the job management node has exited operation due to a fault.

[0115] Optionally, the determining module is further configured to:

[0116] When the container to which the job management node belongs starts, determine whether the container's startup is a fault recovery restart;

[0117] If so, obtain the target snapshot stored during the operation of the job management node;

[0118] The job management node is restarted based on the target snapshot.

[0119] Optionally, the device further includes a recording module, the recording module being used for:

[0120] When the container starts, record the number of times the container starts;

[0121] The determining module is further configured to:

[0122] Obtain the number of times the container has been started;

[0123] When the number of startups is determined to be greater than a preset threshold, the current startup of the container is determined to be a fault recovery startup.

[0124] Optionally, the acquisition module is further configured to:

[0125] The multiple snapshots stored during the operation of the job management node are sorted according to their creation time.

[0126] The last snapshot in the list is identified as the target snapshot.

[0127] Optionally, the acquisition module is further configured to:

[0128] Obtain multiple snapshots stored during the operation of the job management node;

[0129] Send the identification information of multiple snapshots to a preset object;

[0130] The snapshot corresponding to the identification information returned by the preset object is determined as the target snapshot.

[0131] Optionally, the device further includes a deletion module, the deletion module being configured to:

[0132] Delete all snapshots except the target snapshot.

[0133] In this embodiment, during the operation of the job management node, the running status of the job management node is monitored. When the job management node is detected to have exited operation due to a failure, a target snapshot stored during the operation of the job management node is obtained, and the job management node is restarted based on the target snapshot. That is, this application enables high-availability deployment of Flink on Kubernetes by running only one job management node, eliminating the need to run multiple job management nodes simultaneously and saving server resources.

[0134] Based on the same technical concept, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, it includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0135] Memory 113 is used to store computer programs;

[0136] When processor 111 executes a program stored in memory 113, it performs the following steps:

[0137] During the operation of the job management node, monitor the running status of the job management node;

[0138] When the job management node is detected to have exited operation due to a fault, a target snapshot stored during the operation of the job management node is obtained, wherein the target snapshot is used to store intermediate data during the operation of the job management node;

[0139] The job management node is restarted based on the target snapshot.

[0140] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0141] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0142] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as multiple disk drives. Optionally, the memory may also be multiple storage devices located remotely from the aforementioned processor.

[0143] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0144] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described node management methods.

[0145] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the node management methods described above.

[0146] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0147] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0148] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A node management method characterized by comprising: The method includes: During the operation of the only job management node in the Kubernetes cluster, the running status of the job management node is monitored, and snapshots of the job management node are created at set time intervals during the operation of the job management node. The snapshots are used to store intermediate data during the operation of the job management node. When the job management node is detected to have exited operation due to a fault, obtain the target snapshot stored during the operation of the job management node; Restart the job management node based on the target snapshot; The step of obtaining the target snapshot stored during the operation of the job management node includes: Obtain multiple snapshots stored during the operation of the job management node; Send the identification information of multiple snapshots to a preset object; The snapshot corresponding to the identification information returned by the preset object is determined as the target snapshot, wherein the identification information returned by the preset object is selected by the user.

2. The method of claim 1, wherein, The detection that the job management node has exited operation due to a fault includes: When the exit code corresponding to the job management node is detected and it is determined that the exit code is the first value, it is determined that the job management node has exited operation due to a fault.

3. The method of claim 1, wherein, The method further includes: When the container to which the job management node belongs starts, determine whether the container's startup is a fault recovery restart; If so, obtain the target snapshot stored during the operation of the job management node; The job management node is restarted based on the target snapshot.

4. The method of claim 3, wherein, The method further includes: When the container starts, record the number of times the container starts; Determining whether the container's current startup is a fault recovery restart includes: Obtain the number of times the container has been started; When the number of startups is determined to be greater than a preset threshold, the current startup of the container is determined to be a fault recovery startup.

5. The method according to claim 1 or 3, characterized in that, The step of obtaining the target snapshot stored during the operation of the job management node includes: The multiple snapshots stored during the operation of the job management node are sorted according to their creation time. The last snapshot in the list is identified as the target snapshot.

6. The method of claim 1, wherein, The method further includes: Delete all snapshots except the target snapshot.

7. A node management apparatus characterized by comprising: The device includes: The monitoring module is used to monitor the running status of the single job management node in the Kubernetes cluster during its operation, and to create snapshots of the job management node at set time intervals during the operation of the job management node. The snapshots are used to store intermediate data during the operation of the job management node. The acquisition module is used to acquire a target snapshot stored during the operation of the job management node when the job management node is detected to have exited operation due to a fault. The restart module is used to restart the job management node based on the target snapshot; The step of obtaining the target snapshot stored during the operation of the job management node includes: Obtain multiple snapshots stored during the operation of the job management node; Send the identification information of multiple snapshots to a preset object; The snapshot corresponding to the identification information returned by the preset object is determined as the target snapshot, wherein the identification information returned by the preset object is selected by the user.

8. An electronic device, comprising: It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

Images