Network management method applied to distributed cluster and related device

Abstract

The embodiment of the invention discloses a network management method applied to a distributed cluster and a related device. The method is executed by a cluster management server deployed in a distributed cluster, and a timer scheduling program and a thread pool which are independent of each other run in the cluster management server. Generating a network inspection task for each node in the distributed cluster; sending the network inspection task to the thread pool, and concurrently executing the network inspection task through a plurality of task execution threads in the thread pool; and if it is detected that execution abnormity occurs when the task execution thread executes the network check task, performing alarm processing on the network check task which is executed abnormally. According to the invention, reliable operation of cloud network services can be ensured.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a network management method and related devices applied to a distributed cluster. Background Art

[0002] A distributed cluster is a system composed of multiple computer nodes. The nodes communicate and collaborate with each other through the network to complete the same task or provide the same service. The communication methods include point-to-point communication, multicast communication, broadcast communication, etc. Distributed clusters have the advantages of high availability, high performance, and scalability, so they can be applied to large-scale computing, storage, processing and other scenarios.

[0003] A large number of network devices are running in a distributed cluster at the same time. The environment in which the network devices run is complex, and various faults may occur at any time, such as device hardware failure, network connection failure, etc. In order to ensure the stable operation of the distributed cluster, it is necessary to promptly handle the faulty device when a fault occurs. In the existing technical implementation, the nodes in the distributed cluster are regularly checked for network. The network check method is that the cluster management service calls the check interface of the node device. If the device returns a message normally, it means that there is no fault. If the message is not returned after a timeout, it means that a fault has occurred.

[0004] As the scale of cloud network business development grows, the shortcomings of existing network inspection solutions are becoming more and more obvious. Specifically, in the implementation of cloud network business scenarios, distributed nodes may be deployed in various regions. In some failure scenarios, such as when a network in a certain region is abnormal, the cluster management service's inspection request for all nodes in this region will time out. If the number of timed-out nodes is too large, the timer will block the network inspection of the next cycle, and after a period of time, the entire cluster management service will crash, so the reliable operation of cloud network business cannot be guaranteed. Summary of the invention

[0005] To solve the above technical problems, embodiments of the present application provide a network management method and device for a distributed cluster, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] In the first aspect, an embodiment of the present application provides a network management method applied to a distributed cluster, the method being executed by a cluster management server deployed in the distributed cluster, wherein the cluster management server runs independent timer schedulers and thread pools, the method comprising: when the timer scheduler detects that a timing period has arrived, generating a network inspection task for each node in the distributed cluster; sending the network inspection task to the thread pool, and concurrently executing the network inspection task through multiple task execution threads in the thread pool; if it is detected that an execution exception occurs in the task execution thread when executing the network inspection task, performing alarm processing on the network inspection task with the execution exception.

[0007] In the second aspect, an embodiment of the present application provides a network management device for a distributed cluster, wherein the device is configured in a cluster management server deployed in the distributed cluster, and independent timer schedulers and thread pools are running in the cluster management server, and the device includes: a generation module, configured to generate a network inspection task for each node in the distributed cluster when the timer scheduler detects that a timing period has arrived; an execution module, configured to send the network inspection task to the thread pool, and concurrently execute the network inspection task through multiple task execution threads in the thread pool; and an alarm module, configured to perform alarm processing for the network inspection task with the execution exception if it is detected that an execution exception occurs in the task execution thread when executing the network inspection task.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the steps in the network management method applied to a distributed cluster as described above.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of a computer, the computer executes the steps in the network management method applied to a distributed cluster as described above.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the network management method applied to a distributed cluster as described above.

[0011] In the technical solution provided in the embodiments of the present application, since the timer scheduler and the thread pool are independent of each other in the cluster management server, when the distributed cluster is implemented as a cloud network service, even if the task execution threads in the thread pool have timeout exceptions when performing network checks on some nodes, it will not block the timer scheduler from scheduling network inspection tasks, thereby ensuring the reliable operation of the cloud network service.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram of an exemplary network management architecture for a distributed cluster;

[0014] Figure 2 It is for Figure 1 The network management architecture shown is a flowchart of a network management method applied to a distributed cluster;

[0015] Figure 3 is a schematic diagram of another exemplary network management architecture for a distributed cluster;

[0016] Figure 4 is Figure 1 A schematic diagram of another network management architecture for a distributed cluster proposed on the basis of the network management architecture shown;

[0017] Figure 5 It is for Figure 4 The network management architecture shown is a flowchart of another network management method applied to a distributed cluster;

[0018] Figure 6 is Figure 1 A schematic diagram of another network management architecture for a distributed cluster proposed on the basis of the network management architecture shown;

[0019] Figure 7 It is for Figure 6 The network management architecture shown is a flowchart of another network management method applied to a distributed cluster;

[0020] Figure 8 is Figure 2 A flowchart of another network management method applied to a distributed cluster proposed on the basis of the illustrated embodiment;

[0021] Fig. 9 is a schematic diagram of another exemplary network management architecture for a distributed cluster;

[0022] Fig.10 It is for Fig. 9 The network management architecture shown is a flowchart of another network management method applied to a distributed cluster;

[0023] Fig.11 is a block diagram of a network management device applied to a distributed cluster shown in another exemplary embodiment of the present application;

[0024] Fig.12 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0028] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0029] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0030] As mentioned above, a large number of network devices are running simultaneously in the distributed cluster. The environment in which the network devices run is complex, and various faults may occur at any time, such as device hardware failure, network connection failure, etc. In order to ensure the stable operation of the distributed cluster, it is necessary to promptly handle the faulty device when a fault occurs. In the existing technical implementation, the nodes in the distributed cluster are regularly checked for network. The network check method is that the cluster management service calls the check interface of the node device. If the device returns a message normally, it means that there is no fault. If the message is not returned after a timeout, it means that a fault has occurred.

[0031] As the scale of cloud network business development grows, the shortcomings of existing network inspection solutions are becoming more and more obvious. Specifically, in the implementation of cloud network business scenarios, distributed nodes may be deployed in various regions. In some failure scenarios, such as when a network in a certain region is abnormal, the cluster management service's inspection request for all nodes in this region will time out. If the number of timed-out nodes is too large, the timer will block the network inspection of the next cycle, and after a period of time, the entire cluster management service will crash, so the reliable operation of cloud network business cannot be guaranteed.

[0032] It can be understood that cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing. The backend services of the cloud technology network system require a large amount of computing and storage resources, such as video websites, image websites and more portal websites. Cloud network services are also realized with the support of cloud technology.

[0033] To facilitate understanding of the above technical issues, for example, assume that there are 100 nodes in a distributed cluster, and the cluster management service performs network checks on these nodes one by one every minute, that is, the cluster management service initiates calls to the inspection interfaces of these 100 nodes one by one every minute, and the timeout of the network inspection request is 3 seconds. Generally speaking, when the node network is in good condition, the inspection result will be returned to the cluster management service within milliseconds. If 20 nodes in a certain region time out during network inspection due to network anomalies, the total inspection time of this cycle will exceed 60 seconds, and the network inspection task of the next cycle of the timer will be stuck, which will cause the entire cluster management service to crash after a period of time.

[0034] To solve this technical problem, this application proposes a network management architecture for distributed clusters, such as Figure 1 As shown, the cluster management server runs independent timer schedulers and thread pools. When the timing cycle arrives, the timer scheduler obtains the relevant information of each node in the distributed cluster from the database of the distributed cluster, then generates a network inspection task for each node, and sends the generated network inspection task to the thread pool. There are multiple task execution threads running in the thread pool. Exemplarily, the thread pool can be created when the cluster server initializes the system. The number of task execution threads depends on the system resources. Generally speaking, the number of CPU (Central Processing Unit) cores of the cluster management server represents the maximum number of concurrent processes it can handle at the same time. Therefore, the thread pool size can be configured as the number of CPU cores, so that the system concurrency capacity can be maximized. After the thread pool is created, it waits for the addition of the network inspection task, and the task execution thread automatically executes the network inspection task. Since the timer scheduler and the thread pool are independent of each other, even if the network inspection task of some nodes in the distributed cluster is timed out, it will not block the timer scheduler for the scheduled scheduling of the network inspection task, thereby ensuring the reliable operation of the distributed cluster.

[0035] It should be understood that the cluster management server is a device used to provide distributed cluster management services, such as a server device, or a terminal device such as a laptop or computer, which is not limited here. When the cluster management server is deployed as a server, it can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0036] based on Figure 1 As shown in the network management architecture for distributed clusters, an exemplary embodiment of the present application also proposes a network management method applied to distributed clusters, which is executed by a cluster management server deployed in the distributed cluster, and the cluster management server runs independent timer schedulers and thread pools.

[0037] like Figure 2 As shown, the method includes S210-S230, which are described in detail as follows:

[0038] S210, when the timer scheduler detects that the timing period has arrived, a network inspection task is generated for each node in the distributed cluster.

[0039] When each regular cycle arrives, the timer scheduler obtains node information of multiple nodes in the distributed cluster, and then generates network inspection tasks for each node. For example, the node list can be queried from the database to obtain the node information of multiple nodes in the distributed cluster, and then the network inspection tasks are generated for each node in the node list in turn.

[0040] S220, sending the network inspection task to the thread pool, and executing the network inspection task concurrently through multiple task execution threads in the thread pool.

[0041] By sending the network inspection tasks for each node in the distributed cluster to the thread pool in sequence, multiple task execution threads pre-created in the thread pool can concurrently execute the received network inspection tasks. Therefore, the setting of the thread pool can maximize the concurrent capabilities of the cluster management server. The execution of the network inspection tasks is independent of the timer task scheduling. Even if the network inspection tasks of some nodes have execution exceptions, the timer scheduling will not be blocked, thereby ensuring the reliable operation of the distributed cluster.

[0042] It can be understood that, for a task execution thread, a network check task is a function that can be called by the thread, and the parameters of the function include the IP (Internet Protocol, Internet Interconnection Protocol) of the node network detection, port, protocol, timeout period, etc., which are not listed here one by one.

[0043] S230: If it is detected that an execution exception occurs in the task execution thread when executing the network inspection task, an alarm process is performed for the network inspection task with the execution exception.

[0044] If the cluster management server detects that a task execution thread has an execution exception when executing a network check task, it means that the network status of the corresponding node is abnormal. Therefore, an alarm processing is required for the network check task with the execution exception. For example, an alarm message can be sent to the personnel responsible for operating and maintaining the distributed cluster to notify the operation and maintenance personnel to maintain the corresponding node.

[0045] Exemplarily, the operation and maintenance personnel can also configure a general alarm processing process in the cluster management server to realize the automation of alarm processing. For example, the restart interface of each node in the distributed cluster is preset in the cluster management service, so that the cluster management server can try to repair the abnormality of the node by calling the restart interface of the node with abnormal execution. For example, after the node is alarmed multiple times, the cluster management server can call the restart interface to try to restart the node. However, it should be understood that if the cause of the node abnormality is objective reasons such as hardware and network, calling the restart interface usually cannot solve the abnormality problem.

[0046] From the above, it can be concluded that in the method provided in this embodiment, on the one hand, since the execution of the network inspection task is independent of the timer task scheduling, even if the network inspection task of some nodes has execution exceptions, it will not block the timer scheduling, thereby ensuring the reliable operation of the distributed cluster; on the other hand, alarm processing is also performed for the network inspection task with execution exceptions, thereby realizing network recovery processing of the abnormal node, thereby further ensuring the reliable operation of the distributed cluster.

[0047] In another exemplary embodiment, considering that after all network inspection tasks of each timing cycle are placed in the thread pool, the task execution threads corresponding to each network inspection task are randomly assigned, if some nodes in the distributed cluster are abnormal, the execution speed of the threads where the network inspection tasks of these nodes are located will be blocked. For example, assuming that the network inspection task of a node in the previous cycle is blocked by the predecessor task in the thread, and the network inspection task of this node in the next cycle is also placed in another thread for execution, and the speed of executing the node task by the other thread is faster, it will cause the task of the next cycle of the node to be executed earlier than the task of the previous cycle, for example Figure 3 As shown, the dotted circle in the thread pool represents the network inspection tasks of the previous and next cycles for the same node, and the network inspection task of the previous cycle is placed in the first thread, and the network inspection task of the next cycle is placed in the second thread. Due to the blocking of the predecessor task in the first thread, the time when the first thread executes the network inspection task of the previous cycle is later than the time when the second thread executes the network inspection task of the next cycle. In some business scenarios, some business information will be carried in the parameters of the network inspection task. If the network inspection tasks are executed out of order, business errors may occur.

[0048] In some more coincidental cases, two cycles of network inspection tasks of the same node are executed simultaneously in different task execution threads, and the inspection interface of the node is scheduled by different task execution threads at the same time. However, the node may not be able to handle both schedules at the same time, resulting in failure of both schedules. In fact, the node is operating normally, which obviously affects the reliability of network management for distributed clusters.

[0049] To solve the above problems, a dedicated task execution thread can be created for each node in the thread pool, and the network inspection task of the same node can be put into a fixed task execution thread for execution. However, in the cloud network business scenario, the number of nodes in the distributed cluster is very large. For example, the cloud network system will manage tens of thousands of network devices at the same time. Obviously, the software service cannot create tens of thousands of threads to run simultaneously. Therefore, the present application further proposes that the network inspection task corresponding to the same node can be sent to the same task execution thread in the thread pool, so that the network inspection task of the same node can be executed by the same task execution thread in the thread pool.

[0050] See also Figure 4 ,exist Figure 4 In another network management architecture for a distributed cluster, Figure 1 Based on the network management architecture shown, a distributor program is further run on the cluster management server. The function of the distributor program is to distribute network inspection tasks with the same ID to the same task execution thread for execution according to the ID (identity document) of the network inspection task. The ID of the network inspection task can also be called the task identifier, which can be implemented as the node identifier of the node to which it belongs, or as the sequence number generated by the timer scheduling system when generating the network inspection task. Different network inspection tasks correspond to different sequence numbers.

[0051] It should be understood that the same task execution thread mentioned in this embodiment executes the network inspection task of the same node, which means that one task execution thread can execute the network inspection tasks of multiple nodes, but all network inspection tasks of the same node are specifically executed by the same task execution thread. Figure 4 The thread pool shown in the figure is used to illustrate that the circle contained in each task execution thread represents a network inspection task. The network inspection tasks of the same node have the same grayscale, while the network inspection tasks of different nodes have different grayscales, which are indicated by Figure 4 It can be seen from the network inspection tasks assigned to each thread in the thread pool that each task execution thread can be assigned network inspection tasks of one or more nodes, but the network inspection tasks of the same node are assigned to the same task execution thread.

[0052] based on Figure 4 Another network management architecture for distributed clusters is shown in FIG. 1 . An exemplary embodiment of the present application further proposes the following Figure 5 The network management method applied to distributed clusters is shown in FIG. Figure 5 As shown, this method Figure 2 Based on the embodiment shown, the network inspection task corresponding to the same node is sent to the same task execution thread in the thread pool through steps S510-S520, which is described in detail as follows:

[0053] S510, sending the network detection task to the distributor program.

[0054] In this embodiment, a distributor program is also run in the cluster management server, and the distributor program is used to distribute network inspection tasks corresponding to the same task identifier to the same task execution thread in the thread pool according to the task identifier of each network inspection task.

[0055] S520, in the distributor program, according to the task identifiers of the network inspection tasks, the network inspection tasks corresponding to the same task identifiers are allocated to the same task execution thread in the thread pool.

[0056] Since the total number of task execution threads created in the thread pool is much smaller than the number of network inspection tasks to be executed, in the allocator program, the task identifier of each network inspection task can be hashed to obtain the corresponding hash result value, and then the quotient between each hash result value and the total number of task execution threads in the thread pool is calculated, and the task execution thread that executes each network inspection task is determined based on the remainder obtained.

[0057] Since the hash algorithm can ensure that the same output is obtained for the same input, this embodiment uses a hash operation on the task identifier to determine the task execution thread. It can be understood that the hash algorithm is also called a secure hash algorithm, which can calculate a string of fixed length for a digital message. If the input digital message is different, the string obtained by the operation is also different. Hash algorithms include, for example, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512, etc., and the specific algorithm type is not limited here.

[0058] This embodiment pre-assigns different thread numbers to each task execution thread in the thread pool, and then divides the hash result value corresponding to each network inspection task by the total number of task execution threads in the thread pool, and a remainder can be obtained accordingly. The remainder is also called a modulus value, and the remainder is the thread number of the task execution thread to which the network inspection task is assigned. When each network inspection task is placed in the thread pool, by specifying the task execution thread with the thread number, the network inspection tasks corresponding to the same task identifier can be assigned to the same task execution thread in the thread pool.

[0059] It should also be noted that the dispatcher program needs to distinguish the network inspection tasks from different timing periods of the same node, that is, the dispatcher program needs to distinguish the order of multiple network inspection tasks of the same node, and assign them to the same task execution thread in sequence according to the order of the network inspection tasks, so as to ensure the reliability of network management. Figure 5 The network management method applied to the distributed cluster shown in the figure further includes S530 which is executed cyclically in the distributor program, which is described in detail as follows:

[0060] S530, when it is detected that the receiving task entry is called, a lock mark is added to the receiving task entry, the lock mark indicates that the receiving task entry only receives the currently incoming network inspection task; when it is detected that the currently incoming network inspection task has been assigned, the lock mark is released.

[0061] The receiving task entry of the distributor program means that the timer scheduler sends the network check task to the distributor program by calling the receiving task entry of the distributor program.

[0062] Since the dispatcher program needs to ensure the order of receiving network inspection tasks and distributing network inspection tasks, this embodiment adds a lock identifier to the receiving task entry when it detects that the receiving task entry is called. This can also be understood as locking the function of the receiving task entry of the dispatcher program. When the dispatcher program is in a locked state, only one network detection task is allocated at the same time, until the network detection task is allocated and the lock is released to continue to allocate the next task.

[0063] From this, it can be seen that the dispatcher program includes a process of locking and releasing the lock for the allocation of each network detection task. Since the timer scheduler sends the network inspection tasks for each node to the dispatcher program in sequence when each regular period arrives, the process of receiving and allocating network inspection tasks for each node in the dispatcher program can be guaranteed to be orderly, which further improves the reliability of network management for distributed clusters.

[0064] See also Figure 6 ,exist Figure 6In another network management architecture for a distributed cluster, Figure 1 Based on the network management architecture shown, a message queue is further run on the cluster management server. The function of this message queue is to split the task publishing topic into multiple partitions, each partition can only be consumed by one thread instance, and then use the task identifier of each network inspection task as the message publishing key to publish each network inspection task to each partition, and make the network inspection tasks corresponding to the same message publishing key be published to the same partition. Since the messages published to the same partition are consumed by the same task execution thread, it is possible to send the network inspection tasks corresponding to the same node to the same task execution thread in the thread pool.

[0065] Exemplarily, the above-mentioned message queue can use a Kafka message queue, and create a topic in the Kafka message queue for publishing network inspection tasks. This topic is used as the task publishing topic, and multiple partitions are created for the task publishing topic. Multiple thread instances in the thread pool use the same consumer group to consume the task publishing topic. Kafka's mechanism can ensure that a partition will only be consumed by one thread instance. As the producer of the task, the timer scheduler uses the task identifier as the message publishing key when publishing the network inspection task to the Kafka message queue. Tasks with the same message publishing key will be published to the same partition, which can ensure that the network inspection tasks of the same node are consumed sequentially by the same thread instance. It can be seen from this that the use of Kafka message queues does not require additional software service development to support multi-instance processing.

[0066] It should be understood that this embodiment is not limited to running the Kafka message queue in the cluster management server, and any other message queue type that can support the above message queue function can be used, and this is not limited here.

[0067] based on Figure 6 Another network management architecture for distributed clusters is shown in FIG. 1 . An exemplary embodiment of the present application further proposes the following Figure 7 The network management method applied to distributed clusters is shown in FIG. Figure 7 As shown, this method Figure 2 On the basis of the embodiment shown, the network inspection task corresponding to the same node is sent to the same task execution thread in the thread pool through S710-S720, which is described in detail as follows:

[0068] S710, using the task identifier as a message publishing key, publishing each network detection task to each partition of the task publishing topic, so that the network detection tasks corresponding to the same message publishing key are published to the same partition;

[0069] S720: According to the consumption thread information corresponding to each partition, the network detection task of each partition is transmitted to the corresponding task execution thread.

[0070] It should be noted that this embodiment uses the publish-consume mode provided by the message queue to implement sending the network inspection tasks corresponding to the same node to the same task execution thread in the thread pool. The detailed process has been recorded in the previous embodiments and will not be repeated here.

[0071] In another exemplary embodiment, considering that the cluster management service end performs network inspection on the node by calling the node inspection interface, if the interface returns an exception, or does not return after a timeout, it means that the node is running abnormally at this time. In actual application scenarios, it is relatively rare that the node is still running but the interface returns an exception. This situation is usually caused by the node itself detecting problems such as excessive CPU occupancy, insufficient memory or disk remaining space, etc. The more common abnormal situation is that the node itself is down or the region or computer room where the node is located causes access timeout due to network reasons. In these abnormal situations, although the network management architecture provided by the aforementioned embodiment can ensure that the same node will not be accessed concurrently, it may still block the execution of network inspection tasks of other nodes in the same thread execution queue. For example, the node inspection timeout is set to 30 seconds. When a node in the thread execution queue has the above-mentioned exception, in each timing cycle, the other nodes in the thread execution queue will be delayed by 30 seconds to perform network inspection tasks. After a long time, the thread execution queue will inevitably be full, and subsequent network inspection tasks cannot be inserted, thereby affecting the reliability of the cluster management service.

[0072] To solve this problem, this embodiment proposes that for a node where a check interface call timeout has occurred, the node will be specially processed. Based on this, the embodiment of the present application also proposes another network management method applied to a distributed cluster, such as Figure 8 As shown, this method Figure 2 The exemplary embodiment also includes S810-S830, which are described in detail as follows:

[0073] S810, when an abnormal network inspection task is detected, shorten the call timeout duration of the node corresponding to the abnormal network inspection task, where the call timeout duration represents the maximum response waiting duration allowed for the detection interface of the corresponding node when it is called by the task execution thread; if the call timeout duration of the first target node is shortened to a preset time limit, the call timeout duration of the first target node will no longer be shortened;

[0074] S830, when the number of times the network inspection task corresponding to the second target node is executed abnormally reaches a first threshold, sending the network inspection task corresponding to the second target node to the abnormal thread pool; executing the network inspection task corresponding to the second target node by the abnormal task execution thread in the abnormal thread pool;

[0075] S850, when the number of abnormal executions of the network inspection task corresponding to the third target node reaches a second threshold, the frequency of network inspection on the third target node is reduced by adjusting the scheduling parameters in the timer scheduling program, and the second threshold is greater than the first threshold.

[0076] First of all, it should be noted that in S810-S830, each step represents a special processing method for a node where a check interface call timeout has occurred. In actual application, at least one processing method corresponding to a step can be set according to actual needs. This embodiment does not limit the corresponding processing of each step to be performed simultaneously.

[0077] It should also be noted that the first target node, the second target node, and the third target node mentioned in this embodiment are only used to distinguish the nodes that are specially processed in different steps. In actual application scenarios, the first target node, the second target node, and the third target node can be different nodes or the same nodes, and this embodiment does not limit this.

[0078] In S810, it can be understood that the starting point for setting the timeout period for the interface call is to consider the network environment, so that occasional jitters in network quality will not affect the cluster management service's judgment of the node's network status as much as possible. If a node has an interface call timeout once, the timeout period of the next call of the node will be shortened. If it continues to timeout, it will continue to be shortened until it is shortened to the preset time limit. That is, when an abnormal network inspection task is detected, the call timeout period of the node corresponding to the abnormal network inspection task is shortened. If the call timeout period of the first target node is shortened to the preset time limit, the call timeout period of the first target node will no longer be shortened.

[0079] For ease of understanding, for example, if the initial interface call timeout of a node is 30 seconds, if a node has continuous interface call timeouts, the interface call timeout will be shortened by 5 seconds each time it times out, until it is shortened to the shortest time limit of 10 seconds. It should be understood that the time values ​​listed here are only examples and can be set according to actual needs in actual application scenarios.

[0080] Therefore, by shortening the call timeout duration of the first target node where the interface call timeout occurred, the influence of the abnormal node on other nodes in the thread execution queue can be reduced to a certain extent.

[0081] In S820, it can be understood that for nodes that have been in an abnormal state for a long time, even if the call timeout period is shortened to the shortest time limit, it may still cause congestion in the thread execution queue. For this reason, it is proposed that when the number of times the network inspection task corresponding to the second target node is executed abnormally reaches a first threshold, the network inspection task corresponding to the second target node is sent to the abnormal thread pool, and the network inspection task corresponding to the second target node is executed by the abnormal task execution thread in the abnormal thread pool. The second target node refers to the node that has been in an abnormal state for a long time.

[0082] The above process can also be understood as building a separate abnormal thread pool on the cluster management server to place network inspection tasks for nodes that are in abnormal state for a long time. The size of the abnormal thread pool can usually be set to be 1-2 orders of magnitude smaller than the regular thread pool. Based on this method, it can be ensured that nodes in abnormal state for a long time in the distributed cluster will not affect the network inspection of other normal nodes.

[0083] It should also be noted that the network inspection task corresponding to the same node can be sent to the same abnormal task execution thread in the abnormal thread pool, so that the network inspection task of the same node can be executed by the same abnormal task execution thread, thereby ensuring that in the abnormal thread pool, the network inspection task of the same node is also specifically executed by the same abnormal task execution thread.

[0084] In S830, it can be understood that if the node device has a hardware failure, the network inspection task for this node is always placed in the abnormal thread pool for execution. For the same reason, it will still cause the same thread execution queue in the abnormal thread pool to be blocked. Therefore, this embodiment proposes that when the number of times the network inspection task corresponding to the third target node is executed abnormally reaches the second threshold, the scheduling parameters in the timer scheduler are adjusted to reduce the frequency of network inspection on the third target node. Among them, since the second threshold is greater than the first threshold, the third target node can be understood as a node with a hardware failure or the like.

[0085] It is easy to understand that in the case of hardware failure of the node device, the cluster management server will always perform network inspection on the node and will always perform alarm processing until the operation and maintenance personnel complete the repair of the node and restore the network, or stop the network inspection task of the node. This embodiment reduces the frequency of network inspection for the node, for example, from once every 30 seconds to once every 5 minutes, thereby reducing system consumption and avoiding blocking of abnormal task execution threads.

[0086] It can be seen from the above that in the method provided in this embodiment, a variety of special processing methods are proposed for nodes where the check interface call timeout has occurred. This can solve the problem of thread blocking caused by node network abnormalities from multiple dimensions, which can greatly improve the effectiveness of network management for distributed clusters, and thus improve the reliability of the cluster management server.

[0087] In another exemplary embodiment, considering that when an abnormality occurs in the execution of a network inspection task, the task execution thread in the thread pool will send an alarm message to the operation and maintenance system. The method of sending the alarm message is usually through an interface call. Therefore, it is also possible that the alarm sending timeout may occur due to network or system failures, thereby blocking the execution of subsequent network inspection tasks in the thread pool.

[0088] To this end, this embodiment also proposes Fig. 9 Another network management architecture for distributed clusters shown in the figure further sets a storage module on the cluster management server, stores the alarm notification information in the storage module, and periodically detects the alarm notification information in the storage module and sends the detected alarm notification information in batches to the operation and maintenance system.

[0089] based on Fig.10 The network management architecture provided, this embodiment also proposes another exemplary network management method applied to a distributed cluster, such as Fig.10 As shown, this method Figure 2 Based on the illustrated embodiment, the process of performing alarm processing for executing abnormal network inspection tasks may specifically include S1010-S1020, which are described in detail as follows:

[0090] S1010, generating alarm notification information for the network inspection task that performs abnormality, and saving the alarm notification information to a storage module;

[0091] S1020, regularly detect the alarm notification information in the storage module, and send the detected alarm notification information to the operation and maintenance system.

[0092] In the above process, the storage module can refer to the memory or cache of the cluster management server, which is not limited here. After the alarm notification information is saved in the storage module, another scheduled task cycle is started to detect the unsent alarm notification information in the storage module, and then send it. If there are multiple alarm notification information saved in the storage module, the scheduled task reads them at one time and sends them in batches. After the sending is successful, the sent alarm notification information can also be deleted from the storage module.

[0093] Therefore, the method provided in this embodiment can avoid the problem of blocking the execution of the network inspection task in the thread pool due to a timeout in sending an alarm by setting an asynchronous alarm mechanism.

[0094] See also Fig.11 , Fig.11 is a block diagram of a network management device applied to a distributed cluster, shown in another exemplary embodiment of the present application. The device is configured in a cluster management server deployed in a distributed cluster, and the cluster management server runs independent timer schedulers and thread pools. Fig.11 As shown, an exemplary network management device 1100 applied to a distributed cluster includes:

[0095] The generating module 1110 is configured to generate a network inspection task for each node in the distributed cluster when the timer scheduler detects that the timing period has arrived;

[0096] An execution module 1120 is configured to send the network inspection task to a thread pool, and execute the network inspection task concurrently through multiple task execution threads in the thread pool;

[0097] The alarm module 1130 is configured to perform alarm processing for the network inspection task with execution abnormality if it is detected that the task execution thread has an execution abnormality when executing the network inspection task.

[0098] In the network management device provided in this embodiment, since the timer scheduler and the thread pool are independent of each other in the cluster management server, when the distributed cluster is implemented as a cloud network service, even if the task execution threads in the thread pool have timeout exceptions when performing network checks on some nodes, it will not block the timer scheduler from scheduling network inspection tasks, thereby ensuring the reliable operation of the cloud network service.

[0099] In another exemplary embodiment, the execution module 1120 is configured to send the network inspection tasks corresponding to the same node to the same task execution thread in the thread pool, so that the network inspection tasks of the same node are executed by the same task execution thread in the thread pool.

[0100] In another exemplary embodiment, a distributor program is also running in the cluster management server; the execution module 1120 includes:

[0101] a sending unit configured to send the network detection task to the distributor program;

[0102] The allocating unit is configured to allocate the network inspection tasks corresponding to the same task identifier to the same task execution thread in the thread pool according to the task identifier of each network inspection task in the allocator program.

[0103] In another exemplary embodiment, the allocation unit is configured to: perform a hash operation on the task identifier of each network inspection task to obtain a corresponding hash result value; determine the total number of task execution threads in the thread pool, and determine the corresponding task execution thread according to the remainder obtained by calculating the quotient between each hash result value and the total number.

[0104] In another exemplary embodiment, the timer scheduler sends the network check task to the distributor program by calling the receiving task entry of the distributor program; the execution module 1120 also includes a loop unit, which is configured to loop the following steps in the distributor program:

[0105] When it is detected that the receiving task entry is called, a lock mark is added to the receiving task entry, and the lock mark indicates that the receiving task entry only receives the currently incoming network inspection task;

[0106] The lock flag is released when it is detected that the current incoming network check task has completed allocation.

[0107] In another exemplary embodiment, a message queue is also running in the cluster management server, and the message queue contains a task publishing topic, which is composed of multiple partitions, and each partition is consumed by only one thread instance; the execution module 1120 includes:

[0108] The publishing unit is configured to use the task identifier as a message publishing key to publish each network detection task to each partition of the task publishing topic, so that the network detection tasks corresponding to the same message publishing key are published to the same partition;

[0109] The transmission unit is configured to transmit the network detection task of each partition to the corresponding task execution thread according to the consumption thread information corresponding to each partition.

[0110] In another exemplary embodiment, the network management device 1100 applied to a distributed cluster also includes a first processing module, which is configured to: when an abnormal network inspection task is detected, shorten the call timeout of the node corresponding to the abnormal network inspection task, where the call timeout represents the maximum response waiting time allowed for the detection interface of the corresponding node when it is called by the task execution thread; if the call timeout of the first target node is shortened to a preset time limit, the call timeout of the first target node will no longer be shortened.

[0111] In another exemplary embodiment, the network management device 1100 applied to a distributed cluster also includes a second processing module, which is configured to: when the number of abnormal executions of the network inspection task corresponding to the second target node reaches a first threshold, send the network inspection task corresponding to the second target node to the abnormal thread pool; execute the network inspection task corresponding to the second target node through the abnormal task execution thread in the abnormal thread pool.

[0112] In another exemplary embodiment, the second processing module is further configured to: send the network inspection task corresponding to the same node to the same abnormal task execution thread in the abnormal thread pool, so as to execute the network inspection task of the same node through the same abnormal task execution thread.

[0113] In another exemplary embodiment, the network management device 1100 applied to a distributed cluster also includes a third processing module, and the third processing module is configured as follows: when the number of abnormal executions of the network inspection task corresponding to the third target node reaches a second threshold, the frequency of network inspections on the third target node is reduced by adjusting the scheduling parameters in the timer scheduler, and the second threshold is greater than the first threshold.

[0114] In another exemplary embodiment, the alarm module 1130 includes:

[0115] A storage unit configured to generate alarm notification information for a network inspection task that performs abnormality, and save the alarm notification information to a storage module;

[0116] The detection unit is configured to periodically detect the alarm notification information in the storage module and send the detected alarm notification information to the operation and maintenance system.

[0117] It should be noted that the network management device applied to a distributed cluster provided in the above embodiment and the network management method applied to a distributed cluster provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here. In actual applications, the network management device applied to a distributed cluster provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0118] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the network management method applied to a distributed cluster provided in the above-mentioned embodiments.

[0119] Fig.12 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0120] like Fig.12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM) 1203, such as executing the method described in the above embodiment. In the RAM 1203, various programs and data required for system operation are also stored. The CPU 1201, the ROM 1202 and the RAM 1203 are connected to each other through the bus 1204. The input / output (I / O) interface 1205 is also connected to the bus 1204.

[0121] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0122] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1609, and / or installed from a removable medium 1611. When the computer program is executed by a central processing unit (CPU) 1601, various functions defined in the system of the present application are executed.

[0123] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer program contained in the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0124] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0126] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the network management method applied to a distributed cluster as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0127] Another aspect of the present application also provides a computer program product or a computer program, the computer program product or the computer program includes a computer instruction, the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the network management method applied to the distributed cluster provided in each of the above embodiments.

[0128] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A network management method applied to a distributed cluster, characterized in that: The method is executed by a cluster management server deployed in the distributed cluster, wherein a timer scheduler and a thread pool independently of each other are running in the cluster management server, and the method comprises: When the timer scheduler detects that a timing period has arrived, a network inspection task is generated for each node in the distributed cluster; Sending the network inspection task to the thread pool, and concurrently executing the network inspection task through multiple task execution threads in the thread pool; If it is detected that the task execution thread has an execution exception when executing the network inspection task, an alarm process is performed for the network inspection task with the execution exception.

2. The method according to claim 1, characterized in that The sending the network inspection task to the thread pool, and concurrently executing the network inspection task through multiple task execution threads in the thread pool, includes: The network inspection tasks corresponding to the same node are sent to the same task execution thread in the thread pool, so that the network inspection tasks of the same node are executed by the same task execution thread in the thread pool.

3. The method according to claim 2, characterized in that The cluster management server also runs a distributor program; the sending of the network inspection tasks corresponding to the same node to the same task execution thread in the thread pool includes: sending the network detection task to the distributor program; In the distributor program, according to the task identifiers of the various network inspection tasks, the network inspection tasks corresponding to the same task identifiers are allocated to the same task execution thread in the thread pool.

4. The method according to claim 3, characterized in that The allocating the network inspection tasks corresponding to the same task identifier to the same task execution thread in the thread pool according to the task identifiers of the network inspection tasks includes: Perform hash operation on the task identifier of each network inspection task to obtain the corresponding hash result value; The total number of task execution threads in the thread pool is determined, and the corresponding task execution thread is determined according to the remainder obtained by calculating the quotient between each hash result value and the total number.

5. The method according to claim 3, characterized in that: The timer scheduling program sends the network inspection task to the distributor program by calling the receiving task entry of the distributor program; the method also includes the following steps cyclically executed in the distributor program: When it is detected that the receiving task entry is called, a lock mark is added to the receiving task entry, wherein the lock mark indicates that the receiving task entry only receives the currently incoming network inspection task; The lock identifier is released when it is detected that the currently incoming network inspection task has been allocated.

6. The method according to claim 2, characterized in that The cluster management server also runs a message queue, which contains a task publishing topic. The task publishing topic consists of multiple partitions, and each partition is consumed by only one thread instance. The network inspection task corresponding to the same node is sent to the same task execution thread in the thread pool, including: Using the task identifier as a message publishing key, each network detection task is published to each partition of the task publishing topic, so that the network detection tasks corresponding to the same message publishing key are published to the same partition; According to the consumption thread information corresponding to each partition, the network detection task of each partition is transmitted to the corresponding task execution thread.

7. The method according to claim 1, characterized in that The method further comprises: When an abnormal network inspection task is detected, shorten the call timeout duration of the node corresponding to the abnormal network inspection task, wherein the call timeout duration represents the maximum response waiting time allowed for the detection interface of the corresponding node when being called by the task execution thread; If the calling timeout duration of the first target node is shortened to a preset time limit, the calling timeout duration of the first target node will no longer be shortened.

8. The method according to claim 7, characterized in that The method further comprises: When the number of times that the network inspection task corresponding to the second target node is executed abnormally reaches a first threshold, sending the network inspection task corresponding to the second target node to the abnormal thread pool; The network inspection task corresponding to the second target node is executed by the abnormal task execution thread in the abnormal thread pool.

9. The method according to claim 8, characterized in that The executing the network inspection task corresponding to the second target node by the abnormal task execution thread in the abnormal thread pool includes: The network inspection task corresponding to the same node is sent to the same abnormal task execution thread in the abnormal thread pool, so that the network inspection task of the same node is executed by the same abnormal task execution thread.

10. The method according to claim 8, characterized in that The method further comprises: When the number of abnormal executions of the network inspection task corresponding to the third target node reaches a second threshold, the frequency of network inspection on the third target node is reduced by adjusting the scheduling parameters in the timer scheduling program, and the second threshold is greater than the first threshold.

11. The method according to any one of claims 1 to 10, characterized in that: The alarm processing for the abnormal network inspection task includes: Generate alarm notification information for abnormally executed network inspection tasks, and save the alarm notification information to a storage module; The alarm notification information in the storage module is detected regularly, and the detected alarm notification information is sent to the operation and maintenance system.

12. A network management device applied to a distributed cluster, characterized in that: The device is configured on a cluster management server deployed in the distributed cluster, wherein a timer scheduler and a thread pool independently of each other are running on the cluster management server, and the device comprises: A generating module, configured to generate a network inspection task for each node in the distributed cluster when the timer scheduler detects that a timing period has arrived; An execution module, configured to send the network inspection task to the thread pool, and concurrently execute the network inspection task through multiple task execution threads in the thread pool; The alarm module is configured to perform alarm processing on the network inspection task with execution abnormality if it is detected that the task execution thread has an execution abnormality when executing the network inspection task.

13. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the network management method applied to a distributed cluster as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the network management method applied to a distributed cluster according to any one of claims 1 to 11.

Images