Computing node management method based on cloud management platform, and cloud management platform

Abstract

The present application discloses a computing node management method based on a cloud management platform, and a cloud management platform, capable of ensuring the performance of the service operated by a computing node. The method of the present application comprises: when a tenant needs to formulate a management policy for a plurality of computing nodes, the tenant can send, to a management interface provided by a cloud management platform, the management policy formulated for the plurality of computing nodes by the tenant. Then, the cloud management platform can collect, on the basis of the management policy, a start packet and an end packet comprised in a data stream sent by a first computing node among the plurality of computing nodes to a second computing node among the plurality of computing nodes. Next, the cloud management platform can detect, on the basis of information of the start packet and information of the end packet, whether an anomaly occurs in the first computing node during data stream processing, and if an anomaly occurs, notify the tenant of the anomaly occurring in the first computing node, and execute a handling measure on the first computing node to address the anomaly occurring in the first computing node.

Description

A computing node management method based on cloud management platform and cloud management platform

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on March 26, 2024, with application number 202410371815.4 and application name “A computing node management method and cloud management platform based on cloud management platform”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of cloud technology, and in particular to a computing node management method based on a cloud management platform and a cloud management platform. Background Art

[0003] With the rapid development of cloud technology, more and more tenants are choosing to use remote computing clusters provided by cloud vendors to execute their business operations and provide cloud-based services. For example, tenants can use multiple computing nodes provided by cloud vendors to complete neural network model training and application. To ensure the completion of tenant business operations, it is often necessary to monitor multiple computing nodes to identify and resolve performance issues.

[0004] In related technologies, cloud vendors can deploy profiling tools in each computing node in a computing node cluster. The profiling tools can collect various performance indicators of each computing node in real time, and then quickly determine whether each computing node has performance problems. If the profiling tool determines that a computing node has a performance problem, it can notify the operation and maintenance personnel so that the operation and maintenance personnel can solve the performance problem of the computing node in a timely manner, thereby providing stable cloud services to tenants.

[0005] In the above process, since the analysis tool needs to be deployed on the computing node to collect various performance indicators of the computing node when running the business, the analysis tool will not only invade the business run by the computing node, but also occupy certain resources of the computing node, thereby affecting the performance of the business run by the computing node. Summary of the Invention

[0006] The embodiments of the present application provide a computing node management method and a cloud management platform based on a cloud management platform, which can prevent the cloud management platform from intruding into the business run by the computing node and occupying too many resources of the computing node, thereby ensuring the performance of the business run by the computing node.

[0007] A first aspect of an embodiment of the present application provides a computing node management method based on a cloud management platform. The cloud management platform used to implement the method can manage infrastructure that provides cloud services, including multiple computing nodes of tenants. The method includes:

[0008] When a tenant needs to formulate management policies for its multiple computing nodes, the cloud management platform can provide the tenant with a management interface. The tenant can input the management policies formulated by the tenant for its multiple computing nodes into the management interface. Therefore, the cloud management platform can receive the management policies formulated by the tenant for its multiple computing nodes through the management interface. The management policies can be used to indicate the analysis intent set by the tenant for its multiple computing nodes, for example, slow network analysis and / or slow computing analysis for multiple computing nodes.

[0009] After obtaining the tenant's management policy for its multiple computing nodes, the cloud management platform can collect the start message and end message contained in the data flow sent by the first computing node of the multiple computing nodes to the second computing node of the multiple computing nodes, and extract the information of the start message and the information of the end message.

[0010] After obtaining the information of the start message and the end message, the cloud management platform can detect whether there is any abnormality in the process of the first computing node processing the data flow based on this information (for example, there is a slow network situation or slow computing situation in the process of the first computing node processing the data flow, etc.). If it is determined that there is an abnormality in the process of the first computing node processing the data flow, the cloud management platform will prompt the tenant that there is an abnormality in the first computing node, and execute processing measures on the first computing node to resolve the abnormality in the first computing node.

[0011] It can be seen from the above method that the cloud management platform that is physically isolated from multiple computing nodes has a new analysis function. Its analysis object is the start message and end message contained in the data flow sent by the first computing node to the second computing node, rather than the various performance indicators of the first computing node itself. Therefore, the cloud management platform only needs to collect the start message and end message of the data flow in the process of sending the data flow from the first computing node to the second computing node to determine whether there is an abnormality in the first computing node. This new analysis function prevents the cloud management platform from intruding into the business run by the first computing node, nor does it occupy too many resources of the first computing node, thereby ensuring the performance of the business run by the first computing node.

[0012] In one possible implementation, the data stream includes multiple messages, and the multiple messages include a start message and an end message; the information of the start message includes at least one of the following: a five-tuple of the start message, the time when the cloud management platform collects the start message, the sequence number of the start message among the multiple messages, and the message type of the start message; the information of the end message includes at least one of the following: a five-tuple of the end message, the time when the cloud management platform collects the end message, the sequence number of the end message among the multiple messages, and the message type of the end message. In the aforementioned implementation, the information of the start message may include one or more of the five-tuple of the start message (including the network address of the first computing node, the network address of the second computing node, the source port, the destination port, and other information), the time when the cloud management platform collects the start message, the sequence number of the start message among the multiple messages, and the message type of the start message. Similarly, the information of the end message may include one or more of the five-tuple of the end message (including the network address of the first computing node, the network address of the second computing node, the source port, the destination port, etc.), the time when the cloud management platform collects the end message, the sequence number of the end message in multiple messages, and the message type of the end message.

[0013] In one possible implementation, the cloud management platform detects whether there is an abnormality in the process of processing the data flow by the first computing node based on the information of the start message and the information of the end message, including: the cloud management platform determines the duration used by the first computing node to send the data flow to the second computing node based on the time when the cloud management platform collects the start message and the time when the cloud management platform collects the end message; the cloud management platform determines the number of multiple messages based on the sequence number of the start message in the multiple messages and the sequence number of the end message in the multiple messages; the cloud management platform determines the bandwidth used by the first computing node to send the data flow to the second computing node based on the duration and the number of the data flow sent by the first computing node to the second computing node; the cloud management platform detects whether the bandwidth meets the preset conditions to detect whether there is an abnormality in the process of processing the data flow by the first computing node. In the aforementioned implementation, after obtaining the information of the start message and the end message contained in the data stream sent by the first computing node to the second computing node, the cloud management platform can extract from this information the time when the cloud management platform collected the start message, the time when the cloud management platform collected the end message, the sequence number of the start message among the multiple messages contained in the data stream, and the sequence number of the end message among the multiple messages contained in the data stream. Then, the cloud management platform can calculate the time when the cloud management platform collected the start message and the time when the cloud management platform collected the end message, thereby obtaining the duration used by the first computing node to send the data stream to the second computing node. Then, the cloud management platform can also calculate the sequence number of the start message among the multiple messages and the sequence number of the end message among the multiple messages, thereby obtaining the number of multiple messages. Subsequently, based on the duration used by the first computing node to send the data stream to the second computing node and the number of multiple messages, the cloud management platform obtains the bandwidth used by the first computing node to send the data stream to the second computing node. Next, the cloud management platform detects whether the bandwidth used by a computing node to send a data stream to a second computing node meets a preset condition, so as to detect whether there is a slow network situation when the first computing node processes the data stream.

[0014] In one possible implementation, the cloud management platform detects whether the bandwidth meets the preset conditions, including: the cloud management platform determines the used bandwidth of the communication channel between the first computing node and the second computing node based on the bandwidth; the cloud management platform detects whether the used bandwidth is greater than or equal to the maximum bandwidth of the communication channel. In the aforementioned implementation, after obtaining the bandwidth used by the first computing node to send the data stream to the second computing node, the cloud management platform can calculate the used bandwidth of the communication channel between the first computing node and the second computing node based on the bandwidth used by the first computing node to send the data stream to the second computing node. Then, the cloud management platform can detect whether the used bandwidth of the communication channel between the first computing node and the second computing node is greater than or equal to the maximum bandwidth of the communication channel between the first computing node and the second computing node. If it is greater than or equal to, the cloud management platform can determine that the first computing node has a slow network situation in the process of processing the data stream.

[0015] In one possible implementation, the cloud management platform performs processing measures on the first computing node, including: the cloud management platform generates a new communication channel between the first computing node and the second computing node based on the communication channel, and replaces the communication channel with the new communication channel, where the communication channel is specified by the tenant. In the aforementioned implementation, after determining that the first computing node has a slow network condition, the cloud management platform may generate a new communication channel between the first computing node and the second computing node based on the communication channel between the first computing node and the second computing node, and replace the communication channel between the first computing node and the second computing node with the new communication channel between the first computing node and the second computing node, so as to resolve the slow network condition of the first computing node.

[0016] In one possible implementation, multiple computing nodes are used to perform training of a neural network model, and the training includes multiple rounds of iterations, each of which includes a computing phase and a communication phase. The cloud management platform detects whether there is an abnormality in the process of processing a data stream by the first computing node based on the information of the start message and the information of the end message, including: the cloud management platform determines the duration of the first computing node in the computing phase in the current round of iteration based on the time when the cloud management platform collects the start message and the time when the cloud management platform collects the end message, and the data stream is generated by the first computing node in the computing phase of the current round of iteration; the cloud management platform detects whether the duration of the first computing node in the computing phase in the current round of iteration meets a preset condition to detect whether there is an abnormality in the process of processing the data stream by the first computing node. In the above implementation, it is assumed that multiple computing nodes of the tenant are used to perform training of the neural network model, and the training process includes multiple rounds of iterations, each of which includes a computing phase and a communication phase. It should be noted that for any one of the multiple computing nodes, the computing node will be in the computing phase and the communication phase in each round of iteration. For the sake of convenience, the following is a schematic introduction using the data stream generated by the first computing node in the current round of iterations as an example. After obtaining the information of the start message and the end message contained in the data stream sent by the first computing node to the second computing node, the cloud management platform can extract the time when the cloud management platform collected the start message and the time when the cloud management platform collected the end message from this information. Then, the cloud management platform can use the time when the cloud management platform collected the start message and the time when the cloud management platform collected the end message to perform calculations, thereby obtaining the duration of the first computing node in the calculation phase in the current round of iterations. Subsequently, the cloud management platform can detect whether the duration of the first computing node in the calculation phase in the current round of iterations meets the preset conditions to detect whether there is a slow calculation situation in the process of processing the data stream by the first computing node.

[0017] In one possible implementation, the cloud management platform detects whether the duration for which the first computing node is in the computing phase in the current round of iterations satisfies a preset condition, including: the cloud management platform detects whether the difference between the duration for which the first computing node is in the computing phase in the current round of iterations and the duration for which the remaining computing nodes are in the computing phase in the current round of iterations is greater than or equal to a preset first threshold; or, the cloud management platform detects whether the difference between the duration for which the first computing node is in the computing phase in the current round of iterations and the duration for which the first computing node is in the computing phase in the remaining rounds of iterations is greater than or equal to a preset second threshold. In the above-mentioned implementation method, after obtaining the duration of time that the first computing node is in the computing stage in the current round of iterations, the cloud management platform can detect whether the difference between the duration of time that the first computing node is in the computing stage in the current round of iterations and the duration of time that the remaining computing nodes are in the computing stage in the current round of iterations is greater than or equal to a preset first threshold, or the cloud management platform can detect whether the difference between the duration of time that the first computing node is in the computing stage in the current round of iterations and the duration of time that the first computing node is in the computing stage in the remaining rounds of iterations is greater than or equal to a preset second threshold. If so, the cloud management platform can determine that the first computing node has slow computing in the process of processing the data stream.

[0018] In one possible implementation, the cloud management platform performing processing measures on the first computing node includes: the cloud management platform removing the first computing node from the plurality of computing nodes. In the aforementioned implementation, after determining that the first computing node is experiencing slow computing, the cloud management platform removes the first computing node from the tenant's plurality of computing nodes, preventing the first computing node from participating in training the neural network model, thereby resolving the slow computing condition of the first computing node.

[0019] In one possible implementation, the multiple computing nodes include any of the following: a physical server, a virtual machine, a container, a micro virtual machine, and a bare metal server.

[0020] The second aspect of an embodiment of the present application provides a cloud management platform, which is used to manage an infrastructure that provides cloud services. The infrastructure includes multiple computing nodes of tenants. The cloud management platform includes: a receiving module for receiving management policies for multiple computing nodes from tenants through a management interface; a collection module for collecting, based on the management policy, a start message and an end message contained in a data stream sent by a first computing node of the multiple computing nodes to a second computing node of the multiple computing nodes; a processing module for detecting, based on information in the start message and information in the end message, whether there is an abnormality in the process of processing the data stream by the first computing node. If so, the tenant is prompted that the first computing node has an abnormality, and processing measures are executed on the first computing node, where the processing measures are used to resolve the abnormality of the first computing node.

[0021] In one possible implementation, the data stream includes multiple messages, and the multiple messages include a start message and an end message; the information of the start message includes at least one of the following: a five-tuple of the start message, the time when the cloud management platform collects the start message, the sequence number of the start message among the multiple messages, and the message type of the start message; the information of the end message includes at least one of the following: a five-tuple of the end message, the time when the cloud management platform collects the end message, the sequence number of the end message among the multiple messages, and the message type of the end message.

[0022] In one possible implementation, the processing module is used to: determine the duration used by the first computing node to send the data stream to the second computing node based on the time when the cloud management platform collects the start message and the time when the cloud management platform collects the end message; determine the number of multiple messages based on the sequence number of the start message in the multiple messages and the sequence number of the end message in the multiple messages; determine the bandwidth used by the first computing node to send the data stream to the second computing node based on the duration and number of the data stream sent by the first computing node to the second computing node; and detect whether the bandwidth meets preset conditions to detect whether there is any abnormality in the process of the first computing node processing the data stream.

[0023] In one possible implementation, the processing module is configured to: determine a used bandwidth of a communication channel between the first computing node and the second computing node based on the bandwidth; and detect whether the used bandwidth is greater than or equal to a maximum bandwidth of the communication channel.

[0024] In one possible implementation, the processing module is configured to generate a new communication channel between the first computing node and the second computing node based on the communication channel, and replace the communication channel with the new communication channel, where the communication channel is specified by the tenant.

[0025] In one possible implementation, multiple computing nodes are used to perform training of a neural network model, and the training includes multiple rounds of iterations, each round of iteration includes a computing phase and a communication phase, and a processing module is used to: determine the duration of time that the first computing node is in the computing phase in the current round of iteration based on the time when the cloud management platform collects the start message and the time when the cloud management platform collects the end message, and the data stream is generated by the first computing node in the computing phase of the current round of iteration; detect whether the duration of time that the first computing node is in the computing phase in the current round of iteration meets preset conditions, so as to detect whether there is any abnormality in the process of the first computing node processing the data stream.

[0026] In one possible implementation, the processing module is used to: detect whether the difference between the length of time the first computing node is in the computing phase in the current round of iterations and the length of time the remaining computing nodes are in the computing phase in the current round of iterations is greater than or equal to a preset first threshold; or, detect whether the difference between the length of time the first computing node is in the computing phase in the current round of iterations and the length of time the first computing node is in the computing phase in the remaining rounds of iterations is greater than or equal to a preset second threshold.

[0027] In a possible implementation, the processing module is configured to remove the first computing node from the plurality of computing nodes.

[0028] In one possible implementation, the multiple computing nodes include any of the following: a physical server, a virtual machine, a container, a micro virtual machine, and a bare metal server.

[0029] A third aspect of an embodiment of the present application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory: the memory is used to store instructions; the processor is used to enable the computing device cluster to execute the method described in the first aspect or any possible implementation method of the first aspect according to the instructions.

[0030] A fourth aspect of an embodiment of the present application provides a computer storage medium, which stores one or more instructions. When the instructions are executed by one or more computers, the one or more computers implement the method described in the first aspect or any possible implementation method of the first aspect.

[0031] A fifth aspect of the embodiments of the present application provides a computer program product, which stores instructions. When the instructions are executed by a computer, the computer implements the method described in the first aspect or any possible implementation method of the first aspect.

[0032] In an embodiment of the present application, when a tenant needs to formulate a management policy for its multiple computing nodes, the tenant can send the management policy formulated by the tenant for the multiple computing nodes to the management interface provided by the cloud management platform. Then, based on the management policy, the cloud management platform can collect the start message and the end message contained in the data stream sent by the first computing node of the multiple computing nodes to the second computing node of the multiple computing nodes. Then, based on the information of the start message and the information of the end message, the cloud management platform can detect whether there is an abnormality in the process of processing the data stream by the first computing node. If it is determined that there is an abnormality in the process of processing the data stream by the first computing node, the cloud management platform will prompt the tenant that there is an abnormality in the first computing node, and perform processing measures on the first computing node to resolve the abnormality of the first computing node. In the above process, the cloud management platform that is physically isolated from the computing nodes has a new analysis function. Its analysis object is the start message and end message contained in the data stream sent by the first computing node to the second computing node, rather than the various performance indicators of the first computing node itself. Therefore, the cloud management platform only needs to collect the start message and end message of the data stream in the process of sending the data stream from the first computing node to the second computing node to determine whether there is an abnormality in the first computing node. This new analysis function prevents the cloud management platform from intruding into the business run by the first computing node, nor does it occupy too many resources of the first computing node, thereby ensuring the performance of the business run by the first computing node. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a schematic diagram of the structure of a cloud service system provided in an embodiment of the present application;

[0034] FIG2 is a schematic diagram of the structure of a cloud management platform provided in an embodiment of the present application;

[0035] FIG3 is a schematic diagram of the structure of a computing node cluster provided in an embodiment of the present application;

[0036] FIG4 is a schematic diagram of an observation system and an analysis system provided in an embodiment of the present application;

[0037] FIG5 is another schematic diagram of the observation system and analysis system provided in an embodiment of the present application;

[0038] FIG6 is another schematic diagram of the observation system and analysis system provided in an embodiment of the present application;

[0039] FIG7 is another schematic diagram of the observation system and the analysis system provided in an embodiment of the present application;

[0040] FIG8 is a flow chart of a computing node management method based on a cloud management platform according to an embodiment of the present application;

[0041] FIG9 is another schematic diagram of the structure of a computing node cluster provided in an embodiment of the present application;

[0042] FIG10 is another schematic diagram of the structure of the cloud management platform provided in an embodiment of the present application;

[0043] FIG11 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0044] FIG12 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0045] FIG13 is a schematic diagram of computer devices in a computer cluster provided by an embodiment of the present application being connected via a network. DETAILED DESCRIPTION

[0046] The embodiments of the present application provide a computing node management method and a cloud management platform based on a cloud management platform, which can prevent the cloud management platform from intruding into the business run by the computing node and occupying too many resources of the computing node, thereby ensuring the performance of the business run by the computing node.

[0047] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0048] With the rapid development of cloud technology, more and more tenants are choosing to use remote computing clusters provided by cloud vendors to execute their business operations and provide cloud-based services. For example, tenants can use multiple computing nodes provided by cloud vendors to complete neural network model training and application. To ensure the completion of tenant business operations, it is often necessary to monitor multiple computing nodes to identify and resolve performance issues.

[0049] In related technologies, cloud vendors can deploy analysis tools in each computing node in a computing node cluster. The analysis tools can collect various performance indicators of each computing node in real time, and then quickly determine whether each computing node has performance problems. If the analysis tool determines that a computing node has a performance problem, it can notify the operation and maintenance personnel so that the operation and maintenance personnel can solve the performance problem of the computing node in a timely manner, thereby providing stable cloud services to tenants.

[0050] In the above process, since the analysis tool needs to be deployed on the computing node to collect various performance indicators of the computing node when running the business, the analysis tool will not only invade the business run by the computing node, but also occupy certain resources of the computing node, thereby affecting the performance of the business run by the computing node.

[0051] Furthermore, the analysis tool mainly collects the performance indicators of computing nodes through sampling, and only uses the performance indicators at certain moments as the analysis object. It cannot represent and analyze the performance of the computing nodes over a period of time, resulting in low analysis precision and accuracy for the computing nodes.

[0052] Furthermore, analysis tools are deployed on computing nodes, and the processor models of the computing nodes are not necessarily compatible with the analysis tools, which often leads to various maintenance problems and results in high maintenance costs for the computing nodes.

[0053] Furthermore, analysis tools have high requirements for detailed information about business operations on computing nodes, and often require tenants to provide specific business parameters and specific data to analyze the performance of computing nodes, which will result in a certain loss of tenants' business privacy.

[0054] To address the above issues, the present application provides a computing node management method based on a cloud management platform. This method can be implemented through a cloud service system. Figure 1 is a schematic diagram of the structure of the cloud service system provided by the present application. As shown in Figure 1, the cloud service system includes an infrastructure that can provide cloud services and a cloud management platform that manages the infrastructure. The cloud management platform and infrastructure are introduced separately below:

[0055] The cloud management platform can coordinate the management of the infrastructure in the entire cloud service system (for example, in the infrastructure, according to the tenant's instructions, a computing node cluster dedicated to the tenant is created for the tenant. The cluster can contain multiple computing nodes, and multiple computing nodes are used to execute the tenant's target business). Among them, the cloud management platform can authenticate the tenant's client through the login interface, and after successful authentication, the tenant's client can be allowed to log in to the cloud management platform. For example, the cloud management platform can also allow the tenant's client to send the management policy customized by the tenant for multiple computing nodes to the cloud management platform through the management interface. The management policy can be used to indicate the tenant's analysis (profiling) intention for multiple computing nodes, for example, slow computing analysis for multiple computing nodes and slow network analysis for multiple computing nodes, etc. Then, the cloud management platform can collect the start message and end message contained in the data flow sent by the source computing node to the destination computing node in multiple computing nodes in real time according to the management policy customized by the tenant for multiple computing nodes, and then analyze whether there is any abnormality in the source computing node in the process of processing the data flow based on the information of the start message and the information of the end message (for example, whether there is a slow computing situation or a slow network situation, etc.). If it is determined that there is an abnormality in the source computing node in the process of processing the data flow, the cloud management platform can execute corresponding processing measures for the source computing node to resolve the abnormality of the source computing node.

[0056] Specifically, the cloud management platform may include two major modules: an observation system and an analysis system, as shown in Figure 2 (Figure 2 is a structural diagram of the cloud management platform provided in an embodiment of the present application). For multiple computing nodes of a tenant, the observation system can monitor the data flow sent from the source computing node to the destination computing node in real time, and mirror the start message and the end message of the data flow to itself, and then extract the information of the start message and the information of the end message, and provide the information of the start message and the information of the end message to the analysis system. The analysis system can detect whether there is an abnormality in the process of processing the data flow by the source computing node based on the information of the start message and the information of the end message. If so, the analysis system can prompt the tenant through the management interface that there is an abnormality in the source computing node, and execute corresponding processing measures on the source computing node to resolve the abnormality of the source computing node. The internal workflow of the cloud management platform will not be expanded here.

[0057] The infrastructure includes multiple computing nodes of the tenant, and any two computing nodes among these multiple computing nodes can be directly connected or indirectly connected to each other, so there is a certain communication channel between any two computing nodes. The communication channel between these multiple computing nodes is usually set by the tenant when the cloud management platform creates these multiple computing nodes for the tenant. That is, the cloud management platform can create multiple tenant-specific computing nodes for the tenant under the tenant's computing node creation request, and connect these multiple computing nodes to build a communication channel between any two computing nodes among these multiple computing nodes. For example, as shown in Figure 3 (Figure 3 is a structural diagram of a computing node cluster provided by an embodiment of the present application), it is assumed that the cloud management platform creates multiple computing nodes for the tenant, and the multiple computing nodes can be connected to each other through access switches and aggregation switches. That is to say, for any two computing nodes among the multiple computing nodes, the two computing nodes can be connected through an access switch, a aggregation switch and another access switch (of course, the two computing nodes can be connected through an access switch). Then, these switches between the two computing nodes form a communication channel between the two computing nodes.

[0058] Since a communication channel is built between any two of these multiple computing nodes, any two computing nodes can transmit data streams through their communication channels, and the cloud management platform can monitor the data streams transmitted between any two computing nodes in real time, thereby determining whether there are any anomalies in the process of computing nodes processing data streams.

[0059] Furthermore, for the observation system and analysis system included in the cloud management platform, the observation system and analysis system can also be presented in a variety of ways. For example, as shown in Figure 4 (Figure 4 is a schematic diagram of the observation system and analysis system provided in an embodiment of the present application), the analysis system and the observation system can be different functional modules (or virtual instances, such as virtual machines, containers, and micro virtual machines, etc.) in the same physical server, and the physical server is the cloud management platform. For example, as shown in Figure 5 (Figure 5 is another schematic diagram of the observation system and analysis system provided in an embodiment of the present application), the analysis system and the observation system can be two different physical servers respectively, and the combination of these two physical servers is the cloud management platform. For example, as shown in Figure 6 (Figure 6 is another schematic diagram of the observation system and analysis system provided in an embodiment of the present application), the observation system is a physical server, and the analysis system can be a virtual instance on another (or some) physical server, and the combination of these physical servers is the cloud management platform. For example, as shown in Figure 7 (Figure 7 is another schematic diagram of the observation system and analysis system provided in an embodiment of the present application), the observation system is a combination of certain physical servers, and the analysis system can be a virtual instance on certain physical servers, and the combination of these physical servers is the cloud management platform, and so on.

[0060] Furthermore, in the tenant's computing node cluster, any computing node can also be regarded as a cloud instance in the infrastructure, and the cloud instance can be presented in a variety of ways. For example, the cloud instance can be a physical server selected by the cloud management platform. For another example, the cloud instance can be a bare metal server selected by the cloud management platform. For another example, the cloud instance can be a virtual machine (VM) created by the cloud management platform on the physical server through virtualization technology. For another example, the cloud instance can also be a container (docker) created by the cloud management platform on the physical server through virtualization technology. For another example, the cloud instance can also be a micro virtual machine (microVM) created by the cloud management platform on the physical server through virtualization technology, and so on.

[0061] Furthermore, in a tenant's computing node cluster, multiple computing nodes can be deployed in the same site or different sites. The site can be presented in various forms. For example, the site can be a region in the infrastructure, or an availability zone in the infrastructure, or a data center (DC) in the infrastructure, or a room in the infrastructure, or a cabinet in the infrastructure, etc.

[0062] Based on the above cloud service system, it can be seen that the cloud management platform can receive management policies for multiple computing nodes sent by tenants. Based on this management policy, the cloud management platform can mirror the start and end messages contained in the data flow sent by the source computing node to the destination computing node among the multiple computing nodes. Based on the information of the start and end messages, the cloud management platform can analyze whether there are any anomalies in the source computing node's data flow processing process. If there are any anomalies, the cloud management platform can take appropriate measures on the source computing node to resolve the anomaly. Thus, the cloud management platform, which is physically isolated from the computing nodes, has a new analysis function. Its analysis object is the start and end messages contained in the data flow sent by the source computing node to the destination computing node, rather than the various performance indicators of the source computing node itself. Therefore, the cloud management platform only needs to collect the start and end messages of the data flow during the process of sending the data flow from the source computing node to the destination computing node to determine whether there are any anomalies in the source computing node. This new analysis function prevents the cloud management platform from intruding on the business running on the source computing node and does not occupy too many resources of the source computing node, thus ensuring the performance of the business running on the source computing node. To further understand this process, the following further describes this process in conjunction with FIG8 . FIG8 is a flow chart of a computing node management method based on a cloud management platform provided in an embodiment of the present application. As shown in FIG8 , the method can be implemented by the cloud service system shown in FIG1 . The cloud service system includes an infrastructure that provides cloud services to tenants and a cloud management platform that manages these infrastructures. These infrastructures include multiple computing nodes of the tenants. These multiple computing nodes run the tenant's applications and can be used to execute the tenant's target business. The method includes:

[0063] 801. The cloud management platform receives management policies for multiple computing nodes from a tenant through a management interface.

[0064] In this embodiment, when a tenant needs to formulate a management policy for its multiple computing nodes, the cloud management platform can provide a management interface to the tenant's client (for example, a management policy input bar on the tenant interface, etc.), and the tenant inputs the management policy formulated by the tenant for its multiple computing nodes into the management interface through its client, so the cloud management platform can receive the management policy formulated by the tenant for its multiple computing nodes through the management interface. The management policy can be used to indicate the analysis intention set by the tenant for its multiple computing nodes (for example, slow network analysis and / or slow computing analysis for multiple computing nodes), so after the cloud management platform parses the management policy, it can obtain the tenant's analysis intention for its multiple computing nodes, which triggers the cloud management platform's new analysis function, and performs subsequent analysis on the tenant's multiple computing nodes based on the new analysis function to complete the tenant's analysis intention for its multiple computing nodes.

[0065] For example, as shown in Figures 2 and 3, assuming that a tenant needs to formulate a management policy for its computing node cluster, the tenant can first log in to the cloud management platform. The cloud management platform can provide the tenant with a tenant interface, which can include a management policy input field. Then, the tenant can input the management policy formulated by the tenant for its computing node cluster into the management policy input field. The management policy includes the identifier of the tenant's computing node cluster (for example, 12344455) and the tenant's analysis intention for the computing node cluster (for example, slow network analysis and slow computing analysis). After obtaining and parsing the management policy, the cloud management platform can determine that the tenant needs to perform slow network analysis and slow computing analysis on its computing node cluster based on the identifier of the computing node and the analysis intention for the computing node cluster.

[0066] Then, the cloud management platform can start its own new analysis function. First, the observation system of the cloud management platform can first deploy the network configuration to the access switch in the computing node cluster. For the sake of convenience, the following is a schematic introduction using an access switch in the computing node cluster as an example. Assume that the internet protocol (IP) of the access switch is 26.71.102.199, and the IP of the observation system of the cloud management platform is 215.1.1.1. For slow network analysis and slow computing analysis, the analysis objects of the cloud management platform are the start message (rocev2 write_first) and end message (rocev2 write_last) of the data flow, so the observation system of the cloud management platform can deploy the following network configuration to the access switch:

[0067] mirror rocev2 write_first rocev2 write_last from 26.71.102.199 to 215.1.1.1.

[0068] Based on the above, it can be seen that for the access switch, since the observation system of the cloud management platform enables the access switch to have the above configuration, the access switch will monitor the rocev2 data stream sent by all computing nodes connected to itself, and mirror the starting message rocev2 write_first and the ending message rocev2 write_last in the data stream to the observation system of the cloud management platform. Therefore, the subsequent observation system and analysis system can perform slow network analysis and slow computing analysis on all computing nodes connected to the access switch based on these messages.

[0069] For the remaining access switches of the computing node cluster, the cloud management platform can also perform similar operations on the remaining access switches to deploy corresponding network configurations in the remaining access switches, which will not be repeated here.

[0070] 802. The cloud management platform collects, based on the management policy, a start message and an end message included in a data flow sent by a first computing node of the multiple computing nodes to a second computing node of the multiple computing nodes.

[0071] After obtaining the tenant's management policy for its multiple computing nodes, the cloud management platform starts its own new analysis function, so the cloud management platform can collect the start message and end message contained in the data flow sent by the first computing node of the multiple computing nodes (that is, a computing node among the multiple computing nodes, which serves as the source computing node) to the second computing node of the multiple computing nodes (that is, another computing node among the multiple computing nodes, which serves as the destination computing node).

[0072] It should be noted that the data stream sent by the first computing node to the second node may include at least one communication between the first computing node and the second node, and each communication between the first computing node and the second node may include multiple messages. The multiple messages of any communication between the first computing node and the second node all include the start message of the communication and the end message of the communication.

[0073] After obtaining the start message and end message contained in the data stream sent by the first computing node to the second node, the cloud management platform can extract the information of the start message and the end message, and use this information in the subsequent analysis to see whether there are any abnormalities in the process of processing the data stream by the first computing node, that is, whether there is a slow computing situation and / or a slow network situation, etc.

[0074] Specifically, the information of the start message may include one or more of the following information: a quintuple of the start message, the time when the cloud management platform collects the start message, the sequence number of the start message among multiple messages, and the message type of the start message. Similarly, the information of the end message may include one or more of the following information: a quintuple of the end message, the time when the cloud management platform collects the end message, the sequence number of the end message among multiple messages, and the message type of the end message.

[0075] Still using the above example, assuming that the observation system collects a certain start message rocev2 write_first, then the observation system can extract the message fields of the start message, as shown in Table 1:

[0076] Table 1

[0077] The observation system aggregates the fields and arrival times of multiple extracted messages and sends them to the analysis system. The aggregated data is as follows:

[0078] Where NUM indicates the number of messages included, Length is the total length of these message information, followed by the quintuple of each message, the sequence number of each message, the type of each message, and the time when each message arrived at the observation system (that is, the time when each message was collected by the aforementioned cloud management platform), as shown in Table 2:

[0079] Table 2

[0080] After receiving the data sent by the analysis system, the observation system can parse the message information contained therein, as shown in Table 3.

[0081] Table 3

[0082] The analysis system aggregates the message information with the same five-tuple into the same entry in the Rocev2 flow table. For message 1, since there is no entry matching the five-tuple in the Rocev2 flow table, the analysis system creates a new entry in the Rocev2 flow table and records the message type, message sequence number, and message arrival time in the first data group of Value, as shown in data group 1 in Table 4:

[0083] Table 4

[0084] For message 2, since there is a table entry in the Rocev2 flow table that matches the five-tuple, the analysis system records the message type, message sequence number, and message arrival time in message 2 in the second data group of Value in this table entry, such as data group 2 in Table 9.

[0085] It can be seen that the pair of start and end messages corresponding to messages 1 and 2 have the same five-tuple. This pair of start and end messages comes from the same data stream, which is sent from a source computing node (whose IP is 1.1.1.1) to a destination computing node (whose IP is 2.2.2.2).

[0086] 803. Based on the information of the start message and the information of the end message, the cloud management platform detects whether there is an abnormality in the process of processing the data flow by the first computing node. If so, it prompts the tenant that there is an abnormality in the first computing node and executes processing measures on the first computing node. The processing measures are used to resolve the abnormality in the first computing node.

[0087] After obtaining the information of the start message and the end message contained in the data stream sent by the first computing node to the second computing node, the cloud management platform can detect whether there is an abnormality in the process of processing the data stream by the first computing node based on this information (for example, there is a slow network situation or slow computing situation in the process of processing the data stream by the first computing node, etc.). If it is determined that there is an abnormality in the process of processing the data stream by the first computing node, the cloud management platform will prompt the tenant that there is an abnormality in the first computing node, and execute processing measures on the first computing node to resolve the abnormality of the first computing node.

[0088] Specifically, the cloud management platform can detect whether the first computing node has an abnormality in the following ways:

[0089] (1) Since the data stream sent by the first computing node to the second computing node includes at least one communication, the following is a schematic introduction based on a certain communication included in the data stream. After obtaining the information of the start message and the end message included in the communication between the first computing node and the second computing node, the cloud management platform can extract from this information the time when the cloud management platform collected the start message of the communication, the time when the cloud management platform collected the end message of the communication, the sequence number of the start message of the communication among the multiple messages of the communication, and the sequence number of the end message of the communication among the multiple messages of the communication.

[0090] The cloud management platform can then calculate the time at which the cloud management platform collected the start message of the communication and the time at which the cloud management platform collected the end message of the communication, thereby obtaining the duration of the communication between the first computing node and the second computing node. The cloud management platform can then also calculate the sequence number of the start message of the communication within the multiple messages of the communication and the sequence number of the end message of the communication within the multiple messages of the communication, thereby obtaining the number of the multiple messages of the communication.

[0091] The cloud management platform then obtains the bandwidth used for the communication between the first computing node and the second computing node based on the duration of the communication between the first computing node and the second computing node and the number of packets in the communication. The cloud management platform then performs similar operations for the remaining communications included in the data stream sent by the first computing node to the second computing node, thereby obtaining the bandwidth used for each communication included in the data stream sent by the first computing node to the second computing node.

[0092] Next, the cloud management platform detects whether the bandwidth used for any communication contained in the data stream sent by the first computing node to the second computing node meets the preset conditions, so as to detect whether there is a slow network situation when the first computing node processes the data stream.

[0093] As in the above example, the analysis system integrates a batch of message information. The integration results are shown in Table 5:

[0094] Table 5

[0095] Based on the information of each of the above messages, the analysis system can further obtain the following information:

[0096] (1) The amount of each communication in the data flow (Table 5): that is, the number of messages in a certain communication.

[0097] (1.1) Data stream 1: In the first communication, the sequence number of the end message (type 8) is 199, the sequence number of the start message is 100, and the amount of this communication is 100 messages (=199-100+1).

[0098] (1.2) Data stream 1: In the second communication, the sequence number of the ending message is 299, the sequence number of the starting message is 200, and the amount of this communication is 100 messages (=299-200+1).

[0099] (1.3) Data stream 2: In the first communication, the message sequence number of the end message is 199, and the sequence number of the start message is 100. The amount of this communication is 100 messages (=199-100+1).

[0100] (1.4) Data Stream 2: In the second communication, the sequence number of the ending message is 599, and the sequence number of the starting message is 200. The amount of this communication is 400 messages (=599-200+1).

[0101] (1.5) Data Stream 2: In the third communication, the sequence number of the ending message is 699, and the sequence number of the starting message is 600. The amount of this communication is 100 messages (=699-600+1).

[0102] (2) The duration of each communication (transmission) of the data flow: the difference between the arrival time of the end message and the arrival time of the start message in a communication (Table 5).

[0103] (2.1) Data stream 1: For the first communication, the end message arrives at 3010 μs, the start message arrives at 3000 μs, and the duration of the first communication is 10 μs.

[0104] (2.2) Data stream 1: The second communication, the end message arrives at 3032us, the start message arrives at 3020us, and the duration of the second communication is 12us.

[0105] (2.3) Data stream 2: For the first communication, the end message arrives at 1010 μs, the start message arrives at 1000 μs, and the duration of the first communication is 10 μs.

[0106] (2.4) Data stream 2: The second communication, the end message arrives at 1036us, the start message arrives at 1020us, and the duration of the second communication is 16us.

[0107] (2.5) Data flow 2: The third communication, the end message arrives at 1062us, the start message arrives at 1050us, and the duration of the third communication is 12us.

[0108] (3) The duration of each data flow communication: the difference between the arrival time of the successful message (message fed back from the destination computing node to the source computing node, type 17) and the arrival time of the start message in a communication (Table 5).

[0109] (3.1) In data stream 1: No successful message is received, so the communication completion time cannot be calculated.

[0110] (3.2) In data stream 2: In the first communication, the time for the successful reception message (type 17) to arrive is 1015us, the time for the start message to arrive is 1000us, and the time for the first communication to complete is 15us.

[0111] (3.3) In data stream 2: In the second communication, the time for the successful reception message to arrive is 1047us, the time for the start message to arrive is 1020us, and the time for the second communication to complete is 27us.

[0112] (3.4) In data stream 2: In the third communication, the time for the successful reception message to arrive is 1070us, the time for the start message to arrive is 1050us, and the time for the third communication to complete is 20us.

[0113] (4) Bandwidth used for each communication in a data stream: The bandwidth of a communication in a data stream divided by the time used for that communication multiplied by the length of the message. The length of the message is calculated based on the network MTU of 100 bytes when communicating in Rocev2.

[0114] (4.1) Data Flow 1:

[0115] The bandwidth used for the first communication is: 8 Gb / s = 100 messages / 10 us * 100 bytes * 8 bits.

[0116] The bandwidth used for the second communication is: 6.6 Gb / s = 100 messages / 12 us * 100 bytes * 8 bits.

[0117] (4.2) Data Flow 2:

[0118] The bandwidth used for the first communication is: 8 Gb / s = 100 messages / 10 us * 100 bytes * 8 bits.

[0119] The bandwidth used for the second communication is: 20 Gb / s = 400 messages / 16 us * 100 bytes * 8 bits.

[0120] The bandwidth used for the third communication is: 6.6 Gb / s = 100 messages / 12 us * 100 bytes * 8 bits.

[0121] For ease of explanation, the first communication of data stream 1 in Table 5 is schematically introduced below. After the analysis system determines that the bandwidth used for the first communication of data stream 1 is 8 GB / s, it can be determined whether the bandwidth used for the first communication of data stream 1 meets the preset conditions to determine whether the computing node with IP address 1.1.1.1 has a slow network situation when processing data stream 1.

[0122] (2) Assume that multiple computing nodes of the tenant are used to perform training of the neural network model. The training process includes multiple rounds of iterations. Each round of iteration includes a computing phase and a communication phase. It should be noted that for any computing node among the multiple computing nodes, the computing node will be in the computing phase and the communication phase in each round of iteration. When the computing node is in the computing phase, it will generate a data stream. When the computing node processes the communication phase, it will send the generated data stream to another computing node.

[0123] For the sake of convenience, the data stream generated by the first computing node in the current round of iteration is used as an example for schematic introduction below. Since the data stream contains at least one communication, the communications contained in the data stream are schematically introduced below. After obtaining the information of the start message and the end message contained in each communication between the first computing node and the second computing node, the cloud management platform can extract the time when the cloud management platform collected the start message of each communication and the time when the cloud management platform collected the end message of each communication from this information.

[0124] Then, the cloud management platform can use the time when the cloud management platform collects the start message of each communication and the time when the cloud management platform collects the end message of each communication to calculate, so as to obtain the length of time that the first computing node is in the computing stage in the current round of iteration.

[0125] Subsequently, the cloud management platform may detect whether the duration of the calculation phase of the first computing node in the current round of iterations meets a preset condition, so as to detect whether there is a slow calculation situation in the process of processing the data stream by the first computing node.

[0126] As in the previous example, assume that the tenant's computing node cluster is used to train a neural network model. The training process includes multiple rounds of iterations, and data stream 1 is generated by the computing node with IP address 1.1.1.1 during the computation phase of round 1 iteration, and is sent by the computing node with IP address 1.1.1.1 to the computing node with IP address 2.2.2.2 during the communication phase of round 1 iteration. As shown in Table 5, the analysis system can calculate the duration of the computing node with IP address 1.1.1.1 in the iterative computing phase of round 1 based on the arrival time of the start message of the first communication of data stream 1 (3000us), the arrival time of the end message of the first communication of data stream 1 (3010us), the arrival time of the start message of the second communication of data stream 1 (3020us), the arrival time of the end message of the second communication of data stream 1 (3032us), etc., and judge whether the duration meets the preset conditions to determine whether the computing node with IP address 1.1.1.1 has slow computing in the process of processing data stream 1.

[0127] More specifically, the cloud management platform may detect whether the bandwidth used for any communication included in the data flow sent by the first computing node to the second computing node meets the preset condition in the following manner:

[0128] Since the data stream sent by the first computing node to the second computing node may include at least one communication, the following is a schematic introduction using one of the communications. After obtaining the bandwidth used for the communication included in the data stream sent by the first computing node to the second computing node, the cloud management platform can calculate the used bandwidth of the communication channel between the first computing node and the second computing node based on the bandwidth (since the communication channel can also be used by the remaining computing nodes, the used bandwidth of the communication channel not only includes the bandwidth used for the communication between the first computing node and the second computing node, but also includes the bandwidth used for a certain communication between the remaining computing nodes. The time of a certain communication between the remaining nodes overlaps with the time of the communication between the first computing node and the second computing node).

[0129] The cloud management platform can then detect whether the used bandwidth of the communication channel between the first computing node and the second computing node is greater than or equal to the maximum bandwidth of the communication channel between the first computing node and the second computing node. If so, the cloud management platform can determine that the first computing node has a slow network condition when processing the data stream.

[0130] Still taking the above example, as shown in Figure 9 (Figure 9 is another schematic diagram of the computing node cluster provided by an embodiment of the present application), assuming that the communication channel between the computing node with IP address 1.1.1.1 and the computing node with IP address 2.2.2.2 is T1-S1 (the message with srcport=1111 is forwarded by the aggregation switch S1, and the message with srcport=2222 is forwarded by the aggregation switch S2)-T2, the analysis system determines that the bandwidth used for the first communication of data stream 1 is 8GB / s. Based on this bandwidth, it can be calculated that the used bandwidth of the communication channel is 15GB / s. Since the maximum bandwidth of the communication channel is 15GB / s, the analysis system can determine that the computing node with IP address 1.1.1.1 encounters a slow network situation during the process of processing data stream 1.

[0131] More specifically, the cloud management platform can detect whether the duration of the first computing node in the computing phase in the current round of iterations meets the preset conditions in the following ways:

[0132] (1) After obtaining the duration of the first computing node in the computing phase in the current round of iterations, the cloud management platform detects whether the difference between the duration of the first computing node in the computing phase in the current round of iterations and the duration of the remaining computing nodes in the computing phase in the current round of iterations is greater than or equal to a preset first threshold. If so, the cloud management platform may determine that the first computing node has experienced slow computing during the process of processing the data stream.

[0133] Still taking the above example, after the analysis system determines that the computing node with IP address 1.1.1.1 is in the iterative calculation phase of round 1 for 89us, since the computing node with IP address 2.2.2.2 is in the iterative calculation phase of round 1 for 50us, the computing node with IP address 3.3.3.3 is in the iterative calculation phase of round 1 for 51us, and the computing node with IP address 4.4.4.4 is in the iterative calculation phase of round 1 for 50us, the analysis system can determine that the computing node with IP address 1.1.1.1 has slow calculation in the process of processing data stream 1 (that is, in round 1).

[0134] (2) After obtaining the duration of the first computing node being in the computing phase in the current round of iterations, the cloud management platform detects whether the difference between the duration of the first computing node being in the computing phase in the current round of iterations and the duration of the first computing node being in the computing phase in the remaining rounds of iterations is greater than or equal to a preset second threshold. If so, the cloud management platform may determine that the first computing node is experiencing slow computing during the process of processing the data stream.

[0135] Still taking the above example, after the analysis system determines that the computing node with IP address 1.1.1.1 is in the iterative calculation phase of round 1 for 89us, since the computing node with IP address 1.1.1.1 is in the iterative calculation phase of round 2 for 50us, the computing node with IP address 1.1.1.1 is in the iterative calculation phase of round 3 for 53us, and the computing node with IP address 1.1.1.1 is in the iterative calculation phase of round 2 for 52us, the analysis system can determine that the computing node with IP address 1.1.1.1 has slow calculation in the process of processing data stream 1 (that is, in round 1).

[0136] More specifically, the cloud management platform may perform processing measures on the first computing node in the following ways:

[0137] (1) After determining that the first computing node has a slow network condition, the cloud management platform may generate a new communication channel between the first computing node and the second computing node based on the (original) communication channel between the first computing node and the second computing node, and replace the communication channel between the first computing node and the second computing node with the new communication channel between the first computing node and the second computing node. It should be noted that the (original) communication channel between the first computing node and the second computing node is specified in advance by the tenant.

[0138] Still using the above example, after the analysis system determines that the computing node with IP address 1.1.1.1 encounters a slow network condition while processing data stream 1, it can rebuild the communication channel between the computing node with IP address 1.1.1.1 and the computing node with IP address 2.2.2.2, that is, replace their original communication channel T1-S1-T2 with the new communication channel T1-S2-T2.

[0139] (2) After determining that the first computing node has a slow computing situation, the cloud management platform removes the first computing node from the tenant's multiple computing nodes, so that the first computing node no longer participates in the training of the neural network model.

[0140] As in the previous example, if the analysis system determines that the computing node with IP address 1.1.1.1 is slow in processing data stream 1, it can remove the computing node with IP address 1.1.1.1 from the tenant's computing node cluster, so that it no longer participates in the subsequent training of the neural network model.

[0141] In an embodiment of the present application, when a tenant needs to formulate a management policy for its multiple computing nodes, the tenant can send the management policy formulated by the tenant for the multiple computing nodes to the management interface provided by the cloud management platform. Then, based on the management policy, the cloud management platform can collect the start message and the end message contained in the data stream sent by the first computing node of the multiple computing nodes to the second computing node of the multiple computing nodes. Then, based on the information of the start message and the information of the end message, the cloud management platform can detect whether there is an abnormality in the process of processing the data stream by the first computing node. If it is determined that there is an abnormality in the process of processing the data stream by the first computing node, the cloud management platform will prompt the tenant that there is an abnormality in the first computing node, and perform processing measures on the first computing node to resolve the abnormality of the first computing node. In the above process, the cloud management platform that is physically isolated from the computing nodes has a new analysis function. Its analysis object is the start message and end message contained in the data stream sent by the first computing node to the second computing node, rather than the various performance indicators of the first computing node itself. Therefore, the cloud management platform only needs to collect the start message and end message of the data stream in the process of sending the data stream from the first computing node to the second computing node to determine whether there is an abnormality in the first computing node. This new analysis function prevents the cloud management platform from intruding into the business run by the first computing node, nor does it occupy too many resources of the first computing node, thereby ensuring the performance of the business run by the first computing node.

[0142] Furthermore, in an embodiment of the present application, the cloud management platform can accurately obtain certain quantities of the first computing node in the process of processing the data flow (for example, the bandwidth used for each communication between the first computing node and the second computing node and the length of time that the first computing node is in the computing stage in each round of iteration, etc.) based on the information of the start message and the end message of the data flow. These quantities can represent the performance of the first computing node in the process of processing the data flow and can provide analysis precision and accuracy for the first computing node.

[0143] Furthermore, in an embodiment of the present application, the analysis tool (new analysis function) is deployed on a cloud management platform, and the cloud management platform can be deployed with various types of processors so that the models of these processors are compatible with the analysis tool, thereby avoiding various maintenance problems of the analysis tool and reducing the maintenance cost of the analysis tool.

[0144] Furthermore, in the embodiment of the present application, since the analysis object of the cloud management platform is the start message and the end message contained in the data stream transmitted by the first computing node, the cloud management platform requires detailed information about the business operation on the first computing node, and does not require the tenant to provide specific parameters and specific data of the business to analyze the performance of the computing node, which can protect the tenant's business privacy to a certain extent.

[0145] The above is a detailed description of the computing node management method based on the cloud management platform provided in the embodiment of the present application. The cloud management platform provided in the embodiment of the present application will be introduced below. FIG10 is another structural schematic diagram of the cloud management platform provided in the embodiment of the present application. As shown in FIG10, the cloud management platform is used to manage the infrastructure for providing cloud services. The infrastructure includes multiple computing nodes of tenants. The cloud management platform includes:

[0146] The receiving module 1001 is configured to receive management policies for multiple computing nodes from a tenant through a management interface; for example, the receiving module 1001 may be used to implement step 801 of the embodiment shown in FIG. 8 .

[0147] The collection module 1002 is used to collect the start message and the end message contained in the data flow sent by the first computing node of multiple computing nodes to the second computing node of multiple computing nodes based on the management policy; for example, the collection module 1002 can be used to implement step 802 of the embodiment shown in Figure 8.

[0148] Processing module 1003 is configured to detect, based on the information of the start message and the end message, whether an anomaly exists in the data stream processing process of the first computing node. If so, the first computing node is notified to the tenant of the anomaly and a treatment measure is executed on the first computing node, the treatment measure being used to resolve the anomaly. For example, processing module 1003 may be used to implement step 803 of the embodiment shown in FIG. 8 .

[0149] In one possible implementation, the data stream includes multiple messages, and the multiple messages include a start message and an end message; the information of the start message includes at least one of the following: a five-tuple of the start message, the time when the cloud management platform collects the start message, the sequence number of the start message among the multiple messages, and the message type of the start message; the information of the end message includes at least one of the following: a five-tuple of the end message, the time when the cloud management platform collects the end message, the sequence number of the end message among the multiple messages, and the message type of the end message.

[0150] In one possible implementation, the processing module is used to: determine the duration used by the first computing node to send the data stream to the second computing node based on the time when the cloud management platform collects the start message and the time when the cloud management platform collects the end message; determine the number of multiple messages based on the sequence number of the start message in the multiple messages and the sequence number of the end message in the multiple messages; determine the bandwidth used by the first computing node to send the data stream to the second computing node based on the duration and number of the data stream sent by the first computing node to the second computing node; and detect whether the bandwidth meets preset conditions to detect whether there is any abnormality in the process of the first computing node processing the data stream.

[0151] In one possible implementation, the processing module is configured to: determine a used bandwidth of a communication channel between the first computing node and the second computing node based on the bandwidth; and detect whether the used bandwidth is greater than or equal to a maximum bandwidth of the communication channel.

[0152] In one possible implementation, the processing module is configured to generate a new communication channel between the first computing node and the second computing node based on the communication channel, and replace the communication channel with the new communication channel, where the communication channel is specified by the tenant.

[0153] In one possible implementation, multiple computing nodes are used to perform training of a neural network model, and the training includes multiple rounds of iterations, each round of iteration includes a computing phase and a communication phase, and a processing module is used to: determine the duration of time that the first computing node is in the computing phase in the current round of iteration based on the time when the cloud management platform collects the start message and the time when the cloud management platform collects the end message, and the data stream is generated by the first computing node in the computing phase of the current round of iteration; detect whether the duration of time that the first computing node is in the computing phase in the current round of iteration meets preset conditions, so as to detect whether there is any abnormality in the process of the first computing node processing the data stream.

[0154] In one possible implementation, the processing module is used to: detect whether the difference between the length of time the first computing node is in the computing phase in the current round of iterations and the length of time the remaining computing nodes are in the computing phase in the current round of iterations is greater than or equal to a preset first threshold; or, detect whether the difference between the length of time the first computing node is in the computing phase in the current round of iterations and the length of time the first computing node is in the computing phase in the remaining rounds of iterations is greater than or equal to a preset second threshold.

[0155] In a possible implementation, the processing module is configured to remove the first computing node from the plurality of computing nodes.

[0156] In one possible implementation, the multiple computing nodes include any of the following: a physical server, a virtual machine, a container, a micro virtual machine, and a bare metal server.

[0157] It should be noted that the information interaction, implementation process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.

[0158] Please refer to Figure 11, which is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. As shown in Figure 11, the computing device 1100 (which can be used to present the aforementioned cloud management platform) includes: a processor 1101, a memory 1102, a communication interface 1103, and a bus 1104. The processor 1101, the memory 1102, and the communication interface 1103 are coupled via a bus (not labeled in the figure). The memory 1102 stores instructions. When the execution instructions in the memory 1102 are executed, the computing device 1100 executes the method executed by the cloud management platform in the above method embodiment.

[0159] The computing device 1100 may be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms. For example, when a unit in the device can be implemented in the form of a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0160] The processor 1101 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0161] Memory 1102 may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. Nonvolatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0162] Memory 1102 stores executable program code, and processor 1101 executes the executable program code to implement the functions of the aforementioned receiving module, acquisition module, and processing module, thereby implementing the aforementioned cloud management platform-based computing node management method. In other words, memory 1102 stores instructions for executing the aforementioned cloud management platform-based computing node management method.

[0163] The communication interface 1103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1100 and other devices or a communication network.

[0164] In addition to the data bus, bus 1104 may also include a power bus, a control bus, and a status signal bus. The bus may be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a unified bus (Ubus or UB), a Compute Express Link (CXL), or a Cache Coherent Interconnect for Accelerators (CCIX). Buses can be categorized as address buses, data buses, and control buses.

[0165] Please refer to Figure 12 , which is a schematic diagram of a computing device cluster provided in an embodiment of the present application. As shown in Figure 12 , the computing device cluster 1200 includes at least one computing device 1100 .

[0166] As shown in Figure 12, the computing device cluster 1200 includes at least one computing device 1100. The memory 1102 in one or more computing devices 1100 in the computing device cluster 1200 may store the same instructions for executing the above-mentioned computing node management method based on the cloud management platform.

[0167] In some possible implementations, the memory 1102 of one or more computing devices 1100 in the computing device cluster 1200 may also store partial instructions for executing the aforementioned cloud management platform-based computing node management method. In other words, the combination of one or more computing devices 1100 can jointly execute the aforementioned cloud management platform-based computing node management method.

[0168] It should be noted that the memory 1102 in different computing devices 1100 in the computing device cluster 1200 may store different instructions, each for executing a portion of the functions of the aforementioned cloud management platform. In other words, the instructions stored in the memory 1102 in different computing devices 1100 may implement the functions of one or more modules such as the receiving module, the acquisition module, and the processing module.

[0169] In some possible implementations, one or more computing devices 1100 in the computing device cluster 1200 may be connected via a network, which may be a wide area network or a local area network.

[0170] Please refer to Figure 13, which is a schematic diagram of computer devices in a computer cluster provided by an embodiment of the present application being connected via a network. As shown in Figure 13, two computing devices 1100A and 1100B are connected via a network. Specifically, each computing device is connected to the network via a communication interface.

[0171] In one possible implementation, the memory of the computing device 1100A stores instructions for executing functions of the receiving module and the like. Meanwhile, the memory of the computing device 1100B stores instructions for executing functions of the acquisition module and the processing module and the like.

[0172] It should be understood that the functions of the computing device 1100A shown in Figure 13 may also be completed by multiple computing devices. Similarly, the functions of the computing device 1100B may also be completed by multiple computing devices.

[0173] An embodiment of the present application also relates to a computer storage medium, which stores a program for signal processing. When the program is run on a computer, the computer executes the steps executed by the cloud management platform in the embodiment shown in Figure 8.

[0174] An embodiment of the present application also relates to a computer program product, which stores instructions that, when executed by a computer, enable the computer to execute the steps performed by the cloud management platform in the embodiment shown in FIG8 .

[0175] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0179] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A computing node management method based on a cloud management platform, characterized in that: The cloud management platform is used to manage an infrastructure for providing cloud services, wherein the infrastructure includes multiple computing nodes of tenants. The method includes: The cloud management platform receives management policies for the plurality of computing nodes from the tenant through a management interface; The cloud management platform collects, based on the management policy, a start message and an end message included in a data flow sent by a first computing node of the plurality of computing nodes to a second computing node of the plurality of computing nodes; Based on the information of the start message and the information of the end message, the cloud management platform detects whether there is an abnormality in the process of processing the data flow by the first computing node. If so, it prompts the tenant that there is an abnormality in the first computing node and executes processing measures on the first computing node. The processing measures are used to resolve the abnormality in the first computing node.

2. The method according to claim 1, characterized in that The data stream includes a plurality of messages, and the plurality of messages include the start message and the end message; The information of the start message includes at least one of the following: a quintuple of the start message, a time when the cloud management platform collects the start message, a sequence number of the start message in the multiple messages, and a message type of the start message; The information of the end message includes at least one of the following: a five-tuple of the end message, a time when the cloud management platform collects the end message, a sequence number of the end message in the multiple messages, and a message type of the end message.

3. The method according to claim 1 or 2, characterized in that The cloud management platform detecting, based on the information of the start message and the information of the end message, whether there is an abnormality in the process of the first computing node processing the data flow includes: The cloud management platform determines, based on the time when the cloud management platform collects the start message and the time when the cloud management platform collects the end message, the duration taken by the first computing node to send the data stream to the second computing node; The cloud management platform determines the number of the multiple messages based on the sequence number of the start message in the multiple messages and the sequence number of the end message in the multiple messages; The cloud management platform determines, based on the duration and quantity of the data stream sent by the first computing node to the second computing node, a bandwidth used by the first computing node to send the data stream to the second computing node; The cloud management platform detects whether the bandwidth meets a preset condition to detect whether there is an abnormality in the process of the first computing node processing the data flow.

4. The method according to claim 3, characterized in that The cloud management platform detects whether the bandwidth meets the preset conditions including: The cloud management platform determines, based on the bandwidth, a used bandwidth of a communication channel between the first computing node and the second computing node; The cloud management platform detects whether the used bandwidth is greater than or equal to the maximum bandwidth of the communication channel.

5. The method according to claim 4, characterized in that The cloud management platform executing a processing measure on the first computing node includes: The cloud management platform generates a new communication channel between the first computing node and the second computing node based on the communication channel, and replaces the communication channel with the new communication channel, where the communication channel is specified by the tenant.

6. The method according to claim 1 or 2, characterized in that The multiple computing nodes are used to perform training of a neural network model, the training includes multiple rounds of iterations, each round of iteration includes a computing phase and a communication phase, and the cloud management platform detects whether there is an abnormality in the process of processing the data stream by the first computing node based on information of the start message and information of the end message, including: The cloud management platform determines, based on the time when the cloud management platform collects the start message and the time when the cloud management platform collects the end message, a duration for which the first computing node is in a computing phase in a current round of iterations, wherein the data stream is generated by the first computing node in the computing phase of the current round of iterations; The cloud management platform detects whether the duration of the calculation phase of the first computing node in the current round of iteration meets a preset condition, so as to detect whether there is an abnormality in the process of the first computing node processing the data stream.

7. The method according to claim 6, characterized in that The cloud management platform detects whether the duration of the computing phase of the first computing node in the current round of iteration meets a preset condition, including: The cloud management platform detects whether the difference between the duration of the first computing node in the computing phase in the current round of iterations and the duration of the remaining computing nodes in the computing phase in the current round of iterations is greater than or equal to a preset first threshold; or The cloud management platform detects whether the difference between the length of time the first computing node is in the computing phase in the current round of iterations and the length of time the first computing node is in the computing phase in the remaining rounds of iterations is greater than or equal to a preset second threshold.

8. The method according to claim 7, characterized in that The cloud management platform executing a processing measure on the first computing node includes: The cloud management platform removes the first computing node from the multiple computing nodes.

9. The method according to any one of claims 1 to 8, characterized in that The multiple computing nodes include any one of the following: a physical server, a virtual machine, a container, a micro virtual machine, and a bare metal server.

10. A cloud management platform, characterized in that: The cloud management platform is used to manage the infrastructure for providing cloud services, wherein the infrastructure includes multiple computing nodes of tenants. The cloud management platform includes: A receiving module, configured to receive management policies for the plurality of computing nodes from a tenant through a management interface; a collection module, configured to collect, based on the management policy, a start message and an end message contained in a data flow sent by a first computing node of the plurality of computing nodes to a second computing node of the plurality of computing nodes; A processing module is used to detect whether there is an abnormality in the process of processing the data flow by the first computing node based on the information of the start message and the information of the end message. If so, it prompts the tenant that there is an abnormality in the first computing node and executes processing measures on the first computing node, wherein the processing measures are used to resolve the abnormality in the first computing node.

11. The cloud management platform according to claim 10, characterized in that: The data stream includes a plurality of messages, and the plurality of messages include the start message and the end message; The information of the start message includes at least one of the following: a quintuple of the start message, a time when the cloud management platform collects the start message, a sequence number of the start message in the multiple messages, and a message type of the start message; The information of the end message includes at least one of the following: a five-tuple of the end message, a time when the cloud management platform collects the end message, a sequence number of the end message in the multiple messages, and a message type of the end message.

12. The cloud management platform according to claim 10 or 11, characterized in that: The processing module is used to: Determining, based on the time when the cloud management platform collects the start message and the time when the cloud management platform collects the end message, a duration taken by the first computing node to send the data stream to the second computing node; Determining the number of the plurality of messages based on a sequence number of the start message in the plurality of messages and a sequence number of the end message in the plurality of messages; Determining a bandwidth used by the first computing node to send the data stream to the second computing node based on a duration and the quantity used by the first computing node to send the data stream to the second computing node; Detecting whether the bandwidth meets a preset condition to detect whether there is an abnormality in the process of the first computing node processing the data flow.

13. The cloud management platform according to claim 12, characterized in that: The processing module is used to: Determining a used bandwidth of a communication channel between the first computing node and the second computing node based on the bandwidth; It is detected whether the used bandwidth is greater than or equal to the maximum bandwidth of the communication channel.

14. The cloud management platform according to claim 13, characterized in that: The processing module is configured to generate a new communication channel between the first computing node and the second computing node based on the communication channel, and replace the communication channel with the new communication channel, where the communication channel is specified by the tenant.

15. The cloud management platform according to claim 10 or 11, characterized in that: The plurality of computing nodes are used to perform training of a neural network model, wherein the training includes multiple rounds of iterations, each round of iteration includes a computing phase and a communication phase, and the processing module is used to: Determining, based on a time when the cloud management platform collects the start message and a time when the cloud management platform collects the end message, a duration during which the first computing node is in a computing phase in a current round of iterations, the data stream being generated by the first computing node in the computing phase of the current round of iterations; It is detected whether the duration of the calculation phase of the first computing node in the current round of iteration meets a preset condition, so as to detect whether there is an abnormality in the process of the first computing node processing the data stream.

16. The cloud management platform according to claim 15, characterized in that: The processing module is used to: Detecting whether a difference between a duration that the first computing node is in a computing phase in a current round of iterations and a duration that the remaining computing nodes are in a computing phase in the current round of iterations is greater than or equal to a preset first threshold; or, Detect whether a difference between a duration that the first computing node is in a computing phase in a current round of iterations and a duration that the first computing node is in a computing phase in remaining rounds of iterations is greater than or equal to a preset second threshold.

17. The cloud management platform according to claim 16, characterized in that: The processing module is configured to remove the first computing node from the multiple computing nodes.

18. The cloud management platform according to any one of claims 10 to 17, characterized in that: The multiple computing nodes include any one of the following: a physical server, a virtual machine, a container, a micro virtual machine, and a bare metal server.

19. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device including a processor and a memory: The memory is used to store instructions; The processor is configured to cause the computing device cluster to execute the method according to any one of claims 1 to 9 according to the instructions.

20. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 9.

Images