Network measurement method based on cloud computing technology, cloud management platform and computing device

Abstract

The application belongs to the technical field of cloud computing, and particularly relates to a network measurement method of cloud computing technology, a cloud management platform and a computing device, and is applied to the cloud management platform. The cloud management platform can instruct a switch to monitor a message sent by a target computing node performing a job, and the switch forwards the message to a measurement node when it is determined that the message sent by the target computing node belongs to a message type to be monitored, so that the measurement node analyzes and processes the received message, obtains network measurement indexes between multiple target computing nodes, and sends the network measurement indexes to a tenant. In this way, the network measurement indexes can be obtained without dotting in the target computing node, and the efficiency of the target computing node performing the job can be improved.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a network measurement method, cloud management platform and computing device based on cloud computing technology. Background Technology

[0002] With the development of Artificial Intelligence (AI), tenants have increasingly higher requirements for the visibility of the AI ​​model training process in cloud platforms. This visibility refers to obtaining network metrics (such as packet loss rate and bandwidth) from each computing node in the cloud platform during the training of the AI ​​model, thereby reflecting the training status of the computing nodes on the AI ​​model.

[0003] Currently, visibility of the training process is mainly achieved by tracking data on computing nodes. This involves embedding network measurement metrics monitoring programs into computing nodes, which then run the monitoring programs to obtain the corresponding network measurement metrics and send them to tenants.

[0004] As the scale of computing nodes in cloud platforms increases and the complexity of AI model training tasks becomes higher, the method of making the training process visible by marking points will increasingly consume the processing resources of computing nodes, thereby affecting the computing nodes' ability to execute AI model training. Summary of the Invention

[0005] This application provides a network measurement method, cloud management platform, and computing device based on cloud computing technology, which can achieve visibility into the AI ​​model training process without consuming the processing resources of computing nodes. The corresponding technical solution is as follows:

[0006] Firstly, a network measurement method based on cloud computing technology is provided. This method is applied to a cloud management platform for managing infrastructure, including a cloud data center. The cloud data center includes multiple computing nodes and measurement nodes. The method includes:

[0007] The cloud management platform receives job execution requests from tenants, instructing multiple target compute nodes in the cloud data center to execute jobs. Based on the job execution requests, the platform controls the execution of jobs by the multiple target compute nodes in the cloud data center. The cloud management platform obtains the message types to be monitored specified by the tenant, including at least one of start messages, end messages, waiting messages, and retransmission messages. It determines at least one switch used for data transmission between the multiple target compute nodes. A message monitoring notification is sent to the at least one switch, including the network addresses of the multiple target compute nodes and the message types to be monitored. The message monitoring notification instructs the at least one switch, upon receiving the notification, to forward messages of the message types to be monitored sent by the multiple target compute nodes to the measurement nodes based on their network addresses. The measurement nodes analyze and process the received messages to obtain at least one network measurement indicator between the multiple target compute nodes, and then send this indicator to the tenant.

[0008] A job typically refers to a complete task unit submitted by a tenant to a computer system, including a program, dataset, and operating instructions. Examples include AI training tasks, AI inference tasks, or high-performance computing (HPC) tasks.

[0009] In the network measurement method shown in this application, tenants can customize the message types that the computing nodes need to monitor during job execution on the cloud management platform. The switch then sends the messages corresponding to the monitored message types to the test node, which performs statistical analysis of the network measurement metrics. Thus, the process of achieving visibility into the AI ​​training process in the network measurement method provided in this application does not require consuming the processing resources of the computing nodes, thereby improving the efficiency of the computing nodes in executing jobs.

[0010] In one possible implementation, the cloud management platform obtains the packet types to be monitored specified by the tenant. This includes: the cloud management platform receiving a packet monitoring request sent by the tenant, the packet monitoring request carrying the service level selected by the tenant. Based on the service level selected by the tenant, the platform determines the packet types to be monitored according to the correspondence between service levels and packet types, wherein the number of service levels and corresponding packet types is positively correlated. In this way, tenants can customize packet type monitoring by specifying service levels, thereby improving the flexibility of obtaining network measurement metrics during the execution of jobs on compute nodes.

[0011] In one feasible implementation, the switch is a top-of-rack switch, and the measurement nodes include a network processor switch and a measurement server. The measurement nodes analyze and process received packets to obtain at least one network measurement metric among multiple target computing nodes. This includes: the network processor switch receiving packets forwarded by the top-of-rack switch and sending the corresponding packet information to the measurement server; the packet information including at least one of the following for each type of packet: the number of packets received, the sequence number, the reception time, and a 5-tuple. The measurement server analyzes and processes the received packet information to obtain at least one network measurement metric among the multiple target computing nodes.

[0012] In the network measurement method shown in this application, after receiving the packets to be monitored, the top-of-rack switch can first forward them to the network processor switch, which then sends the corresponding packet information to the measurement server to obtain network measurement indicators. This preprocessing of the packets sent by the top-of-rack switch via the network processor switch before sending the obtained packet information to the measurement server reduces the connection complexity between the network processor switch and the measurement server, and also leverages the processing power of the network processor switch to reduce the complexity of statistical analysis of the packets by the measurement server.

[0013] In one feasible approach, network measurement metrics obtained by analyzing and processing start and end messages include at least one of the following metrics: the number, size, duration, and transmission path of data blocks sent by each target computing node executing the job; data transmission pairs between multiple target computing nodes executing the job, and the data transmission volume of each message transmission queue in each data transmission pair; the proportion of internal and external traffic for each switch among multiple target computing nodes, where internal traffic refers to the amount of data sent by the switch to connected computing nodes, and external traffic refers to the amount of data sent by the switch to other switches; and the traffic proportion of each data transmission path among multiple target computing nodes.

[0014] Network measurement metrics obtained from analyzing and processing wait messages and start messages include: the waiting time before the target computing node sends a data block. Network measurement metrics obtained from analyzing and processing wait messages and end messages include: the waiting time after the target computing node sends a data block.

[0015] Network measurement metrics obtained from analyzing and processing retransmitted messages include the number of lost packets.

[0016] As can be seen, the network measurement method shown in this application provides more monitorable message types, obtains more network measurement indicators, and thus provides more analytical data for the visualization of AI model training.

[0017] In one feasible implementation, the network measurement method provided in this application further includes: a cloud management platform receiving an alarm threshold set by a tenant for at least one network measurement metric, the alarm threshold indicating an anomaly in data transmission between multiple target computing nodes. The cloud management platform sends the alarm threshold corresponding to the at least one network measurement metric to the measurement node. When the measurement node determines that the obtained network measurement metric exceeds the corresponding alarm threshold, it sends an alarm notification to the tenant. In this way, the measurement node can perform alarm processing on network measurement metrics that exceed the alarm threshold set by the tenant, further improving the flexibility of monitoring network measurement metrics.

[0018] In one feasible approach, when the job is data-intensive, the tenant specifies retransmission messages as among the message types to be monitored. When the job is compute-intensive, the tenant specifies waiting messages as among the message types to be monitored. Thus, for data-intensive jobs, monitoring retransmission messages can reflect the data transmission efficiency between compute nodes. When data transmission efficiency is low, technicians can be instructed to optimize the execution of the data-intensive job across multiple compute nodes, thereby improving the execution efficiency of the data-intensive job. For compute-intensive jobs, waiting messages can reflect the waiting time of compute nodes before executing the job. When the waiting time is long, technicians can be instructed to optimize the execution of the compute-intensive job across multiple compute nodes, thereby improving the execution efficiency of the compute-intensive job.

[0019] In one possible implementation, the compute node is a server in a cloud data center, or the compute node is a compute card included in the server, wherein the compute card is a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processor (NPU).

[0020] In one feasible approach, multiple computing nodes transmit data based on the Remote Direct Memory Access (RDMA) protocol, with the start message being a write-first message, the end message being a write-last message, the wait message being a write-only message, and the retransmission message being a NACK message.

[0021] Secondly, a cloud management platform is provided for managing infrastructure, including a cloud data center, which includes multiple computing nodes and measurement nodes.

[0022] The cloud management platform is used to receive job execution requests from tenants, which instruct multiple target compute nodes in the cloud data center to execute jobs. Based on the job execution requests, it controls the execution of jobs by the multiple target compute nodes in the cloud data center. It obtains the message types to be monitored specified by the tenant, including at least one of start messages, end messages, wait messages, and retransmission messages. It identifies at least one switch used for data transmission between the multiple target compute nodes. It sends a message monitoring notification to the at least one switch, which includes the network addresses of the multiple target compute nodes and the message types to be monitored. The message monitoring notification instructs the at least one switch, upon receiving the notification, to forward messages of the message types to be monitored sent by the multiple target compute nodes to the measurement node based on the network addresses of the multiple target compute nodes.

[0023] The measurement node is used to analyze and process the received messages to obtain at least one network measurement indicator among multiple target computing nodes, and to send at least one network measurement indicator to the tenant.

[0024] In one feasible approach, the cloud management platform receives message monitoring requests from tenants, the requests carrying the service level selected by the tenant. Based on the tenant's selected service level, and within a mapping between service levels and message types, the platform determines the message types to be monitored specified by the tenant, where the number of service levels and corresponding message types is positively correlated.

[0025] In one feasible implementation, the switch is a top-of-rack switch, and the measurement nodes include a network processor switch and a measurement server. The network processor switch receives packets forwarded by the top-of-rack switch and sends the corresponding packet information to the measurement server. The packet information includes at least one of the following: the number of packets received, sequence number, reception time, and a 5-tuple, for each packet type. The measurement server analyzes and processes the received packet information to obtain at least one network measurement metric among the multiple target computing nodes.

[0026] In one feasible approach, network measurement metrics obtained by analyzing and processing start and end messages include at least one of the following metrics: the number, size, duration, and transmission path of data blocks sent by each target computing node executing the job; data transmission pairs between multiple target computing nodes executing the job, and the data transmission volume of each message transmission queue in each data transmission pair; the proportion of internal and external traffic for each switch among multiple target computing nodes, where internal traffic refers to the amount of data sent by the switch to connected computing nodes, and external traffic refers to the amount of data sent by the switch to other switches; and the traffic proportion of each data transmission path among multiple target computing nodes.

[0027] In one feasible approach, network metrics obtained by analyzing and processing wait messages and start messages include: the waiting time before the target computing node sends a data block. Network metrics obtained by analyzing and processing wait messages and end messages include: the waiting time after the target computing node sends a data block.

[0028] In one feasible approach, network metrics obtained by analyzing and processing retransmitted messages include the number of packet losses.

[0029] In one possible implementation, the cloud management platform is also used to receive alarm thresholds set by tenants for at least one network measurement metric, which indicates an anomaly in data transmission between multiple target compute nodes. The alarm thresholds corresponding to at least one network measurement metric are sent to the measurement nodes. When it is determined that a network measurement metric exceeds its corresponding alarm threshold, an alarm notification is sent to the tenant.

[0030] In one feasible approach, when the job is data-intensive, the tenant specifies that the message types to be monitored include retransmission messages. When the job is compute-intensive, the tenant specifies that the message types to be monitored include wait messages.

[0031] In one possible implementation, the compute node is a server in a cloud data center, or the compute node is a compute card included in the server, wherein the compute card is a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processor (NPU).

[0032] In one feasible approach, multiple computing nodes transmit data based on the Remote Direct Memory Access (RDMA) protocol, with the start message being a write-first message, the end message being a write-last message, the wait message being a write-only message, and the retransmission message being a NACK message.

[0033] Thirdly, a computing device cluster is provided, the computing device cluster including at least one computing device, each computing device including a processor and a memory, wherein the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform a network measurement method as described in the first aspect or any one of the first aspects above.

[0034] Fourthly, a computer program product containing instructions is provided, which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform a network measurement method as described in the first aspect or any one of the first aspects above.

[0035] Fifthly, a computer-readable storage medium is provided, characterized in that it includes computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform a network measurement method as described in the first aspect or any one of the first aspects above. Attached Figure Description

[0036] Figure 1 This is a system architecture diagram of a cloud data center provided in an embodiment of this application;

[0037] Figure 2 This is a flowchart of a network measurement method based on cloud computing technology provided in an embodiment of this application;

[0038] Figure 3 This is a schematic diagram of a network measurement method provided in an embodiment of this application;

[0039] Figure 4 This is a flowchart of a network measurement method provided in an embodiment of this application;

[0040] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0041] Figure 6 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0043] In recent years, with the rapid development of Large Language Model (LLM) technology, research and applications in the field of deep learning have seen unprecedented breakthroughs. The training process of LLM requires substantial computing resources, and to meet this demand, related infrastructure is constantly expanding, particularly with the widespread application of large-scale GPU (Graphics Processing Unit) and NPU (Neural Processing Unit) clusters. These high-performance computing hardware components, along with supporting network hardware such as high-speed switches, fiber optic connections, and optical modules, have significantly improved data transmission rates and computational efficiency, making distributed parallel computing possible. Through massively parallel computing, LLM can be trained efficiently with the support of massive amounts of data and complex models, further driving the development of artificial intelligence technology.

[0044] However, with the rapid expansion of cluster size and the continuous increase in computing resources, the complexity of the system also increases. This not only increases the difficulty of hardware resource scheduling and data flow, but also increases the frequency of failures, especially when multiple GPU / NPU nodes work together. Failures can cause performance degradation, training interruptions, and even loss of model accuracy. Therefore, how to ensure the stability and efficiency of the system in such a complex training environment has become an urgent problem to be solved in technical operations and maintenance. Fault diagnosis and troubleshooting require quickly and accurately locating potential problems in the system, but this process is often limited by hardware scale and training time, placing enormous pressure on operations and maintenance personnel.

[0045] In this context, in-depth, real-time, and fine-grained monitoring of the LLM training process has become crucial for improving training quality and ensuring system stability. Achieving visibility into the training process not only helps in the timely detection and resolution of faults but also optimizes the training process and improves resource utilization efficiency. Specifically, visibility into the training process refers to acquiring network metrics (such as packet loss rate and bandwidth) of each computing node in the cloud platform during AI model training, thereby reflecting the training status of the computing nodes on the AI ​​model.

[0046] Furthermore, tenants have diverse measurement needs, including various monitoring requirements such as bandwidth, packet loss rate, and waiting order among multiple SIM cards. Therefore, the key to measuring efficiency and cost lies in flexibly and selectively measuring the appropriate types of packets based on specific needs, rather than monitoring all metrics comprehensively. By accurately selecting measurement targets, resource consumption can be significantly reduced while ensuring performance and stability, thereby balancing system performance and operational costs and ensuring optimal resource utilization.

[0047] Currently, to achieve visibility into the training process, existing visibility acquisition strategies typically rely on tracking computing nodes during training to comprehensively record key events. By capturing and recording these events, the system can monitor each stage of the training process, analyze the state of each computing node, and thus achieve transparency and optimization of the training process. Specifically, the monitoring program for tracking corresponding events is embedded in the computing nodes, which then run the monitoring program to obtain the relevant event information.

[0048] While existing point-based methods offer some visibility, they also face limitations, particularly during large-scale parallel computing and long-term training. These methods often fail to meet the demands for real-time performance and fine-grained analysis. To further improve the visibility of the training process and address the bottlenecks in current technologies, it is crucial to explore new methods and strategies, especially those that can more comprehensively and in real-time analyze and monitor the training process while maintaining training efficiency.

[0049] This application provides a network measurement method based on cloud computing technology. By increasing the types of measurement messages and supporting tenants to select network measurement indicators on demand, it improves the flexibility and accuracy of measurement while effectively reducing costs. The training process for the AI ​​model described above can be executed on computing nodes within a cloud data center. Computing nodes can be servers or computing cards included in servers, such as CPUs, GPUs, or NPUs. Figure 1 This is a system architecture diagram of a cloud data center provided in an embodiment of this application. For example... Figure 1 As shown, a cloud data center includes a cloud management platform and multiple servers. The cloud management platform and the servers are connected via the data center's internal network, and the servers are distributed across different racks, rack rows, and server rooms.

[0050] The cloud management platform's functions include: providing access interfaces (such as user interfaces or APIs), allowing tenants to remotely access the cloud management platform via clients, register a cloud account and password, and log in. After successful authentication of the cloud account and password, tenants can further select and purchase virtual machines of specific specifications (processor, memory, disk) on the cloud management platform. After successful purchase, the cloud management platform provides the remote login account and password for the purchased virtual machine, allowing clients to remotely log in to the virtual machine, install and run the tenant's applications within it. The virtual machines purchased by the tenant can be provided by servers in the cloud data center.

[0051] The cloud management platform is logically divided into the following functions: Tenant Console, Compute Management Service, Network Management Service, Storage Management Service, Authentication Service, and Image Management Service. The Tenant Console provides an interface or API for interaction with tenants. The Compute Management Service manages servers running virtual machines and containers, as well as bare metal servers. The Network Management Service manages network services (such as gateways and firewalls). The Storage Management Service manages storage services (such as data bucket services). The Authentication Service manages tenant account passwords. The Image Management Service manages virtual machine images.

[0052] The network measurement method based on cloud computing technology provided in this application can be applied to the aforementioned cloud data centers. Figure 2This is a flowchart of a network measurement method based on cloud computing technology provided in an embodiment of this application, such as... Figure 2 As shown, the method includes:

[0053] Step 201: The cloud management platform receives a job execution request from the tenant. The job execution request is used to instruct multiple target computing nodes in the cloud data center to execute the job.

[0054] In this context, a job typically refers to a complete task unit submitted by a tenant, including the program, dataset, and operating instructions. For example, a job could be an AI training task, an AI inference task, or a High Performance Computing (HPC) task. The compute node can be a server or compute card (such as a CPU, GPU, or NPU) in a cloud data center. This methodology uses a compute card as an example to illustrate the subsequent solutions.

[0055] In implementation, when a tenant needs compute nodes to execute a job, they can send a job execution request to the cloud management platform. This request can include the data required to execute the job, job information, etc., and the job information can be used to instruct multiple target compute nodes in the cloud data center to execute the job.

[0056] In one example, a job may consist of multiple computational tasks, and the execution of these tasks may have data dependencies. The tenant or cloud management platform can allocate corresponding target computing nodes to each computational task. For example, the job may be a training task for an AI model, which may include convolutional layers, pooling layers, softmax layers, etc. The training task for the AI ​​model may include the computations corresponding to the convolutional layers, pooling layers, and softmax layers, respectively. The computation of each layer can be allocated to one or more computing nodes for execution.

[0057] Step 202: The cloud management platform controls multiple target computing nodes in the cloud data center to execute jobs according to the job execution request.

[0058] After receiving a job execution request, the cloud management platform can control multiple target computing nodes to execute the job, guided by the job information.

[0059] Step 203: The cloud management platform obtains the message type to be monitored specified by the tenant. The message type includes at least one of the following: start message, end message, waiting message, and retransmission message.

[0060] In one example, the job execution request sent by the tenant to the cloud management platform in step 201 can also be used to instruct the cloud management platform to monitor network data during the execution of jobs on multiple target compute nodes, and obtain the corresponding network measurement metrics. In this case, the job execution request may carry the message type that the tenant needs to monitor during the execution of the job.

[0061] In another example, after a tenant sends a job execution request on the cloud management platform, if it needs to monitor the network data during the execution of jobs on multiple target compute nodes, it can send a message monitoring request to the cloud management platform again. The message monitoring request can carry the message type that needs to be monitored, and further, the message monitoring request can also carry job information to indicate the multiple target compute nodes that need to be monitored.

[0062] The tenant can select the message types to be monitored via the client interface. In this embodiment, the types selectable by the tenant include, but are not limited to, start messages, end messages, wait messages, and retransmission messages. The tenant can select a subset of these four types as the message types to be monitored based on the actual needs of the compute node's execution tasks. In cloud data centers, multiple compute nodes transmit data based on the Remote Direct Memory Access (RDMA) protocol. In the RDMA protocol, the start message is a write-first message, the end message is a write-last message, the wait message is a write-only message, and the retransmission message is a NACK message.

[0063] The write-first message is used to mark the start of data packet transmission between computing nodes, that is, the first message.

[0064] The write-last message is used to mark the end of data packet transmission between compute nodes, that is, the last message.

[0065] Write-only messages are used to identify that a compute node (such as a GPU / NPU card) is ready to transmit data. For example, when compute node A sends a write-only message to compute node B, it indicates that compute node A is waiting for compute node B to send the data needed to perform the computation task. Capturing this message helps identify the start time of data transmission.

[0066] A NACK message indicates that the receiver failed to receive the previous data packet, or that data transmission failed due to some error. Capturing this message helps identify problems that occurred during transmission (such as packet loss).

[0067] In one example, when the job is data-intensive, the tenant specifies retransmission messages as one of the message types to be monitored. The execution efficiency of data-intensive jobs is highly correlated with data transmission efficiency. Therefore, by monitoring the number of retransmission messages, the packet loss rate between target computing nodes can be determined, thus reflecting the data transmission efficiency between them. Therefore, monitoring retransmission messages allows for the monitoring of the execution efficiency of data-intensive jobs. These data-intensive jobs can be training or inference tasks for Mixture of Experts (MoE) models.

[0068] When the job is compute-intensive, the tenant specifies that the message types to be monitored include waiting messages. The execution efficiency of compute-intensive jobs is highly correlated with the efficiency of the target compute node in executing the job. Therefore, by monitoring the number of waiting messages, the duration for which the target compute node waits to execute the task can be determined, thus reflecting the transmission of the task by the target compute node. Therefore, by monitoring waiting messages, the execution efficiency of compute-intensive jobs can be monitored. Compute-intensive jobs can be training or inference tasks for dense models.

[0069] In one feasible approach, tenants can select the message types to be monitored by choosing a service level. This service level can be included in the message monitoring request. The cloud management platform determines the message types to be monitored by the tenant based on the service level selected and the corresponding message type mapping. In this mapping, the number of service levels and the number of corresponding message types are positively correlated. Table 1 shows an example of a service level-message type mapping provided in this application embodiment.

[0070] Table 1

[0071]

[0072]

[0073] Table 1, in addition to showing the correspondence between service levels and message types, also illustrates the network measurement metrics obtainable for each measured message type and their corresponding costs. It is evident that, in this embodiment, on the one hand, the cloud management platform can provide tenants with various message types for monitoring, enriching the monitored network measurement metrics and enabling fine-grained monitoring of the tasks performed by computing nodes (in some embodiments, monitoring is only performed on the number of messages, without distinguishing message types, thus only achieving coarse-grained bandwidth monitoring). On the other hand, tenants can select the message types to be monitored according to their needs, which improves the flexibility of obtaining network measurement metrics and avoids wasting monitoring performance. Furthermore, tenants can specify the network measurement metrics to be measured, further improving the flexibility of obtaining network measurement metrics and avoiding wasted monitoring performance.

[0074] Step 204: The cloud management platform determines at least one switch used for data transmission between multiple target computing nodes.

[0075] In implementation, the cloud management platform can obtain the topology of multiple target computing nodes based on job information. This topology includes the connections between the target computing nodes via switches. Thus, it obtains at least one switch used by the multiple target computing nodes for data transmission.

[0076] Step 205: The cloud management platform sends a packet monitoring notification to at least one switch. The packet monitoring notification includes the network addresses of multiple target computing nodes and the packet types to be monitored. The packet monitoring notification is used to instruct at least one switch, after receiving the packet monitoring notification, to forward the packets of the packet types to be monitored sent by the multiple target computing nodes to the measurement node according to the network addresses of the multiple target computing nodes, so that the measurement node can analyze and process the received packets to obtain at least one network measurement indicator among the multiple target computing nodes, and send at least one network measurement indicator to the tenant.

[0077] After identifying at least one switch used by multiple target computing nodes for data transmission, the cloud management platform can send a packet monitoring notification to each switch. The network address of the target computing node carried in the packet monitoring notification can be the IP address of each computing card. The switch can load these computing card IP addresses into its flow table and configure itself to mirror and forward packets with these computing card IP addresses as source IP addresses and belonging to the monitored packet type to the measurement node.

[0078] In implementation, measurement nodes can be servers located in cloud data centers or virtual machines running on servers. After receiving a message, the measurement node can perform statistical analysis on the message according to a set period to obtain at least one network measurement indicator, and then send the obtained network measurement indicator to the tenant.

[0079] In one example, the network measurement metrics obtained by analyzing and processing the start and end messages include at least one of the following metrics:

[0080] 1. Number of Messages (data blocks)

[0081] The number of messages is obtained. The start or end message can indicate how many data blocks the computing card sent or received.

[0082] 2. Message size

[0083] The sequence number (PSN) difference between the closest messages is used to determine the message size. Closest messages are defined as closely spaced write-first and write-last messages, with the write-first message received before the write-last message. By calculating the difference between the sequence numbers of these close write-first and write-last messages, the number of messages included in a message transmitted between computing cards can be determined. Since the data size of each message is fixed, the message size can also be determined.

[0084] 3. Message start / end time

[0085] The arrival times of the statistical messages are obtained, that is, the receiving times of the write first message and the write last message.

[0086] 4. Message path

[0087] The hash is calculated based on the message's 5-tuple. The message path refers to the transmission path of the message from the computing card that sends the message to the computing card that receives the message.

[0088] 5. GPU / NPU communication pair

[0089] The GPU / NPU communication pair is obtained from the source and destination IP addresses in the message. The GPU / NPU communication pair refers to the communication pair between the GPU card and the NPU card in the computing card. The GPU / NPU communication pair can be obtained by statistically analyzing the source and destination IP addresses in the message and the pre-recorded correspondence between IP addresses and card types.

[0090] 6. Tor internal / external flow distribution

[0091] The data is derived from a combination of message path and message size. Tor internal traffic refers to the traffic within the Tor switch, i.e., the traffic sent by the Tor switch to connected compute cards. Tor external traffic refers to the traffic sent by the Tor switch to other connected switches. In one example, by obtaining the Tor internal / external traffic distribution, the data exchange situation of multiple compute cards executing a job can be determined. A high level of Tor external traffic indicates that data between compute cards often needs to pass through more Tor switches, potentially reducing data transmission efficiency. In this case, the order and position of the compute cards executing the job can be adjusted to improve their efficiency.

[0092] 7. Traffic distribution along each path

[0093] This is determined by combining the message path and message size. By obtaining the traffic distribution of each path, it can be determined whether the path settings are reasonable. If the traffic of a certain path is too high or too low, it indicates that there is a problem with the path settings between multiple computing cards, which can be further reduced and optimized.

[0094] 8. GPU / NPU edge-side transmission volume

[0095] It is obtained based on the message size and the source IP address of the message.

[0096] 9. Sending volume of different QPs (Queue pairs)

[0097] It is determined based on the Message size. Here, QP refers to the message sending queue in the transmission pair between multiple computing cards.

[0098] Network measurement metrics obtained from analyzing and processing retransmitted messages include:

[0099] 10. Number of packet losses

[0100] The number of NACK messages is used to determine the packet loss count between computing cards. During data transmission, if the receiving computing card fails to receive a packet, it can send a NACK message to the sending computing card. By counting the number of NACK messages, the number of packet losses during data transmission between computing cards can be determined.

[0101] In addition to the 10 types mentioned above, network measurement metrics obtained by analyzing and processing retransmitted messages, start messages, and end messages also include:

[0102] 11. Message completion time

[0103] The time difference is calculated based on the closest write first and write last messages.

[0104] 12. Message bandwidth

[0105] The result is obtained by dividing the message communication volume and the message completion time.

[0106] In addition to the 12 types mentioned above, network measurement metrics obtained by analyzing and processing retransmitted messages, start messages, end messages, and waiting messages also include:

[0107] 13. Waiting time before stream transmission

[0108] The difference in arrival time is calculated based on the closest write-only and write-first messages. The pre-send waiting time, i.e., the waiting time before the target computing card sends a data block, refers to the time it takes for the target computing card to start sending a write-first message to the next computing card after receiving a write-only message from the next computing card; in other words, the time the next computing card waits to receive data. The next computing card refers to the computing card whose data depends on the output data of the target computing card for the task being executed.

[0109] 14. Waiting time after stream transmission

[0110] The difference in arrival times is calculated based on the closest write-only and write-last messages. The waiting time after the stream is sent, i.e., the waiting time after the target computing card sends a data block, refers to the time it takes for the target computing card to receive a write-only message from the next computing card after sending a write-last message to the next computing card; in other words, the time it takes for the next computing card to execute its task.

[0111] By obtaining the waiting time before or after stream transmission, the waiting time for the computing card to execute tasks can be determined. If the waiting time of the computing card is long, the order of jobs executed by the computing card can be adjusted, thereby improving the execution efficiency of the jobs.

[0112] In summary, when measuring all four types of RDMA characteristic packets, the embodiments of this application can support at least 14 measurement metrics. It is evident that in this embodiment, by using write-only packets, write-first packets, NACK packets, and write-only packets as RDMA characteristic packets, the acquired network measurement metrics can be enriched.

[0113] In this embodiment, the switch between computing nodes can be a top-of-rack (Tor) switch, and the measurement nodes can include a first measurement node and a second measurement node. The first measurement node can be a Network Processor (NP) switch, and the second measurement node can measure servers, which can be servers in a cloud data center or virtual machines running on servers.

[0114] In step 205 above, the cloud management platform can simultaneously send packet monitoring notifications to the Tor switch and the NP switch. This enables the Tor switch to mirror and forward packets whose source IP address is the IP address of the target compute node and belong to the monitored packet type to the NP switch. After receiving the packets forwarded by the Tor switch, the NP switch periodically counts the packet information corresponding to the received packets and sends the packet information to the measurement server. At least one of the following: the number of received packets, sequence number, reception time, and 5-tuple, can be used to calculate at least the aforementioned 14 network measurement metrics. Thus, the measurement server analyzes and processes the received packet information to obtain at least one network measurement metric between multiple target compute nodes.

[0115] Figure 3 This is a schematic diagram of a network measurement method provided in an embodiment of this application. Figure 3 As shown, multiple target compute nodes executing a job can exchange data through multiple Tor switches. During packet forwarding for compute nodes, if a Tor switch detects a packet sent by a target compute node and matching the monitored packet type, it can mirror and forward the corresponding packet to an NP switch. Upon receiving the packet, the NP switch can statistically analyze the packet information according to a set period and send the information to the measurement server. The measurement server can then analyze and process the packet information to obtain multiple network test metrics. Based on the mapping between the target compute node's IP address and the tenant's IP address, it can forward these network test metrics to the tenant.

[0116] This method of processing and forwarding packets via NP switches simplifies the connection between Tor switches and measurement servers, facilitating subsequent expansion of compute nodes or Tor switches. Furthermore, the NP server's pre-acquisition of packet information reduces the computational complexity for the measurement server in obtaining network test metrics.

[0117] In one feasible approach, a tenant can set alarm thresholds for each network measurement metric and send these thresholds to a cloud management platform. Upon receiving the alarm thresholds set by the tenant for the network measurement metric, the cloud management platform can send the corresponding alarm thresholds to the measurement node, such as a measurement server. The alarm thresholds can be set by the tenant based on the actual situation of the compute node's job execution; specific values ​​are not listed in this embodiment. Thus, when the measurement server determines that a measured network measurement metric exceeds the corresponding alarm threshold, it can send a corresponding alarm notification to the tenant, for example, via SMS or email, enabling the tenant to promptly obtain information about abnormalities in the compute node's job execution.

[0118] Figure 4 This is a flowchart illustrating a network measurement method provided in an embodiment of this application. It employs a collaborative approach between a ToR switch and a programmable NP switch to achieve efficient network measurement. The specific operation steps are as follows:

[0119] 1. Tenant Input and Service Configuration

[0120] After the tenant inputs the job information, the system obtains the corresponding topology information based on the job characteristics.

[0121] Tenants can select a service level and specify the metrics that need to be measured (such as the monitoring scope and analysis depth of different characteristic messages).

[0122] Subsequently, the system issues corresponding measurement policies to the switches and measurement devices in the cloud network according to the configuration.

[0123] Specifically, the system loads the card IPs used by the tenants into the flow table of the cloud network ToR switch, configures the ToR switch to mirror, and forwards RDMA characteristic packets (i.e., write first / last / only and tenant-specified subsets in NACK packets) originating from these IPs to the measurement device (e.g., a programmable NP switch).

[0124] 2. Message processing in cloud networks

[0125] In the cloud network, the ToR switch mirrors and forwards RDMA feature packets that match the flow table to the edge measurement device (NP switch).

[0126] After transmission through the cloud network, all RDMA characteristic packets arrive at the NP switch. The NP switch processes these packets in real time on the data plane without relying on backend database analysis, thereby reducing latency and improving processing efficiency.

[0127] 3. Measurement result output and alarms

[0128] The NP switch uploads the processed measurement results to the front-end system for data display and real-time alarms.

[0129] The front-end system matches the measurement results with each tenant based on the source IP.

[0130] The measurement results are used to generate statistical information in cycles (such as minutes or hours) for tenants to view on the front-end interface.

[0131] If an anomaly is detected (such as excessive packet loss), the system will trigger a real-time alarm mechanism to notify the tenant via SMS, email, or other means so that the problem can be located in a timely manner.

[0132] This application primarily addresses the needs of cloud service providers, aiming to offer tenants real-time monitoring services based on AI networks. It allows tenants to select different service levels based on their requirements, with each level corresponding to different network packet measurement types, thereby meeting diverse performance monitoring needs. The main devices involved in the workflow of this application's embodiments are switches in the cloud network and measurement devices (e.g., programmable NP switches) connected to the cloud network switches.

[0133] Tenants can set service levels based on job information and actual needs. Each service level corresponds to different measurement accuracy and overhead (i.e., the set of RDMA characteristic packets for measurement). The cloud service provider then distributes relevant switch configurations to the switches in the cloud network, forwarding the RDMA characteristic packets to the measurement equipment for measurement, and displaying the measurement results on the front end with real-time alarms. Through this process, tenants can obtain detailed measurement metrics, such as traffic volume, bandwidth, path information, and anomalies.

[0134] In one exemplary embodiment: In this exemplary embodiment, a series of metrics in the RDMA network are measured using the number of write-first messages, as shown in Table 2. For a certain tenant, the system received a total of 12 write-first messages within a certain period of time.

[0135] Table 2

[0136]

[0137]

[0138] Based on the above message information, the analysis system can further extract the following data:

[0139] Message count: The number of messages should be the same as the number of write-first messages. Therefore, within a certain time period, the compute node with source IP 1.1.1.1 sent 12 messages.

[0140] Message size: The size of a message is calculated based on the difference in PSN between adjacent write-first messages in the same stream. For a compute node with a source IP of 1.1.1.1, the size of each message is 150 - 100 = 50 packets; for a compute node with a source IP of 3.3.3.3, the size of each message is 200 - 100 = 100 packets.

[0141] Message start time: The start time of a message is obtained from the timestamp of the write-first message. For compute nodes with source IP 1.1.1.1, the message start times are 3000us, 3010us, 3020us, and 3032us; for compute nodes with source IP 3.3.3.3, the message start times are 1000us, 1010us, 1020us, and 1032us.

[0142] Message Path: Assume all messages pass through a five-hop switch, and the switch selects the routing path using the MurmurHash (non-cryptographic hash function) algorithm, with port numbers ranging from 0 to 15. Assume the switch selects MurmurHash seeds 0, 1, 2, 3, and 4 sequentially for routing. For a compute node with a source IP of 1.1.1.1, the calculated message passes through ports 9, 0, 10, 1, and 7 sequentially; for a compute node with a source IP of 3.3.3.3, the message passes through ports 10, 11, 5, 0, and 0 sequentially.

[0143] In another exemplary embodiment:

[0144] In this exemplary embodiment, to measure the number of packet losses in the RDMA network using the number of NACK messages, as shown in Table 3, for a certain tenant, the system received a total of 8 NACK messages within a certain period of time.

[0145] Table 3

[0146]

[0147] Based on the above message information, the analysis system can further extract the following data (assuming the NACK opcode is 0x1f):

[0148] Packet loss count: A total of 8 NACK packets were collected, indicating 8 instances of packet loss. For the compute node with source IP 1.1.1.1, four packet losses occurred, corresponding to packets with PSN numbers 100, 150, 200, and 250, at times of 3000us, 3010us, 3020us, and 3032us respectively. For the compute node with source IP 3.3.3.3, four packet losses occurred, corresponding to packets with PSN numbers 100, 200, 300, and 400, at times of 1000us, 1010us, 1020us, and 1030us respectively.

[0149] The technical problems addressed by the embodiments of this application include improving the flexibility and accuracy of measurements while effectively reducing costs by increasing the types of measurement messages and supporting tenants to select measurement metrics on demand. Specifically:

[0150] 1) Increased Measurement Message Types: By introducing more message types, this application embodiment enriches the measurable metrics, thereby providing tenants with more comprehensive training process information. These newly added message types can capture more dimensions of performance data, such as network latency, bandwidth utilization, packet loss rate, etc., thereby supporting in-depth analysis of complex training processes.

[0151] 2) On-demand selection of measurement metrics: This application embodiment allows tenants to flexibly select measurement metrics according to actual needs. By selecting appropriate message types for monitoring, unnecessary data collection can be effectively reduced, resource utilization optimized, system overhead reduced, and measurement efficiency improved. Through this function, tenants can precisely control the monitoring content according to different training tasks and scenario requirements, avoiding the collection of redundant information, thereby saving computing and storage costs.

[0152] This application proposes a network measurement method for achieving efficient and fine-grained network measurement in AI networks, thereby providing deeper training visibility. During large-scale LLM training, traditional per-packet granularity measurement methods often face significant computational and storage pressure due to the massive amount of data. Therefore, this application does not pursue visibility for every single data packet, but instead focuses on RDMA characteristic messages, performing performance monitoring at the RDMA message granularity. This method not only provides sufficient visibility but also executes efficiently, avoiding excessive system overhead.

[0153] This application also provides a cloud management platform, which can be referred to the above-described cloud management method. Figure 1 This is used to manage infrastructure, which includes a cloud data center. The cloud data center includes multiple compute nodes and measurement nodes, among which:

[0154] The cloud management platform is used to receive job execution requests from tenants, which instruct multiple target compute nodes in the cloud data center to execute jobs. Based on the job execution requests, it controls the execution of jobs by the multiple target compute nodes in the cloud data center. It obtains the message types to be monitored specified by the tenant, including at least one of start messages, end messages, wait messages, and retransmission messages. It identifies at least one switch used for data transmission between the multiple target compute nodes. It sends a message monitoring notification to the at least one switch, which includes the network addresses of the multiple target compute nodes and the message types to be monitored. The message monitoring notification instructs the at least one switch, upon receiving the notification, to forward messages of the message types to be monitored sent by the multiple target compute nodes to the measurement node based on the network addresses of the multiple target compute nodes.

[0155] The measurement node is used to analyze and process the received messages to obtain at least one network measurement indicator among multiple target computing nodes, and to send at least one network measurement indicator to the tenant.

[0156] In one feasible approach, the cloud management platform receives message monitoring requests from tenants, the requests carrying the service level selected by the tenant. Based on the tenant's selected service level, and within a mapping between service levels and message types, the platform determines the message types to be monitored specified by the tenant, where the number of service levels and corresponding message types is positively correlated.

[0157] In one feasible implementation, the switch is a top-of-rack switch, and the measurement nodes include a network processor switch and a measurement server. The network processor switch receives packets forwarded by the top-of-rack switch and sends the corresponding packet information to the measurement server. The packet information includes at least one of the following: the number of packets received, sequence number, reception time, and a 5-tuple, for each packet type. The measurement server analyzes and processes the received packet information to obtain at least one network measurement metric among the multiple target computing nodes.

[0158] In one feasible approach, network measurement metrics obtained by analyzing and processing start and end messages include at least one of the following metrics: the number, size, duration, and transmission path of data blocks sent by each target computing node executing the job; data transmission pairs between multiple target computing nodes executing the job, and the data transmission volume of each message transmission queue in each data transmission pair; the proportion of internal and external traffic for each switch among multiple target computing nodes, where internal traffic refers to the amount of data sent by the switch to connected computing nodes, and external traffic refers to the amount of data sent by the switch to other switches; and the traffic proportion of each data transmission path among multiple target computing nodes.

[0159] In one feasible approach, network metrics obtained by analyzing and processing wait messages and start messages include: the waiting time before the target computing node sends a data block. Network metrics obtained by analyzing and processing wait messages and end messages include: the waiting time after the target computing node sends a data block.

[0160] In one feasible approach, network metrics obtained by analyzing and processing retransmitted messages include the number of packet losses.

[0161] In one possible implementation, the cloud management platform is also used to receive alarm thresholds set by tenants for at least one network measurement metric, which indicates an anomaly in data transmission between multiple target compute nodes. The alarm thresholds corresponding to at least one network measurement metric are sent to the measurement nodes. When it is determined that a network measurement metric exceeds its corresponding alarm threshold, an alarm notification is sent to the tenant.

[0162] In one feasible approach, when the job is data-intensive, the tenant specifies that the message types to be monitored include retransmission messages. When the job is compute-intensive, the tenant specifies that the message types to be monitored include wait messages.

[0163] In one possible implementation, the compute node is a server in a cloud data center, or the compute node is a compute card included in the server, wherein the compute card is a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processor (NPU).

[0164] In one feasible approach, multiple computing nodes transmit data based on the Remote Direct Memory Access (RDMA) protocol, with the start message being a write-first message, the end message being a write-last message, the wait message being a write-only message, and the retransmission message being a NACK message.

[0165] In the cloud management platform shown in this application, tenants can customize the message types that computing nodes in the cloud data center need to monitor during job execution. The switch then sends the messages corresponding to the monitored message types to the test nodes, which perform statistical analysis of network measurement metrics. Therefore, the network measurement method provided in this application does not require the processing resources of computing nodes, thus improving the efficiency of job execution. The cloud management platform and the network measurement method based on cloud computing technology provided in this application belong to the same inventive concept. Specific implementation methods can be found in the description of the network measurement method based on cloud computing technology in the above embodiments, and will not be repeated here.

[0166] This application also provides a computing device 500. For example... Figure 5 As shown, the computing device 500 includes a bus 502, a processor 504, a memory 506, and a communication interface 508. The processor 504, the memory 506, and the communication interface 508 communicate with each other via the bus 502. The computing device 500 can be a tenant terminal included in the above embodiments, or it can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 500.

[0167] Bus 502 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus 502 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 502 may include a path for transmitting information between various components of the computing device 500 (e.g., memory 506, processor 504, communication interface 508).

[0168] Processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0169] Memory 506 may include volatile memory, such as random access memory (RAM). Memory 506 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0170] The memory 506 stores executable program code, which the processor 504 executes to implement the network measurement methods. In other words, the memory 506 stores code for executing network measurement methods.

[0171] The communication interface 508 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 500 and other devices or communication networks.

[0172] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0173] like Figure 6 As shown, the computing device cluster includes at least one computing device 500, which will not be described again in this embodiment. The memory 506 of one or more computing devices 500 in the computing device cluster may store the same instructions for executing network measurement methods.

[0174] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing network measurement methods. In other words, a combination of one or more computing devices 500 can jointly execute instructions for executing network measurement methods.

[0175] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a network measurement method.

[0176] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform a network measurement method.

[0177] In this application, the term "at least one" means one or more, and the term "multiple" means two or more.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network measurement method based on cloud computing technology, characterized in that, The method is applied to a cloud management platform, which manages infrastructure, including a cloud data center. The cloud data center includes multiple computing nodes and measurement nodes. The method includes: The cloud management platform receives job execution requests sent by tenants, and the job execution requests are used to instruct multiple target computing nodes in the cloud data center to execute jobs. The cloud management platform controls multiple target computing nodes in the cloud data center to execute jobs according to the job execution request; The cloud management platform obtains the message type to be monitored specified by the tenant, and the message type includes at least one of start message, end message, wait message and retransmission message; The cloud management platform determines at least one switch used for data transmission between the multiple target computing nodes; The cloud management platform sends a packet monitoring notification to the at least one switch. The packet monitoring notification includes the network addresses of the plurality of target computing nodes and the packet type to be monitored. The packet monitoring notification is used to instruct the at least one switch, upon receiving the packet monitoring notification, to forward packets belonging to the packet type to be monitored sent by the plurality of target computing nodes to the measurement node according to the network addresses of the plurality of target computing nodes. This allows the measurement node to analyze and process the received packets to obtain at least one network measurement indicator among the plurality of target computing nodes, and then send the at least one network measurement indicator to the tenant.

2. The method according to claim 1, characterized in that, The cloud management platform obtains the message types to be monitored specified by the tenant, including: The cloud management platform receives a message monitoring request sent by the tenant, and the message monitoring request carries the service level selected by the tenant; The cloud management platform determines the message types to be monitored by the tenant based on the service level selected by the tenant and the correspondence between service level and message type. In the correspondence, the number of service levels and corresponding message types is positively correlated.

3. The method according to claim 1 or 2, characterized in that, The switch is a rack-top switch, and the measurement node includes a network processor switch and a measurement server; The measurement node analyzes and processes the received messages to obtain at least one network measurement metric among the plurality of target computing nodes, including: The network processor switch receives packets forwarded by the top-of-rack switch and sends the packet information corresponding to the received packets to the measurement server. The packet information includes at least one of the following: the number of packets received, the sequence number, the receiving time, and the five-tuple, for each type of packet. The measurement server analyzes and processes the received message information to obtain at least one network measurement indicator among the multiple target computing nodes.

4. The method according to any one of claims 1 to 3, characterized in that, The network measurement metrics obtained by analyzing and processing the start and end messages include at least one of the following metrics: The number, size, transmission duration, and transmission path of data blocks sent by each target computing node executing the job; Data transmission pairs between multiple target computing nodes executing the job, and the data transmission volume of each message transmission queue in each data transmission pair; The ratio of internal traffic to external traffic for each switch among the multiple target computing nodes, wherein internal traffic refers to the amount of data sent by the switch to the connected computing nodes, and external traffic refers to the amount of data sent by the switch to other switches; The percentage of traffic on each data transmission path between the multiple target computing nodes.

5. The method according to any one of claims 1 to 4, characterized in that, The network measurement metrics obtained by analyzing and processing the waiting message and the starting message include: the waiting time before the target computing node sends a data block; The network measurement metrics obtained by analyzing and processing the waiting messages and the termination messages include: the waiting time after the target computing node sends a data block.

6. The method according to any one of claims 1 to 5, characterized in that, The network measurement metrics obtained by analyzing and processing the retransmitted messages include the number of packet losses.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The cloud management platform receives an alarm threshold set by the tenant for the at least one network measurement indicator, the alarm threshold being used to indicate that data transmission between the plurality of target computing nodes is abnormal; The cloud management platform sends the alarm threshold corresponding to the at least one network measurement indicator to the measurement node; When the measurement node determines that the obtained network measurement metric exceeds the corresponding alarm threshold, it sends an alarm notification to the tenant.

8. The method according to any one of claims 1 to 7, characterized in that, When the job is a data-intensive job, the retransmission message is included in the message type to be monitored specified by the tenant; When the job is a compute-intensive job, the tenant specifies the message types to be monitored, including the waiting message.

9. The method according to any one of claims 1 to 8, characterized in that, The computing node is a server in a cloud data center, or the computing node is a computing card included in the server, wherein the computing card is a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processor (NPU).

10. The method according to any one of claims 1 to 9, characterized in that, The multiple computing nodes transmit data based on the Remote Direct Memory Access (RDMA) protocol. The start message is a write-first message, the end message is a write-last message, the wait message is a write-only message, and the retransmission message is a NACK message.

11. A cloud management platform, characterized in that, The cloud management platform is used to manage infrastructure, which includes a cloud data center. The cloud data center includes multiple computing nodes and measurement nodes, wherein: The cloud management platform is used for: Receive a job execution request sent by a tenant, the job execution request being used to instruct multiple target computing nodes in the cloud data center to execute a job; Based on the job execution request, control multiple target computing nodes in the cloud data center to execute the job; Obtain the message type to be monitored specified by the tenant, wherein the message type includes at least one of start message, end message, wait message and retransmission message; Determine at least one switch used for data transmission between the plurality of target computing nodes; Send a packet monitoring notification to the at least one switch. The packet monitoring notification includes the network addresses of the plurality of target computing nodes and the packet type to be monitored. The packet monitoring notification is used to instruct the at least one switch, after receiving the packet monitoring notification, to forward packets of the packet type to be monitored sent by the plurality of target computing nodes to the measurement node according to the network addresses of the plurality of target computing nodes. The measurement node is used for: The received messages are analyzed and processed to obtain at least one network measurement indicator among the multiple target computing nodes, and the at least one network measurement indicator is sent to the tenant.

12. The cloud management platform according to claim 11, characterized in that, The cloud management platform is used for: Receive a message monitoring request sent by the tenant, the message monitoring request carrying the service level selected by the tenant; Based on the service level selected by the tenant, the message type to be monitored specified by the tenant is determined in the correspondence between service level and message type, wherein the number of service levels and corresponding message types is positively correlated in the correspondence.

13. The cloud management platform according to claim 11 or 12, characterized in that, The switch is a rack-top switch, and the measurement node includes a network processor switch and a measurement server; The network processor switch is used to: receive packets forwarded by the rack top switch, and send the packet information corresponding to the received packets to the measurement server. The packet information includes at least one of the following: the number of packets received, the sequence number, the receiving time, and the five-tuple, for each type of packet. The measurement server is used to analyze and process the received message information to obtain at least one network measurement index among the multiple target computing nodes.

14. The cloud management platform according to any one of claims 11 to 13, characterized in that, The network measurement metrics obtained by analyzing and processing the start and end messages include at least one of the following metrics: The number, size, transmission duration, and transmission path of data blocks sent by each target computing node executing the job; Data transmission pairs between multiple target computing nodes executing the job, and the data transmission volume of each message transmission queue in each data transmission pair; The ratio of internal traffic to external traffic for each switch among the multiple target computing nodes, wherein internal traffic refers to the amount of data sent by the switch to the connected computing nodes, and external traffic refers to the amount of data sent by the switch to other switches; The percentage of traffic on each data transmission path between the multiple target computing nodes.

15. The cloud management platform according to any one of claims 11 to 14, characterized in that, The network measurement metrics obtained by analyzing and processing the waiting message and the starting message include: the waiting time before the target computing node sends a data block; The network measurement metrics obtained by analyzing and processing the waiting messages and the termination messages include: the waiting time after the target computing node sends a data block.

16. The cloud management platform according to any one of claims 11 to 15, characterized in that, The network measurement metrics obtained by analyzing and processing the retransmitted messages include the number of packet losses.

17. The cloud management platform according to any one of claims 11 to 16, characterized in that, The cloud management platform is also used for: Receive an alarm threshold set by the tenant for the at least one network measurement metric, the alarm threshold being used to indicate an anomaly in data transmission between the plurality of target computing nodes; Send the alarm threshold corresponding to the at least one network measurement indicator to the measurement node; When it is determined that the obtained network measurement metric exceeds the corresponding alarm threshold, an alarm notification is sent to the tenant.

18. The cloud management platform according to any one of claims 11 to 17, characterized in that, When the job is a data-intensive job, the retransmission message is included in the message type to be monitored specified by the tenant; When the job is a compute-intensive job, the tenant specifies the message types to be monitored, including the waiting message.

19. The cloud management platform according to any one of claims 11 to 18, characterized in that, The computing node is a server in a cloud data center, or the computing node is a computing card included in the server, wherein the computing card is a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processor (NPU).

20. The cloud management platform according to any one of claims 11 to 19, characterized in that, The multiple computing nodes transmit data based on the Remote Direct Memory Access (RDMA) protocol. The start message is a write-first message, the end message is a write-last message, the wait message is a write-only message, and the retransmission message is a NACK message.

21. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 10.

22. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 10.

Images