Latency measurement method and apparatus, and device and computer-readable storage medium

Abstract

The present application belongs to the technical field of communications. Disclosed are a latency measurement method and apparatus, and a device and a computer-readable storage medium. The method comprises: a first device sending a first message to a second device, wherein the transmission path of the first message is a first path; and the first device acquiring a first latency of the first path, which first latency is determined on the basis of a first timestamp and a second timestamp, wherein the first timestamp is a timestamp of when the first message is sent which is recorded by a network interface card on the first device, and the second timestamp is a timestamp of when the first message is received which is recorded by a network interface card on the second device. In the present application, the first latency, which is determined by means of hardware timestamps recorded by network interface cards, serves as a network latency on a first path between a first device and a second device, and thus a long-tail latency caused by an on-device processing latency of the first device and the second device can be avoided, and the binding of a path and a latency can be realized.

Description

Delay measurement method, device, equipment and computer-readable storage medium

[0001] This application claims priority to Chinese patent application No. 202311172026.X filed on September 12, 2023, entitled “Time Delay Measurement Method, Device, Equipment and Computer-Readable Storage Medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of communication technology, and in particular to a method, apparatus, device, and computer-readable storage medium for measuring time delay. Background Art

[0003] In a communication system, when two devices exchange messages, they are forwarded by multiple network devices between them. This results in a delay between the time the message is sent and the time it is received. Therefore, it is necessary to measure the delay between the two devices to accurately estimate the time it takes for a message to travel from one device to the other.

[0004] Summary of the Invention

[0005] This application provides a latency measurement method, apparatus, device, and computer-readable storage medium to accurately measure network latency on a path. The technical solution is as follows:

[0006] In a first aspect, a delay measurement method is provided, the method including: a first device sends a first message to a second device, where the transmission path of the first message is a first path; the first device obtains a first delay of the first path determined based on a first timestamp and a second timestamp, where the first timestamp is a timestamp recorded by a network card on the first device when sending the first message, and the second timestamp is a timestamp recorded by the network card on the second device when receiving the first message.

[0007] The timestamp recorded by the network card in this application is a hardware timestamp. The first delay determined based on the timestamp is the network delay on the first path between the first device and the second device. It can avoid the long-tail delay caused by the delay in processing messages between the first device and the second device, and can realize the binding of delay and path.

[0008] In one possible implementation, before the first device sends the first message to the second device, it also includes: the first device sends a path detection message to the second device, the path detection message includes first indication information, and the first indication information indicates that a detection result message is returned based on the path detection message; the first device receives at least one detection result message returned based on the path detection message, and the detection result message includes information about the physical port and virtual port of the network device that sends the detection result message; the first device determines at least one path between the first device and the second device based on the information about the physical port and virtual port of at least one network device included in the at least one detection result message; the first device selects a first path from the at least one path.

[0009] The first device specifies each path between the first device and the second device through a path detection message and a detection result message, and selects a first path among each hop path between the first device and the second device, so that the first message can be transmitted along the first path, and the delay on the first path is measured to achieve binding of delay and path.

[0010] In one possible implementation, the first message includes second indication information, and the second indication information indicates that the second message is returned based on the first message; before the first device obtains the first delay of the first path determined based on the first timestamp and the second timestamp, it also includes: the first device receives at least one second message sent by at least one first network device through which the first message passes, and any second message includes information about the first physical port and the first virtual port of the first network device that sends any second message; the first device determines the first path based on the information about the first physical port and the first virtual port of at least one first network device included in the at least one second message, and the first path includes the first device, the second device and at least one first network device through which the first message passes.

[0011] The first device sends a first message carrying the second indication information, receives a second message returned by the first network device based on the first message, clarifies the first network devices through which the first message passes, and further clarifies the first path for transmitting the first message, so that the first device can bind the obtained first delay to the first path.

[0012] In one possible implementation, a first device obtains a first delay of a first path determined based on a first timestamp and a second timestamp, including: the first device receives a message including the second timestamp sent by a centralized device or a second device based on the first message, and obtains the second timestamp from the message including the second timestamp; the first device determines the first delay of the first path based on the first timestamp and the second timestamp. By obtaining the second timestamp, the first device autonomously calculates the first delay based on the first timestamp and the second timestamp, thereby improving the efficiency of obtaining the first delay.

[0013] In one possible implementation, a first device obtains a first delay of a first path determined based on a first timestamp and a second timestamp, including: the first device sending a message including the first timestamp to a second device or a centralized device, where the first timestamp is used by the second device or the centralized device to determine the first delay of the first path; the first device receiving a message including the first delay sent by the second device or the centralized device, where the first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp; and the first device obtaining the first delay from the message including the first delay. The first device obtains the first delay by sending the first timestamp to the second device or the centralized device and receiving the first delay determined and sent by the second device or the centralized device based on the first timestamp and the second timestamp. This eliminates the need for the first device to independently calculate the first delay, thereby reducing the computational burden on the first device.

[0014] In one possible implementation, after the first device obtains the first delay of the first path determined based on the first timestamp and the second timestamp, it also includes: the first device receives the third message sent by the second device based on the first message, and the transmission path of the third message is the second path; the first device obtains the second delay of the second path determined based on the third timestamp and the fourth timestamp, the third timestamp is the timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is the timestamp of sending the third message recorded by the network card on the second device. The first device can also obtain the second delay of the third message returned by the second device on the second path based on a similar method to obtaining the first delay. The second delay is the network delay between the first device and the second device, which can avoid the long tail delay caused by the end processing delay of the first device and the second device, and realize the binding of delay and path.

[0015] In one possible implementation, the first device obtains the second delay of the second path determined based on the third timestamp and the fourth timestamp, including: the first device receives a message including the fourth timestamp sent by the centralized device or the second device based on the third message, and obtains the fourth timestamp; the first device determines the second delay of the second path based on the third timestamp and the fourth timestamp. The first device obtains the fourth timestamp and autonomously calculates the second delay based on the third timestamp and the fourth timestamp, thereby improving the efficiency of obtaining the second delay.

[0016] In one possible implementation, a first device obtains a second delay of a second path determined based on a third timestamp and a fourth timestamp, including: the first device sending a message including the third timestamp to a second device or a centralized device, where the third timestamp is used by the second device or the centralized device to determine the second delay of the second path; the first device receiving a message including the second delay sent by the second device or the centralized device, where the second delay is determined by the second device or the centralized device based on the third timestamp and the fourth timestamp; and the first device obtaining the second delay from the message including the second delay. The first device obtains the second delay by sending the third timestamp to the second device or the centralized device and receiving the second delay determined and sent by the second device or the centralized device based on the third timestamp and the fourth timestamp, eliminating the need for the first device to independently calculate the second delay, thereby reducing the computational burden on the first device.

[0017] In one possible implementation, the method further includes: for any one of at least one first network devices, the first device determining information about a second virtual port corresponding to a first physical port of the first network device; the first device updating a first message based on the information about the second virtual port to obtain an updated message, wherein the updated message includes information about the second virtual port; the first device sending the updated message to the second device, wherein the transmission path of the updated message is the first path; the first device obtaining a third latency of the first path determined based on a fifth timestamp and a sixth timestamp, wherein the fifth timestamp is the timestamp recorded by the network card on the first device when the updated message was sent, and the sixth timestamp is the timestamp recorded by the network card on the second device when the updated message was received. The first device may update the first message based on the information about the second virtual port, so that the first device and the second device re-establish a connection based on the updated message. If the first path remains unchanged, the third latency of the first path is re-detected when transmitting based on the new virtual port.

[0018] In a second aspect, a delay measurement device is provided, which includes: a transceiver module for sending a first message to a second device, where the transmission path of the first message is a first path; a processing module for obtaining a first delay of the first path determined based on a first timestamp and a second timestamp, where the first timestamp is the timestamp of sending the first message recorded by the network card on the first device, and the second timestamp is the timestamp of receiving the first message recorded by the network card on the second device.

[0019] In one possible implementation, the transceiver module is further used to send a path detection message to the second device, where the path detection message includes first indication information, and the first indication information indicates that a detection result message is returned based on the path detection message; receive at least one detection result message returned based on the path detection message, where the detection result message includes information about the physical port and virtual port of the network device that sent the detection result message; the processing module is further used to determine at least one path between the first device and the second device based on information about the physical port and virtual port of at least one network device included in the at least one detection result message; and select a first path from the at least one path.

[0020] In one possible implementation, the first message includes second indication information, and the second indication information indicates that the second message is returned based on the first message; the transceiver module is also used to receive at least one second message sent by at least one first network device through which the first message passes, and any second message includes information about the first physical port and the first virtual port of the first network device that sends any second message; the processing module is also used to determine the first path based on the information about the first physical port and the first virtual port of at least one first network device included in the at least one second message, and the first path includes the first device, the second device and at least one first network device through which the first message passes.

[0021] In one possible implementation, the processing module is configured to receive a message including a second timestamp sent by a centralized device or a second device based on the first message, obtain the second timestamp from the message including the second timestamp, and determine a first delay of the first path based on the first timestamp and the second timestamp.

[0022] In one possible implementation, a processing module is configured to send a message including a first timestamp to a second device or a centralized device, where the first timestamp is used by the second device or the centralized device to determine a first delay of a first path; receive a message including the first delay sent by the second device or the centralized device, where the first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp; and obtain the first delay from the message including the first delay.

[0023] In one possible implementation, the transceiver module is further used to receive a third message sent by the second device based on the first message, and the transmission path of the third message is the second path; the processing module is further used to obtain a second delay of the second path determined based on a third timestamp and a fourth timestamp, the third timestamp is the timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is the timestamp of sending the third message recorded by the network card on the second device.

[0024] In a possible implementation, the processing module is configured to receive a message including a fourth timestamp sent by the centralized device or the second device based on the third message, obtain the fourth timestamp, and determine the second delay of the second path according to the third timestamp and the fourth timestamp.

[0025] In one possible implementation, the processing module is configured to send a message including a third timestamp to a second device or a centralized device, where the third timestamp is used by the second device or the centralized device to determine a second delay of the second path; receive a message including the second delay sent by the second device or the centralized device, where the second delay is determined by the second device or the centralized device based on the third timestamp and a fourth timestamp; and obtain the second delay from the message including the second delay.

[0026] In one possible implementation, the processing module is further used to determine, for any one of the at least one first network devices, information of a second virtual port corresponding to a first physical port of any one of the first network devices; update the first message based on the information of the second virtual port to obtain an updated message, and the updated message includes information of the second virtual port; the transceiver module is further used to send the updated message to the second device, and the transmission path of the updated message is the first path; the processing module is further used to obtain a third delay of the first path determined based on a fifth timestamp and a sixth timestamp, the fifth timestamp being the timestamp of sending the updated message recorded by the network card on the first device, and the sixth timestamp being the timestamp of receiving the updated message recorded by the network card of the second device.

[0027] In a third aspect, another communication device is provided, comprising: a network interface, a memory, and a processor. The network interface, the memory, and the processor communicate with each other via an internal connection path. The memory is configured to store instructions, and the processor is configured to execute the instructions stored in the memory to control the network interface to receive signals and to control the network interface to send signals. When the processor executes the instructions stored in the memory, the processor performs the method of the first aspect or any possible implementation of the first aspect.

[0028] Optionally, there are one or more processors and one or more memories.

[0029] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0030] In a fourth aspect, a communication system is provided, which includes the device in the above-mentioned second aspect or any possible implementation manner of the second aspect.

[0031] In a fifth aspect, a computer program (product) is provided, which includes: computer program code, which, when run by a computer, enables the computer to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0032] In a sixth aspect, a computer-readable storage medium is provided, which stores a program or instruction. When the program or instruction runs on a computer, the method in the above-mentioned first aspect or any possible implementation of the first aspect is executed.

[0033] In the seventh aspect, a chip is provided, comprising a processor for calling and executing instructions stored in a memory from a memory, so that a computer equipped with the chip executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0034] In an eighth aspect, another chip is provided, comprising: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, a computer equipped with the chip executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0035] It should be understood that the beneficial effects achieved by the technical solutions of the second to eighth aspects of this application and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic diagram of a data center network topology structure provided by an embodiment of the present application;

[0037] FIG2 is a schematic diagram of a path of a data center network provided in an embodiment of the present application;

[0038] FIG3 is a schematic diagram of another data center network path provided in an embodiment of the present application;

[0039] FIG4 is a schematic diagram of an implementation scenario provided in an embodiment of the present application;

[0040] FIG5 is a schematic diagram of a flow chart of a delay measurement method provided in an embodiment of the present application;

[0041] FIG6 is a schematic diagram of a partial structure of a path detection message provided in an embodiment of the present application;

[0042] FIG7 is a schematic diagram of a path detection process provided by an embodiment of the present application;

[0043] FIG8 is a schematic diagram of another path detection process provided by an embodiment of the present application;

[0044] FIG9 is a schematic diagram of another path detection process provided by an embodiment of the present application;

[0045] FIG10 is a schematic diagram of another implementation scenario provided by an embodiment of the present application;

[0046] FIG11 is a schematic diagram of a ping network software architecture provided in an embodiment of the present application;

[0047] FIG12 is a schematic diagram of a data matrix of a measurement provided in an embodiment of the present application;

[0048] FIG13 is a schematic structural diagram of a delay measurement device provided in an embodiment of the present application;

[0049] FIG14 is a schematic structural diagram of a delay measurement device provided in an embodiment of the present application;

[0050] FIG15 is a schematic structural diagram of another delay measurement device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0052] Data centers deployed in a data communication network (DCN) include a large number of servers, which are interconnected through various types of media, such as network cards, switches, routers, cables, and optical fibers.

[0053] Figure 1 illustrates a data center network topology. This topology shows three different DCs: DC1, DC2, and DC3. These three DCs are connected via an inter-DC network. The structure of the intra-DC network can be compared to the internal structure of DC1.

[0054] DC1's internal network can include multiple network devices and multiple servers. Network devices can include devices such as switches or routers that can be used for data forwarding. A connection between any two network devices represents a communication connection between any two network devices. Multiple network devices can be divided into different layers of network devices. Taking switches as an example, the multiple switches within a DC can be divided into spine switches, leaf switches, and top of rack (TOR) switches. Spine switches can also be called spine switches. The spine switches within DC1 form the spine layer of DC1's internal network. Leaf switches can also be called leaf switches. The leaf switches within DC1 form the leaf layer of DC1's internal network. TOR switches can achieve direct communication connections with servers. A TOR switch and multiple servers that can achieve direct communication connections with it can be deployed within the same point of delivery (POD), which is a physical partition of resources. Multiple PODs and leaf switches that can achieve direct communication connections with them can be deployed within the same Podset.

[0055] With the development of cloud computing technology, the amount of data that needs to be interacted and calculated is increasing, and the size of data centers is also growing. Based on the physical facilities of data centers, large-scale distributed services such as search, distributed file systems, distributed storage, and distributed computing (MapReduce) are running.

[0056] Due to the large scale of distributed businesses, which include numerous software systems and components, the coupling relationships between different distributed businesses are complex, and the interactions between components of different distributed businesses must be completed through the network, including but not limited to networks within the same data center and networks across different data centers.

[0057] In large-scale digital communication systems, software and hardware failures are frequent. However, due to the distributed nature of services, many of these failures manifest as increased latency. For example, service anomalies not caused by network failures, such as intermittent connectivity of certain components, a sharp increase in end-to-end latency, or a sharp drop in network throughput for servers within a data center, often manifest as increased latency, leading users to mistakenly believe that these issues are network problems. Therefore, a latency measurement method is urgently needed to determine whether service anomalies are caused by network failures.

[0058] In addition, when the network's service-level agreement (SLA) is broken due to various network failures, it will cause live-site failures, where a live-site failure is any event that may affect customers, partners or revenue. Live-site failures need to be quickly detected, migrated and resolved. However, a data center network may include hundreds of thousands or even millions of servers, hundreds of thousands of switches, millions of cables and optical fibers, and in a data center network, there can be multiple paths for traffic transmission between two servers, and different paths can be used to share different traffic interacting between the two servers. When a path between two servers is abnormal, the traffic transmission task of the abnormal path can be assumed by other paths between the two servers, or the traffic on the abnormal path can be switched to be transmitted by other paths between the two servers. Therefore, the multi-path feature of the data center network can ensure network reliability and traffic load balancing.

[0059] For example, see Figure 2, which is a schematic diagram of a data center network path. This data center network includes spine switches 1 through 4 and multiple leaf switches. Leaf switch 1 can directly communicate with server 1, and leaf switch 2 can directly communicate with server 2. There are four paths between server 1 and server 2: Path 1: Server 1 - Leaf switch 1 - Spine switch 1 - Leaf switch 2 - Server 2; Path 2: Server 1 - Leaf switch 1 - Spine switch 2 - Leaf switch 2 - Server 2; Path 3: Server 1 - Leaf switch 1 - Spine switch 3 - Leaf switch 2 - Server 2; and Path 4: Server 1 - Leaf switch 1 - Spine switch 4 - Leaf switch 2 - Server 2. Each path only defines the devices along the path, not the direction of the path. Each path is bidirectional. Taking path 1 as an example, data can be transmitted from server 1 to server 2 via path 1, and data can also be transmitted from server 2 to path 1 via path 1. If path 1 fails, traffic transmitted via path 1 can be switched to path 2, path 3, or path 1 to ensure normal transmission of that traffic.

[0060] Alternatively, see Figure 3 for another data center network path diagram. This data center network includes core layer network devices, network devices within an aggregation, and edge devices. Network devices 11 and 12 are core layer network devices, network devices 211 to 214 and network devices 221 to 224 are network devices within the aggregation, and edge devices can be servers, i.e., edge devices 31 to 34 can be servers. Multiple different paths exist between two different edge devices.

[0061] Because multiple paths can exist between two devices in a data center network, there is an urgent need for a method that can quickly detect and locate faults. For example, there is an urgent need for a method that can bind each device and the path formed by the device to the delay to quickly determine the location of the network fault, thereby further realizing fault detection and fault location.

[0062] The embodiments of the present application provide a delay measurement method that can accurately measure the end-to-end network delay, determine whether the service anomaly is caused by a network anomaly, and bind the delay to each device and each path to determine the location of the network failure.

[0063] Referring to Figure 4, a schematic diagram of an implementation scenario provided by an embodiment of the present application is shown. The implementation scenario includes a first device 41, a network device 42, and a second device 43. The first device 41 and the second device 43 are communicatively connected via the network device 42. The first device 41 may also be referred to as a source device. The first device 41 may, for example, be a server, a server cluster, or a storage service node in cloud storage. The storage service node includes a storage client, a storage controller, a storage array, and the like. The second device may also be referred to as a destination device. The second device 43 may also be a server, a server cluster, or a storage service node in cloud storage. The network device 42 is a device used for message forwarding in a communication network. For example, it may be a switch, a router, or other forms of forwarding devices. The number of network devices 42 may be one or more, and the embodiment of the present application does not limit this.

[0064] Optionally, the implementation scenario may also include a centralized device (not shown in FIG4 ), which may be respectively connected to the first device 41, the network device 42, and the second device 43. The centralized device may also be referred to as a centralized collector or analyzer. Furthermore, the scenario shown in FIG4 is not limited to the aforementioned data center and may also be applied to network topologies employing other topologies, such as torus networks, dragonfly networks, mesh networks, and CLOS (a next-generation data center network architecture).

[0065] 5 , which shows a flow chart of a delay measurement method provided in an embodiment of the present application. The method can be applied to the implementation scenario shown in FIG. 4 , and includes but is not limited to the following S501 and S502 .

[0066] S501: A first device sends a first message to a second device, where a transmission path of the first message is a first path.

[0067] The embodiment of the present application does not limit the type of the first message. For example, the first message may be a message obtained by extending the precision time protocol (PTP) based on 1588 version 2, or may be another type of message. If the first message is a message obtained by extending the PTP based on 1588v2, the first message may be sent by a PTP agent deployed on the first device.

[0068] Optionally, the first path may be determined by the first device before or after sending the first message. Below, different situations in which the first device determines the first path are described by taking situations A1 and A2 as examples.

[0069] In Case A1, the first device determines a first path before sending a first message. Multiple network devices for forwarding messages exist between the first and second devices, so at least one path may exist between the first and second devices. In this implementation, before the first device sends the first message to the second device, the first device may perform path detection or path sensing on the at least one path between the first and second devices, determine the at least one path between the first and second devices, and select the first path from the at least one path between the first and second devices.

[0070] Exemplarily, the first device may send a path detection message to the second device, the path detection message including first indication information, the first indication information indicating that a detection result message is returned based on the path detection message, and the first indication information is the first mark for path perception. Afterwards, the first device receives at least one detection result message returned based on the path detection message, the detection result message including information about the physical port and virtual port of the network device that sent the detection result message. Among them, the first indication information may be an identifier (flag) with an indicative function, such as characters such as letters or numbers, or the first indication information may also be information about the virtual port of at least one network device or an identifier of at least one network device. The first indication information may be added to a reserved identification bit of the path detection message, the reserved identification bit may be located in the message header of the path detection message, the reserved identification bit may be a field extended by the message or may reuse an existing field in the message.

[0071] Before the first device sends a path detection message to the second device, a matching or identification rule can be issued to the network device in advance, so that when the network device matches or identifies the path detection message, it can generate and return a detection result message to the first device. For example, if the virtual port information in the first indication information in the path detection message includes the virtual port information of network device A, the path detection message is a unicast message. After receiving the path detection message, network device A identifies that the virtual port information in the first indication information includes the virtual port information of this device. It can be considered that the path detection message matches network device A, and thus network device A can generate and return a detection result message to the first device. If the virtual port information in the first indication information in the path detection message does not include the virtual port information of network device B, after receiving the path detection message, network device B determines that the virtual port information in the first indication information does not include the virtual port information of this device, and then it is considered that the path detection message does not match network device B. Therefore, network device B can discard the path detection message. In one possible implementation, the network device can copy the received path detection message, that is, mirror the path detection message. The network device can then add information about the physical port on the network device that received the path detection message to the mirrored message obtained by mirroring, complete the encapsulation and second marking of the mirrored message, and obtain a detection result message, wherein the information of the physical port can be a physical port identifier (ID).

[0072] The path detection message also includes the equal-cost multipath routing (ECMP) hash fields and payload fields corresponding to the path detection message. For example, a partial structure of the path detection message can be seen in Figure 6. The ECMP Hash Fields include the source Internet Protocol (IP) address field, destination IP address field, protocol field, source port information field, and destination port information field of the path detection message. Optionally, the ECMP Hash Fields corresponding to the path detection message may also include the source media access control (MAC) address and destination MAC address of the message.

[0073] The source port information in the path detection message can also be called the source port number, which corresponds to the information of the virtual port in the embodiment of the present application, and is used to identify the virtual port that the path detection message needs to pass through. Since the detection result message includes the same mirror message as the path detection message, and the path detection message includes the information of the virtual port through which the path detection message passes, the detection result message also includes the information of the virtual port corresponding to the network device. Optionally, the network device can also add information of the network device to the mirror message, such as the ID of the network device. After the encapsulation is completed, the network device can return the detection result message to the first device. In addition, after completing the copying of the path detection message, the network device can forward the path detection message to the next hop, and the next hop is determined based on the ECMP Hash Fields in the path detection message.

[0074] Below, the process of a network device determining the next hop of a path detection message and forwarding it is described. After receiving the path detection message, the network device can use the information in all or part of the fields in the ECMP Hash Fields corresponding to the path detection message (for example, the source port information in the source port information field that matches the network device) as a hash key value (HASH KEY) and generate a bucket through a hash algorithm. The bucket can be an array with an index that stores multiple data generated by a hash algorithm. The network device then performs a modulo (MOD) operation on the data in the bucket to obtain the next hop and outbound interface information of the message, and forwards the path detection message to the determined next hop based on the determined outbound interface.

[0075] Each network device that the path detection message passes through can use the above method to determine the next hop and outbound interface information of the path detection message, until the next hop determined by the last network device through which the path detection message passes is the destination device. Because different network devices extract different hash key values, the next hops determined by each network device are also different. Therefore, based on the information of different virtual ports, a single path can be determined from multiple paths for the transmission of the path detection message.

[0076] Optionally, the process of the first device receiving at least one detection result message returned based on the path detection message can be divided into different situations. For example, the first device can receive a detection result message sent directly to the first device by the network device. Alternatively, after generating the detection result message, the network device can send the detection result message to the centralized device. After receiving all the detection result messages corresponding to the same path detection message, the centralized device can send all the detection result messages corresponding to the same path detection message to the first device, so that the first device obtains all the detection result messages corresponding to the path detection message. The method in which each network device through which the path detection message passes directly sends the detection result message to the first device can be called distributed path detection, and the method in which the centralized device concentrates each detection result message and then sends each detection result message to the first device can be called centralized path detection.

[0077] Referring to FIG7 , a schematic diagram of a path detection process is shown. The first device and the second device include network device 21, network device 22, network device 23, network device 24, network device 11, and network device 12. The connection relationship between each network device can be referred to the lines in FIG7 . As shown in FIG7 or FIG8 , multiple paths based on different network devices exist between the first device and the second device. After the first device sends a path detection message to the second device, the path detection message can be transmitted along one of the multiple paths. The path detection message determines one of the multiple paths for transmission of the path detection message. The process can refer to the above description based on FIG6 . Taking the transmission of the path detection message along the path indicated by the arrow in FIG7 as an example, the first device first sends the path detection message to network device 21. Network device 21 mirrors the received path detection message and, based on the virtual port information in the path detection message, determines that the next hop is network device 11 and forwards the path detection message to network device 11. The network device 21 adds the information of the physical port receiving the path detection message and the information of the network device 21 to the mirror message, completes the encapsulation of the mirror message, obtains the detection result message, and sends the detection result message to the first device. Alternatively, as shown in Figure 9, the detection result message can be sent to the centralized device.

[0078] After receiving the path detection message, network device 11 performs the same steps as network device 21, forwarding the path detection message to the determined next-hop network device 12 and returning the detection result message to the first device. Network device 12 then forwards the path detection message to the determined next-hop network device 24 and returns the detection result message to the first device. After receiving the path detection message, network device 24 forwards the path detection message to the determined next-hop second device and returns the detection result message to the first device.

[0079] In a distributed path detection scenario, the path of the path detection message and the path of the detection result message can be asymmetric, that is, the path detection message and the detection result message can pass through the same or different network devices and port sets. For example, in Figure 7, the path detection message reaches network device 24 via the forwarding of network device 21, network device 11, and network device 12. The detection result message generated by network device 24 can be returned to the first device along the forwarding path of the path detection message, or it can be returned to the first device along a path different from that of the path detection message. For example, the detection result message generated by network device 24 can reach the first device via the forwarding of network device 12, network device 11, and network device 22.

[0080] The first device may transmit at least one path detection message and receive at least one detection result message returned by at least one network device based on the at least one path detection message, thereby detecting each path between the first device and the second device. The first device may then determine at least one path between the first device and the second device based on information about the physical port and virtual port of the at least one network device included in the at least one detection result message, and select a first path from the at least one path.

[0081] For example, the first device may determine the physical ports of each network device between the first device and the second device, as well as the communication connection relationship between the physical ports of each network device, based on the physical port information included in each acquired detection result message, to determine at least one path between the first device and the second device. Furthermore, the first device may also determine the correspondence between the information of each virtual port and the information of the physical port based on at least one detection result message.

[0082] After the first device selects a first path from at least one path, it determines the network device on the first path as the first network device, determines and adds information of the virtual ports corresponding to the physical ports of each of the first network devices to the first message, and then sends the first message, so that the first message can be forwarded according to the first path based on the information of the virtual ports corresponding to the physical ports of each of the first network devices.

[0083] In scenario A1, the process of performing path detection based on path detection messages can be referred to as the initial connection establishment process or initial connection establishment phase. The first device selects multiple candidate virtual ports and adds different virtual port information to different path detection messages, allowing different path detection messages to be transmitted along different paths. If N paths exist between the first and second devices, path detection can be performed by selecting information from more than N candidate virtual ports, achieving full coverage of the paths between the first and second devices.

[0084] In case A2, the first device determines the first path after sending the first message. In a possible implementation, the first message may include second indication information, and the second indication information indicates that the second message is returned based on the first message.

[0085] In this case, the first device can receive at least one second message sent by at least one first network device through which the first message passes, and any second message includes information about the first physical port and the first virtual port of the first network device that sends any second message; the first device determines the first path based on the information about the first physical port and the first virtual port of at least one first network device included in the at least one second message, and the first path includes the first device, the second device, and at least one first network device through which the first message passes.

[0086] The second indication information may be the same as the first indication information in case A1. The second indication information may also include an identifier with an indicative function, or may include information about the first virtual port of at least one first network device on the first path. In this implementation, the second message is equivalent to the detection result message in case A1. In addition to being used for delay measurement, the first message can also perform path detection, that is, the first message is given the ability to perform path detection, thereby achieving simultaneous path detection and delay measurement. If the second indication information is information about the first virtual port of at least one first network device, the first message may be a unicast message forwarded based on the first path.

[0087] The process of the first device receiving at least one second message can be referred to as the process of the first device receiving at least one detection result message in situation A1, and will not be repeated here. After receiving the at least one second message, the first device can determine each first network device that the first message passed through, that is, determine the first path of the first message, and determine the correspondence between the physical port information and the virtual port information of each network device.

[0088] In one possible implementation, scenario A1 can be combined with scenario A2. Specifically, before sending the first message, the first device performs path detection to determine each path between the first device and the second device, and proactively adds the first virtual port information and second indication information corresponding to the first physical port of each first network device to the first message, completing the first path marking and making the first message a unicast message. The first message is transmitted along the first path, and after receiving the first message, each first network device on the first path verifies whether the virtual port information in the first message includes its own virtual port information. If so, the first network device mirrors the first message and encapsulates it to obtain a second message, completing the second path marking and sending the mirrored message back to the first device, allowing the first device to again determine whether the network device passing through is the first network device on the first path.

[0089] Optionally, in this implementation, the first message may further include an inband network telemetry header (INT header) or, after receiving the first message, the first network device may further combine INT or in-situ flow information telemetry (IFIT) technology to directly add metadata (MD) and timestamps (TS) such as information about the physical port of the first network device, the ID of the first network device, information about the physical port on which the first network device receives the first message, and information about the physical port on which the first network device sends the first message to the first message, thereby achieving secondary marking of the path and marking of the hop-by-hop delay. The first network device may send the second-marked first message to the determined next hop until the first message is transmitted to the second device. The second device then returns the first message to the first device, allowing the first device to determine whether the network device along which the first message passes is a network device on the first path, thereby further clarifying the transmission path of the first message. Alternatively, the second device, as a tail hop server, may report the second-marked first message to a centralized device, which then analyzes the various tags in the first message.

[0090] S502, the first device obtains a first delay of the first path determined based on a first timestamp and a second timestamp, where the first timestamp is a timestamp recorded by the network card on the first device when sending the first message, and the second timestamp is a timestamp recorded by the network card on the second device when receiving the first message.

[0091] In the embodiment of the present application, different devices can determine the first delay based on the first timestamp and the second timestamp. Depending on the device that determines the first delay, the situation in which the first device obtains the first delay can be divided into multiple different situations, including but not limited to the following situation B1 and situation B2.

[0092] In case B1, the first delay is calculated autonomously by the first device.

[0093] In this case, the first device receives a message including a second timestamp sent by the centralized device or the second device based on the first message, obtains the second timestamp from the message including the second timestamp, and determines the first delay of the first path based on the first timestamp and the second timestamp.

[0094] After receiving the first message, the second device records the second timestamp of receiving the first message through the network card on the second device. After the second device generates the message including the second timestamp, the second device may directly send the message including the second timestamp to the first device, or may send the message including the second timestamp to the centralized device, which then forwards the message including the second timestamp to the first device.

[0095] After receiving the message including the second timestamp, the first device may obtain the second timestamp from the message including the second timestamp, and determine the first path with the first delay according to the first timestamp and the second timestamp.

[0096] Optionally, the first delay may be the difference between the second timestamp and the first timestamp. The first timestamp and the second timestamp may be standard timestamps or non-standard timestamps. If the first timestamp and the second timestamp are standard timestamps, the first device may determine the difference between the second timestamp and the first timestamp as the first delay of the first path. If at least one of the first timestamp and the second timestamp is non-standard timestamp, the standard timestamp corresponding to the non-standard timestamp may be calculated based on the difference between the non-standard timestamp and the standard timestamp, and then the first delay may be determined based on the standard timestamp corresponding to the non-standard timestamp.

[0097] Alternatively, in an embodiment of the present application, it is also possible to ignore whether the first timestamp and the second timestamp are standard timestamps. Since the difference between the timestamp recorded by the network card on the first device and the standard timestamp is certain, the difference between the timestamp recorded by the network card on the second device and the standard timestamp is also certain. Therefore, for different paths, the difference between the delay determined directly based on the first timestamp and the second timestamp and the delay determined based on the standard timestamp is the same, that is, the relative size of the delays corresponding to different paths will not change due to whether the first timestamp and the second timestamp are standard, and the accuracy of path selection directly based on the delay determined based on the first timestamp and the second timestamp will not be affected.

[0098] In case B2, the first delay is calculated by the second device or the centralized device.

[0099] In this case, the first device sends a message including a first timestamp to the second device or the centralized device. The first timestamp is used by the second device or the centralized device to determine the first delay of the first path and receive the message including the first delay sent by the second device or the centralized device. The first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp. The first device obtains the first delay from the message including the first delay.

[0100] After sending the first message to the second device, the first device may send a message including the first timestamp to the second device, so that the second device can determine the first delay of the first path based on the second timestamp and the first timestamp recorded by the network card on the second device. Alternatively, after sending the first message to the second device, the first device may send a message including the first timestamp to the centralized device. After receiving the first message, the second device may send a message including the second timestamp to the centralized device, so that the centralized device can obtain both the first timestamp and the second timestamp, and determine the first delay of the first path based on the first timestamp and the second timestamp.

[0101] In one possible implementation, the message including the first timestamp sent by the first device to the second device may also be the first message itself. That is, after recording the first timestamp, the network card on the first device may mark the first timestamp in the first message, so that when the second device receives the first message, it can obtain the first timestamp. The second device can then send the first timestamp and the second timestamp to the centralized device, enabling the centralized device to obtain the first timestamp and the second timestamp, thereby allowing the centralized device to determine the first delay based on the first timestamp and the second timestamp.

[0102] After the central device or the second device determines the first delay, it may send a message including the first delay to the first device, so that the first device can obtain the first delay from the message including the first delay.

[0103] Optionally, after the first device obtains the first delay of the first path determined based on the first timestamp and the second timestamp, the first device may further receive a third message sent by the second device based on the first message, where the transmission path of the third message is the second path.

[0104] The process of the second device sending the third message to the first device and the process of the first device determining the second path can refer to the process of the first device sending the first message to the second device and the process of the first device determining the first path in the above S501, and will not be repeated here.

[0105] The first device obtains a second delay of the second path determined based on a third timestamp and a fourth timestamp, where the third timestamp is a timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is a timestamp of sending the third message recorded by the network card on the second device.

[0106] When receiving the third message, the first device may record a third timestamp of receiving the third message through the network card on the first device.

[0107] In the embodiment of the present application, different devices can determine the second delay based on the third timestamp and the fourth timestamp. Depending on the device that determines the second delay, the situation where the first device obtains the second delay can be divided into multiple different situations, including but not limited to the following situation C1 and situation C2.

[0108] In case C1, the second delay is calculated autonomously by the first device.

[0109] In this case, the first device receives a message including a fourth timestamp sent by the centralized device or the second device based on the third message, obtains the fourth timestamp, and determines the second delay of the second path according to the third timestamp and the fourth timestamp.

[0110] Among them, the process of the second device or centralized device sending a message including the fourth timestamp to the first device can refer to the process of the second device or centralized device sending a message including the second timestamp to the first device in situation B1, and the process of the first device determining the second delay according to the third timestamp and the fourth timestamp can refer to the process of the first device determining the first delay according to the first timestamp and the second timestamp in situation B1, which will not be repeated here.

[0111] In case C2, the second delay is calculated by the second device or the centralized device.

[0112] In this case, the first device sends a message including a third timestamp to the second device or the centralized device, and the third timestamp is used by the second device or the centralized device to determine the second delay of the second path; and receives a message including the second delay sent by the second device or the centralized device, and the second delay is determined by the second device or the centralized device based on the third timestamp and the fourth timestamp; the first device obtains the second delay from the message including the second delay.

[0113] In this case, the process of the first device obtaining the second delay can refer to the process of the first device obtaining the first delay in case B1, and will not be repeated here.

[0114] In an embodiment of the present application, N unicast virtual ports can be selected, virtual port information can be added to each first message, and unicast normalized delay measurement can be performed on each path between the first device and the second device according to the above-mentioned Case B1, Case B2, Case C1, and Case C2, thereby achieving full coverage of accurate delay measurement. The normalized delay measurement can be a periodic measurement with a fixed frequency. The embodiment of the present application does not limit the size of the fixed frequency. The size of the fixed frequency can be statically configured or dynamically adjusted according to the status of each device.

[0115] In one possible implementation, the correspondence between the information of the physical ports and the information of the virtual ports of the network devices on each path between the first device and the second device may change. Taking the change in the correspondence between the information of the physical port and the information of the virtual port of the first network device A on the first path as an example, the first device again sends the first message for measuring the first delay of the first path to the second device. Since the information of the virtual port in the first message is the information of the virtual port of the first network device A before the change in the correspondence, the first message cannot be transmitted along the first path. Therefore, it is necessary to update the first message according to the changed virtual port information so that the updated message can continue to be transmitted along the first path. In addition, the first device can also determine whether it needs to be re-established based on the connection correlation between the path to be newly connected and the existing path. If necessary, the virtual port information is replaced to re-establish the connection with the second device to achieve adaptive reconstruction.

[0116] Exemplarily, for any one of at least one first network devices, the first device determines information about a second virtual port corresponding to a first physical port of the at least one first network device. The embodiments of the present application do not limit the manner in which the first device determines information about the second virtual port. Exemplarily, the first device may first determine information about a virtual port that is close or similar to information about the first virtual port corresponding to the first physical port of the first network device. For example, if the information about the first virtual port is A1, information about a virtual port that is close to information about the first virtual port may be A2 or A3, etc. The first device may then update a first message based on information about virtual ports that are close to information about the first virtual port, and send the updated first message to the second device, thereby performing full coverage path detection within a small range corresponding to the first network device. Based on the results of the path detection, information about the second virtual port that enables the updated first message to be transmitted along the first path is determined. Optionally, information about the second virtual port may be selected from multiple virtual port information that enables the updated first message to be transmitted along the first path based on a correlation between information about each virtual port and information about the first virtual port, where the correlation between information about the second virtual port and information about the first port is less than a correlation threshold. The correlation threshold may be set empirically.

[0117] Afterwards, when the connection used by the service corresponding to the first message is abnormal, that is, because the correspondence between the first virtual port and the first physical port of the first network device on the first path changes, and the first message cannot be forwarded normally along the first path, the first device can update the first message based on the information of the second virtual port to obtain an updated message, and the updated message includes the information of the second virtual port. The first device sends the updated message to the second device, so that the transmission path of the updated message is the first path. The first device can then obtain the third delay of the first path determined based on the fifth timestamp and the sixth timestamp, wherein the fifth timestamp is the timestamp recorded by the network card on the first device when sending the updated message, and the sixth timestamp is the timestamp recorded by the network card of the second device when receiving the updated message.

[0118] The process of the first device acquiring the third delay determined based on the fifth timestamp and the sixth timestamp can refer to the process of the first device acquiring the first delay determined based on the first timestamp and the second timestamp, and is not repeated here.

[0119] The first device sends the updated message to the second device, switching the old connection to the new connection and implementing a bypass switch for the flow path. The new connection is selected based on its relevance to the old connection, bypassing the anomaly or selecting a less relevant connection, that is, selecting the information of the second virtual port that has less similarity to the information of the first virtual port.

[0120] Referring to FIG10 , another implementation scenario of the latency measurement method provided in an embodiment of the present application is shown, which is used for clock synchronization. The first device can be a master device, and the second device can be a slave device. The first device sends a synchronization (sync) message to the second device, where the synchronization message is the first message described above, and records the first timestamp t1' of sending the synchronization message through the network card. The first device then sends a follow-up message to the second device, where the follow-up message is the message described above including the first timestamp.

[0121] The second device receives the synchronization message and records the second timestamp t2' through the network card. The second device receives the follow message and obtains the first timestamp t1'. The second device sends a delay request message to the first device and records the fourth timestamp t3' of the delay request message through the network card. The delay request message is the third message described above. The first device records the third timestamp t4' of the delay request message through the network card and sends a delay response message to the second device. The delay response message is the message described above that includes the third timestamp.

[0122] After obtaining the four timestamps, the second device calculates the round-trip delay including the first delay and the second delay, where the difference between t1' and the standard timestamp t1 is Δtm, the difference between t2' and the standard timestamp t2 is Δts, the difference between t3' and the standard timestamp t3 is Δts, and the difference between t4' and the standard timestamp t4 is Δtm.

[0123] Round trip delay: (t2′-t1′)+(t4′-t3′)=(t2-t1)+(Δts-Δtm)+(t4-t3)+(Δtm-Δts)=(t2-t1)+(t4-t3).

[0124] In addition, the latency measured by the latency measurement method provided in the embodiments of the present application can also be used in scenarios such as troubleshooting, high-performance computing (HPC), high-performance distributed storage, big data development, artificial intelligence (AI), and traffic scheduling. For example, in a traffic scheduling scenario, the average or variance of the latency of each path can be calculated based on the latency obtained by the first device, and a path recommendation can be made based on a set threshold.

[0125] Below, the beneficial effects of the embodiments of the present application are described in conjunction with relevant technologies.

[0126] Related technology 1 is a method for measuring latency based on a ping network. Referring to FIG11 , a schematic diagram of a ping network software architecture is shown. The software architecture includes a ping network controller (pingmesh controller), a ping network agent (pingmesh agent) and a data storage and analysis (DSA). The ping network controller is connected to the resource management system, and the ping network generator (pingmesh generator) generates a ping list (pinglist file) based on the network graph obtained from the resource management system, so that the servers in the data center perform network probing based on the ping list to ensure the normal provision of network services (web services). The ping list includes a probing list of each server in the data center and the virtual internet protocol (VIP) address of each server. The format of the ping list can be, for example, an extensible markup language (XML) format.

[0127] The ping network agent runs on every server in the data center. For example, Server 1 and Server 2 shown in Figure 11 both run ping network agents. Taking the ping network agent running on Server 2 as an example, the ping network agent obtains a ping list from the ping network controller. Based on this list, it determines the servers that need to detect each other with Server 2 and the VIPs of the servers that need to detect each other with Server 2. It then performs detection according to this list of servers. For example, if Server 2 needs to detect each other with Server 1, Server 2 can send a probe request to Server 1 based on Server 1's VIP. This probe request can be based on the Transmission Control Protocol (TCP) or Hypertext Transfer Protocol (HTTP) probe request. Servers 1 and 2 send probe requests to each other to measure latency and packet loss between the servers. The ping network agent running on the server generates a log based on the detection results and uploads the log to the DSA. The DSA stores the log in a blockchain (cosmos store) according to the VIP of each server. The DSA then stores the log stored in the blockchain in the database through a range operation. In addition, the ping network agent can report detection results to the performance counter aggregator in the DSA, enabling the performance counter aggregator to monitor network status. The performance counter aggregator can also run big data analysis algorithms to analyze and process measurement data for data sheet generation or fault detection and location, enabling visualization of network status and prompting alerts in the event of network failures.

[0128] Since there are a large number of servers in a data center, if any two servers in the data center detect each other, it will bring huge pressure to the ping network proxy. Therefore, a related technology uses hierarchical control to measure latency. For example, within a POD, servers detect each other in pairs; between PODs within a DC, servers within PODm or PODn are sorted, and servers in corresponding positions in the two PODs detect each other; between DCs, a detection relationship is established between selected servers within the DC; N servers are selected in each DC, sorted, and the corresponding servers detect each other. Specifically, within the DC, at least two machines can be selected within each POD; finally, under the control of the controller, a server establishes a detection relationship with up to 5,000 servers to prevent overloading of the server where the ping network proxy is located.

[0129] The global operating status of the DC network is shown in Figure 12: the horizontal and vertical axes represent servers. Figure 12 is a data matrix measured between servers. A blank rectangle represents an empty test result, a rectangle with a check mark represents a normal test result, and a rectangle with a cross represents an abnormal test result.

[0130] (a) in Figure 12 is the data matrix under normal operating conditions of the DC; (b) is the overall failure of a Podset (for example, power outage), resulting in data loss; (c) is the failure of a Podset causing the delay of a Podset to exceed the standard, which is generally manifested as packet loss or broadcast storm in the leaf switch, resulting in packet loss and delay; (d) is the failure of the spine switch causing packet loss on the spine switch, affecting the entire DC, but the internal communication of the Podset is normal.

[0131] Because the latency measured by the related technology includes not only the end-to-end network latency, but also the processing latency on the stack and the latency generated by the CPU parsing the message, it is impossible to determine whether the service anomaly is caused by a network anomaly based on the latency measured by the related technology. In addition, the reliability monitoring of the related technology one relies on the controller and a large number of analyzers. The network is invisible to the service nodes and is a black box to the service. It relies solely on a large number of probes to improve coverage, and the actual coverage is unknown. The large amount of measurement data used to infer faults is complex to analyze, making it impossible to accurately determine the fault and prone to misjudgment. Fault location takes a long time. The dial-up testing solution of the related technology one treats the network as a black box. The blind test requires a large amount of overhead (bandwidth and computing resources) to construct more probes to improve the coverage of paths, links, and forwarding behaviors, making it difficult to detect problems in a timely manner.

[0132] Related Technology 2 considers input / output (IO) latency to be a key factor affecting storage services. It is also a core indicator for customers' high-performance business needs and distinguishes between high, medium, and low-end storage services. Related Technology 2 perceives the end-to-end round-trip time (RTT) at the storage service node. The service layer continuously measures the RTT of the end-to-end connection. The end side distinguishes the load of the path based on the RTT and distributes more traffic to the low-RTT path, thereby ensuring the average latency and tail latency of IO. The RTT measured between nodes includes the protocol stack delay, which has large errors and inaccurate adjustments, resulting in inaccurate service path switching and affecting service performance.

[0133] Related technology 3 deploys INT on network nodes, which measure the latency of each hop and perceive the path of service flows in the network. The end-to-end network latency is then used to differentiate path loads and distribute more traffic to low-latency paths, thereby ensuring average and tail I / O latency. However, INT uses in-band measurement technology, which is not supported by existing devices. INT also requires identifying and mirroring service or measurement traffic, incurring significant network overhead and making it difficult to deploy in existing networks.

[0134] In the embodiment of the present application, the first delay determined by the hardware timestamp recorded by the network card is the network delay on the first path between the first device and the second device, which can avoid the long tail delay caused by the end processing delay of the first device and the second device, and can realize the binding of the delay and the path, and realize the visualization of the delay.

[0135] For example, the embodiment of the present application can sense multiple paths between the first device and the second device, and perform normalized delay measurement on each path based on the hardware timestamp recorded by the network card, without relying on the hop-by-hop delay generated by the network device, and can achieve end-to-end accurate delay measurement. Therefore, the ability to record timestamps through the network card in the embodiment of the present application only requires the support of the network card on the server, and does not require the modification of the network device. It has high deployability and can eliminate the delay generated by the end protocol stack and the central processing unit (CPU) parsing the message. In addition, the embodiment of the present application combines the two capabilities of network delay measurement and path perception, and can achieve full coverage delay measurement of network paths at a relatively low cost through multiple unicast streams. Moreover, it can more accurately allocate network traffic, improve service performance, and reduce long-tail delays caused by network traffic queuing.

[0136] The above describes the latency measurement method provided by the embodiments of the present application. Corresponding to the above method, the embodiments of the present application also provide a latency measurement device. This device is applied to a first device. This device is configured to execute the latency measurement method performed by the first device in FIG. 5 , using the modules shown in FIG. 13 . As shown in FIG. 13 , the latency measurement device provided by the embodiments of the present application includes the following modules.

[0137] The transceiver module 1301 is used to send a first message to the second device, and the transmission path of the first message is the first path; the processing module 1302 is used to obtain a first delay of the first path determined based on a first timestamp and a second timestamp, where the first timestamp is the timestamp of sending the first message recorded by the network card on the first device, and the second timestamp is the timestamp of receiving the first message recorded by the network card on the second device.

[0138] In one possible implementation, the transceiver module 1301 is also used to send a path detection message to the second device, where the path detection message includes first indication information, and the first indication information indicates that a detection result message is returned based on the path detection message; receive at least one detection result message returned based on the path detection message, where the detection result message includes information about the physical port and virtual port of the network device that sent the detection result message; the processing module 1302 is also used to determine at least one path between the first device and the second device based on information about the physical port and virtual port of at least one network device included in the at least one detection result message; and select a first path from the at least one path.

[0139] In one possible implementation, the first message includes second indication information, and the second indication information indicates that the second message is returned based on the first message; the transceiver module 1301 is also used to receive at least one second message sent by at least one first network device through which the first message passes, and any second message includes information about the first physical port and the first virtual port of the first network device that sends any second message; the processing module 1302 is also used to determine the first path based on the information about the first physical port and the first virtual port of at least one first network device included in the at least one second message, and the first path includes the first device, the second device and at least one first network device through which the first message passes.

[0140] In one possible implementation, the processing module 1302 is configured to receive a message including a second timestamp sent by a centralized device or a second device based on the first message, obtain the second timestamp from the message including the second timestamp, and determine a first delay of the first path based on the first timestamp and the second timestamp.

[0141] In one possible implementation, the processing module 1302 is configured to send a message including a first timestamp to a second device or a centralized device, where the first timestamp is used by the second device or the centralized device to determine a first delay of a first path; receive a message including the first delay sent by the second device or the centralized device, where the first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp; and obtain the first delay from the message including the first delay.

[0142] In one possible implementation, the transceiver module 1301 is further used to receive a third message sent by the second device based on the first message, and the transmission path of the third message is the second path; the processing module 1302 is further used to obtain a second delay of the second path determined based on a third timestamp and a fourth timestamp, the third timestamp is the timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is the timestamp of sending the third message recorded by the network card on the second device.

[0143] In one possible implementation, the processing module 1302 is configured to receive a message including a fourth timestamp sent by the centralized device or the second device based on the third message, obtain the fourth timestamp, and determine the second delay of the second path according to the third timestamp and the fourth timestamp.

[0144] In one possible implementation, the processing module 1302 is configured to send a message including a third timestamp to a second device or a centralized device, where the third timestamp is used by the second device or the centralized device to determine a second delay of the second path; receive a message including the second delay sent by the second device or the centralized device, where the second delay is determined by the second device or the centralized device based on the third timestamp and a fourth timestamp; and obtain the second delay from the message including the second delay.

[0145] In one possible implementation, the processing module 1302 is further used to determine, for any one of the at least one first network devices, information of a second virtual port corresponding to a first physical port of any one of the first network devices; update the first message based on the information of the second virtual port to obtain an updated message, and the updated message includes information of the second virtual port; the transceiver module 1301 is further used to send the updated message to the second device, and the transmission path of the updated message is the first path; the processing module 1302 is further used to obtain a third delay of the first path determined based on a fifth timestamp and a sixth timestamp, the fifth timestamp being the timestamp of sending the updated message recorded by the network card on the first device, and the sixth timestamp being the timestamp of receiving the updated message recorded by the network card of the second device.

[0146] It should be understood that the beneficial effects of the device provided in FIG13 above when realizing its functions are the same as the beneficial effects of the method provided in FIG5 , and will not be described in detail here. In addition, when realizing its functions, the device provided in FIG13 only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept. The specific implementation process is detailed in the method embodiment, and will not be described here.

[0147] 14 , which shows a schematic structural diagram of an exemplary delay measurement device 1400 of the present application. The delay measurement device 1400 includes at least one processor 1401 , a memory 1403 , and at least one network interface 1404 .

[0148] The processor 1401 is, for example, a general-purpose central processing unit, a digital signal processor (DSP), a network processor (NP), a GPU, a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASICs) for implementing the solution of the present application, a programmable logic device (PLD), other general-purpose processors or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, or any combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is worth noting that the processor can be a processor that supports the advanced reduced instruction set machine (ARM) architecture. It can implement or execute the various logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on.

[0149] Optionally, latency measurement device 1400 also includes a bus 1402. Bus 1402 is used to transmit information between the various components of latency measurement device 1400. Bus 1402 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Bus 1402 may be classified as an address bus, a data bus, a control bus, etc. For ease of illustration, FIG14 shows only a single line, but this does not imply that there is only one bus or only one type of bus.

[0150] The memory 1403 may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache.

[0151] By way of example and not limitation, many forms of ROM and RAM are available. For example, ROM is a compact disc read-only memory (CD-ROM). RAM includes, but is not limited to, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0152] The memory 1403 may also be other types of storage devices that can store static information and instructions. Or it may be other types of dynamic storage devices that can store information and instructions. Or it may be other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 1403 is, for example, independent and connected to the processor 1401 via the bus 1402. The memory 1403 may also be integrated with the processor 1401.

[0153] The network interface 1404 uses any transceiver-like device to communicate with other devices or communication networks. The communication network can be Ethernet, a radio access network (RAN), or a wireless local area network (WLAN). The network interface 1404 can include a wired network interface or a wireless network interface. Specifically, the network interface 1404 can be an Ethernet interface, such as a Fast Ethernet (FE) interface, a Gigabit Ethernet (GE) interface, an Asynchronous Transfer Mode (ATM) interface, a WLAN interface, a cellular network interface, or a combination thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In some embodiments of the present application, the network interface 1404 can be used for the delay measurement device 1400 to communicate with other devices.

[0154] In a specific implementation, as some embodiments, the processor 1401 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG14 . Each of these processors may be a single-core processor or a multi-core processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0155] In a specific implementation, as some embodiments, the latency measurement device 1400 may include multiple processors, such as processor 1401 and processor 1405 shown in FIG14 . Each of these processors may be a single-core processor or a multi-core processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0156] In some implementations, the memory 1403 is used to store program instructions 1410 for executing the solution of the present application, and the processor 1401 can execute the program instructions 1410 stored in the memory 1403. That is, the delay measurement device 1400 can implement the method provided in the method embodiment, i.e., the method of FIG. 5 , through the processor 1401 and the program instructions 1410 in the memory 1403. The program instructions 1410 may include one or more software modules. Optionally, the processor 1401 itself may also store program instructions for executing the solution of the present application.

[0157] During the specific implementation process, the delay measurement device 1400 of the present application may correspond to a first network element device for executing the above method. The processor 1401 in the delay measurement device 1400 reads the instructions in the memory 1403, so that the delay measurement device 1400 shown in Figure 14 can execute all or part of the steps in the method embodiment.

[0158] Among them, each step of the method shown in Figure 5 is completed by the hardware integrated logic circuit or software instructions in the processor of the delay measurement device 1400. The steps of the method embodiment disclosed in this application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method embodiment in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0159] Referring to Figure 15 , a schematic diagram illustrating the structure of an exemplary latency measurement device 1300 of the present application is shown. Latency measurement device 1300 includes a main control board 1310 and an interface board 1330. Latency measurement device 1300 shown in Figure 15 is configured to perform the operations described in the latency measurement method shown in Figure 5 . Latency measurement device 1300 may be, for example, a switch, router, or controller.

[0160] Main control board 1310, also known as the main processing unit (MPU) or route processor card, controls and manages various components in latency measurement device 1300, including routing calculation, device management, device maintenance, and protocol processing. Main control board 1310 includes a central processing unit (CPU) 1311 and memory 1312.

[0161] Interface board 1330 is also known as a line processing unit (LPU), line card, or service board. It provides various service interfaces and implements data packet forwarding. Service interfaces include, but are not limited to, Ethernet interfaces and POS (Packet over SONET / SDH) interfaces. Ethernet interfaces, for example, are interfaces for flexible Ethernet clients (FlexE Clients). Interface board 1330 includes a central processing unit (CPU) 1331, a network processor (NPU) 1332, a forwarding table memory 1334, and a physical interface card (PIC) 1333.

[0162] The central processing unit 1331 on the interface board 1330 is used to control and manage the interface board 1330 and communicate with the central processing unit 1311 on the main control board 1310 .

[0163] The network processor 1332 is used to implement message forwarding processing. The network processor 1332 can be in the form of a forwarding chip. Specifically, the network processor 1332 is used to forward received messages based on the forwarding table stored in the forwarding table memory 1334. If the destination address of the message is the address of the delay measurement device 1300, the message is sent to the CPU (such as the central processing unit 1311) for processing. If the destination address of the message is not the address of the delay measurement device 1300, the next hop and outgoing interface corresponding to the destination address are searched in the forwarding table based on the destination address, and the message is forwarded to the outgoing interface corresponding to the destination address. The processing of uplink messages includes: processing of the message input interface and forwarding table search; processing of downlink messages includes forwarding table search, etc.

[0164] Physical interface card 1333 implements the physical layer interconnection function. Raw traffic enters interface board 1330 through this card, and processed packets are sent out from this physical interface card 1333. Physical interface card 1333, also known as a daughter card, can be installed on interface board 1330. It is responsible for converting optical and electrical signals into packets, performing a validity check on these packets, and forwarding them to network processor 1332 for processing. In some embodiments, a central processing unit can also perform the functions of network processor 1332, such as implementing software forwarding based on a general-purpose CPU, thus eliminating the need for network processor 1332 in physical interface card 1333.

[0165] Optionally, the delay measurement device 1300 includes multiple interface boards. For example, the delay measurement device 1300 further includes an interface board 1340 . The interface board 1340 includes a central processing unit 1341 , a network processor 1342 , a forwarding table entry memory 1344 , and a physical interface card 1343 .

[0166] Optionally, the latency measurement device 1300 further includes a switching fabric board 1320. The switching fabric board 1320 may also be referred to as a switch fabric unit (SFU). If the latency measurement device includes multiple interface boards 1330, the switching fabric board 1320 is used to exchange data between the interface boards. For example, the interface board 1330 and the interface board 1340 can communicate via the switching fabric board 1320.

[0167] The main control board 1310 and the interface board 1330 are coupled. For example, the main control board 1310, the interface board 1330, the interface board 1340, and the switching network board 1320 are connected to the system backplane via a system bus to achieve intercommunication. In one possible implementation, an inter-process communication (IPC) channel is established between the main control board 1310 and the interface board 1330, and communication between the main control board 1310 and the interface board 1330 is performed via the IPC channel.

[0168] Logically, the latency measurement device 1300 comprises a control plane and a forwarding plane. The control plane includes a main control board 1310 and a central processing unit 1331. The forwarding plane includes various components that perform forwarding, such as a forwarding table entry memory 1334, a physical interface card 1333, and a network processor 1332. The control plane performs router functions, generates forwarding tables, processes signaling and protocol messages, and configures and maintains device status. The control plane sends the generated forwarding tables to the forwarding plane. On the forwarding plane, the network processor 1332 forwards messages received by the physical interface card 1333 based on the forwarding tables sent by the control plane. The forwarding tables sent by the control plane can be stored in the forwarding table entry memory 1334. In some embodiments, the control plane and forwarding plane can be completely separate and not located on the same device.

[0169] It's worth noting that there may be one or more main control boards (SCUs), which can include both active and standby SCUs. There may also be one or more interface boards. The higher the data processing capabilities of the latency measurement device, the more interface boards it provides. The interface boards can also have one or more physical interface cards. There may be no SCUs, one or more SCUs, and multiple SCUs can be used to achieve load balancing and redundant backup. In a centralized forwarding architecture, the latency measurement device may not require SCUs; the interface boards handle service data processing for the entire system. In a distributed forwarding architecture, the latency measurement device can have at least one SCU, which enables data exchange between multiple interface boards, providing high-capacity data exchange and processing capabilities. Therefore, the data access and processing capabilities of latency measurement devices with a distributed architecture are greater than those of devices with a centralized architecture. Alternatively, the latency measurement device can consist of a single card, without a switching network board (SBU). The functions of the interface board and the main control board are integrated on this single card. In this case, the central processing unit (CPU) on the interface board and the CPU on the main control board can be combined into a single CPU on this card to perform the combined functions. This type of device has lower data exchange and processing capabilities (for example, low-end communications equipment such as switches or routers). The specific architecture used depends on the specific network deployment scenario and is not limited here.

[0170] In an exemplary embodiment, a delay measurement system is provided. The system includes a first device, a second device, and a network device. The first device is configured to execute the method executed by the first device in FIG. 5 .

[0171] In an exemplary embodiment, a computer program (product) is provided. The computer program (product) includes: computer program code. When the computer program code is executed by a computer, the computer is caused to perform the method shown in FIG. 5 .

[0172] In an exemplary embodiment, a computer-readable storage medium is provided. The computer-readable storage medium stores a program or instruction. When the program or instruction is executed on a computer, the computer executes the method shown in FIG. 5 .

[0173] In an exemplary embodiment, a chip is provided, including a processor for calling and executing instructions stored in a memory, so that a computer equipped with the chip executes the method shown in FIG5 .

[0174] In an exemplary embodiment, another chip is provided, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the computer equipped with the chip executes the method shown in Figure 5.

[0175] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0176] In this application, the terms "first," "second," and the like are used to distinguish between identical or similar items having substantially the same function or effect. It should be understood that "first," "second," and "nth" do not have a logical or temporal dependency, nor do they limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms. These terms are simply used to distinguish one element from another.

[0177] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0178] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, "plurality of second devices" means two or more second devices. The terms "system" and "network" are often used interchangeably herein.

[0179] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0180] It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the listed items. The term "and / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this application generally indicates that the associated objects are in an "or" relationship.

[0181] It should also be understood that the terms “if” and “if” may be interpreted to mean “when” or “upon” or “in response to determining” or “in response to detecting.” Similarly, the phrases “if it is determined that ” or “if [stated condition or event] is detected” may be interpreted to mean “upon determining ” or “in response to determining ” or “upon detecting [stated condition or event]” or “in response to detecting [stated condition or event],” depending on the context.

[0182] The above description is merely an embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A delay measurement method, characterized in that: The method comprises: The first device sends a first message to the second device, where a transmission path of the first message is a first path; The first device obtains a first delay of the first path determined based on a first timestamp and a second timestamp, wherein the first timestamp is a timestamp of sending the first message recorded by the network card on the first device, and the second timestamp is a timestamp of receiving the first message recorded by the network card on the second device.

2. The method according to claim 1, characterized in that Before the first device sends the first message to the second device, the method further includes: The first device sends a path detection message to the second device, where the path detection message includes first indication information, and the first indication information indicates to return a detection result message based on the path detection message; The first device receives at least one detection result message returned based on the path detection message, wherein the detection result message includes information about a physical port and information about a virtual port of a network device that sends the detection result message; The first device determines at least one path between the first device and the second device based on information about a physical port and a virtual port of at least one network device included in the at least one detection result message; The first device selects the first path among the at least one path.

3. The method according to claim 1, characterized in that The first message includes second indication information, and the second indication information indicates that a second message is returned based on the first message; Before the first device acquires the first delay of the first path determined based on the first timestamp and the second timestamp, the method further includes: The first device receives at least one second message sent by at least one first network device through which the first message passes, wherein any second message includes information about a first physical port and information about a first virtual port of the first network device that sends the any second message; The first device determines the first path based on information about a first physical port and a first virtual port of at least one first network device included in the at least one second message, wherein the first path includes the first device, the second device and the at least one first network device through which the first message passes.

4. The method according to any one of claims 1 to 3, characterized in that: The first device acquires a first delay of the first path determined based on the first timestamp and the second timestamp, including: The first device receives a message including the second timestamp sent by the centralized device or the second device based on the first message, and obtains the second timestamp from the message including the second timestamp; The first device determines a first delay of the first path according to the first timestamp and the second timestamp.

5. The method according to any one of claims 1 to 3, characterized in that: The first device acquires a first delay of the first path determined based on the first timestamp and the second timestamp, including: The first device sends a message including the first timestamp to the second device or the centralized device, where the first timestamp is used by the second device or the centralized device to determine a first delay of the first path; The first device receives a message including the first delay sent by the second device or the centralized device, where the first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp; The first device obtains the first delay from the message including the first delay.

6. The method according to any one of claims 1 to 5, characterized in that: After the first device acquires the first delay of the first path determined based on the first timestamp and the second timestamp, the method further includes: The first device receives a third message sent by the second device based on the first message, where a transmission path of the third message is a second path; The first device obtains a second delay of the second path determined based on a third timestamp and a fourth timestamp, wherein the third timestamp is a timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is a timestamp of sending the third message recorded by the network card on the second device.

7. The method according to claim 6, characterized in that The first device acquires a second delay of the second path determined based on the third timestamp and the fourth timestamp, including: The first device receives a message including the fourth timestamp sent by the centralized device or the second device based on the third message, and acquires the fourth timestamp; The first device determines a second delay of the second path according to the third timestamp and the fourth timestamp.

8. The method according to claim 6, characterized in that The first device acquires a second delay of the second path determined based on the third timestamp and the fourth timestamp, including: The first device sends a message including the third timestamp to the second device or the centralized device, where the third timestamp is used by the second device or the centralized device to determine the second delay of the second path; The first device receives a message including the second delay sent by the second device or the centralized device, where the second delay is determined by the second device or the centralized device based on the third timestamp and the fourth timestamp; The first device obtains the second delay from the message including the second delay.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: For any one of the at least one first network devices, the first device determines information of a second virtual port corresponding to a first physical port of the any one of the first network devices; The first device updates the first message based on the information of the second virtual port to obtain an updated message, where the updated message includes the information of the second virtual port; The first device sends the updated message to the second device, and the transmission path of the updated message is the first path; The first device obtains a third delay of the first path determined based on a fifth timestamp and a sixth timestamp, wherein the fifth timestamp is a timestamp recorded by the network card on the first device for sending the updated message, and the sixth timestamp is a timestamp recorded by the network card of the second device for receiving the updated message.

10. A delay measurement device, characterized in that: The device comprises: A transceiver module, configured to perform the reception and / or transmission related operations performed by the first device in the method according to any one of claims 1 to 9; A processing module, used to perform other operations other than the receiving and / or sending related operations performed by the first device in any one of the methods described in claims 1-9.

11. A delay measurement device, characterized in that: The device includes a processor, which is coupled to a memory; the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor, so that the delay measurement device implements the delay measurement method described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the instruction is loaded and executed by the processor to implement the delay measurement method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product includes a computer program / instructions, and the computer program / instructions are executed by a processor to enable a computer to implement the delay measurement method according to any one of claims 1 to 9.

Images