A method, apparatus, system, and computing device for data collection for network measurement

Abstract

Embodiments of the present application provide a network measurement data collection method, device, system and computing device, related to the field of network measurement, which can reduce the load of CPU in the data collector. The method is applied to a network data collection system, which includes a data collector and a plurality of forwarding devices in a network topology, and an intermediate device arranged between the data collector and the plurality of forwarding devices. The method includes: the intermediate device obtaining registration information of a first flow; the registration information includes: an identifier of the first flow, an identifier of a first forwarding device, and a to-be-stored location of network data of the first flow in a memory of the data collector; the first forwarding device is a forwarding device corresponding to a first hop of the first flow among the plurality of forwarding devices; the intermediate device obtains the network data of the first flow from a target forwarding device based on the identifier of the first forwarding device and the identifier of the first flow; and the intermediate device writes the network data into the to-be-stored location through a write message of RDMA.

Description

Technical Field

[0001] This application relates to the field of network measurement, and more particularly to a data collection method, apparatus, system, and computing device for network measurement. Background Technology

[0002] With the rise of programmable network technology, network measurement technology has become a common method in this field for evaluating network performance, identifying network faults, and detecting network security. The core of network measurement technology is the collection of network data.

[0003] Common methods for collecting network data include: multiple forwarding devices cache the network data they collect in their respective data structures (e.g., sketch data structures); at the end of the measurement window, the multiple forwarding devices send their respective sketch data structures to the data collector, so that the data collector can query the network data in the sketch data structure and store the network data in a preset data structure (e.g., a hash table).

[0004] However, the overhead of the central processing unit (CPU) in the data collector increases because the data collector receives sketch data structures, queries network data for sketch data structures, and stores network data. Therefore, the CPU load in the data collector is increased. Summary of the Invention

[0005] This application provides a data collection method, apparatus, system, and computing device for network measurement, which can reduce the CPU load in the data collector.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a data collection method for network measurement. This method is applied to a network data collection system, which includes: a data collector and multiple forwarding devices in a network topology, and an intermediate device disposed between the data collector and the multiple forwarding devices. The method includes: the intermediate device acquiring first-level registration information; wherein the first-level registration information includes: a first-level identifier, an identifier of a first forwarding device, and the storage location of the first-level network data in the memory of the data collector; the first forwarding device is the forwarding device corresponding to the first hop of the first-level among the multiple forwarding devices; the intermediate device acquiring the first-level network data from a target forwarding device based on the identifier of the first forwarding device and the identifier of the first-level; wherein the target forwarding device is the forwarding device on the first-level routing path among the multiple forwarding devices; and the intermediate device writing the first-level network data to the storage location via a write message from Remote Direct Memory Access (RDMA).

[0008] This embodiment obtains the identifier of the first-level network, the first forwarding device corresponding to the first hop of the first-level network, and the storage location of the first-level network data in the memory of the data collector through an intermediate device; and obtains the first-level network data from the forwarding devices on the first-level routing path based on the identifier of the first forwarding device and the identifier of the first-level network. Then, the first-level network data is written to the storage location in the memory of the data collector via RDMA. Since the process of storing the first-level network data in the storage location does not require the intervention of the data collector, the CPU overhead of the data collector is reduced, thus lowering the CPU load of the data collector.

[0009] In one possible implementation, the intermediate device obtains the first-level registration information by: the intermediate device reading the first-level registration information from the registration information set in the memory of the data collector through an RDMA read message; wherein the registration information set includes: registration information of at least one stream; or, the intermediate device obtains the first-level registration information from the registration information set stored in the intermediate device; wherein the registration information set includes: registration information of at least one stream.

[0010] In this embodiment, the intermediate device directly reads the first-level registration information from the registration information set cached in the memory of the data collector via RDMA, without requiring the data collector to send the first-level registration information to the intermediate device; thus, the CPU overhead of the data collector is reduced, thereby lowering the CPU load of the data collector.

[0011] In addition, by storing the registration information set in the intermediate device, the intermediate device can directly obtain the first-level registration information from the local machine; therefore, it is not necessary to maintain the registration information set in the data collector, and the data collector does not need to intervene in the process of obtaining the first-level registration information; thus, while saving the storage resources of the data collector, the CPU load of the data collector is also reduced.

[0012] In one possible implementation, when the target forwarding device includes a first forwarding device and a second forwarding device, and both the first and second forwarding devices collect first-stream network data, the intermediate device obtains the first-stream network data from the target forwarding device based on the identifier of the first forwarding device and the identifier of the first-stream data. This includes: the intermediate device sending a collection message including the identifier of the first-stream data to the first forwarding device based on the identifier of the first forwarding device; the collection message is used by the first forwarding device to obtain first data based on the identifier of the first-stream data and triggers the second forwarding device to obtain second data based on the identifier of the first-stream data; wherein, the first data is the first-stream network data collected by the first forwarding device; the second data is the first-stream network data collected by the second forwarding device; and the intermediate device receiving the first-stream network data sent by the second forwarding device; wherein, the first-stream network data includes: first data and second data.

[0013] In this embodiment, an intermediate device sends a collection message including the identifier of the first-level network to the first forwarding device based on the identifier of the first forwarding device, thereby triggering the first forwarding device to acquire the network data (i.e., first data) of the first-level network collected by the first forwarding device. Since the first forwarding device is the forwarding device corresponding to the first hop of the first-level network, the first forwarding device can trigger other forwarding devices (i.e., second forwarding devices) on the routing path of the first-level network to acquire the network data (i.e., second data) of the first-level network collected by the second device. This allows the intermediate device to acquire the network data (including first data and second data) of the first-level network along the entire routing path of the first-level network. It eliminates the need for the intermediate device to send collection messages to multiple forwarding devices along the entire routing path of the first-level network separately, thus saving the transmission resources of the intermediate device.

[0014] In one possible implementation, before the intermediate device obtains the first-level registration information, the method further includes: the intermediate device receiving first-level identification information sent by the first forwarding device; the identification information includes: the identifier of the first-level device and the identifier of the first forwarding device; the intermediate device updating the registration information set based on the identification information; wherein the registration information set includes registration information of at least one stream; the updated registration information set includes the first-level registration information.

[0015] This application embodiment saves the storage location corresponding to the first stream by registering the information set, which avoids the problem of reduced data collection efficiency caused by having to determine the storage location corresponding to the first stream every time the network data of the first stream is collected during the data collection stage.

[0016] In one possible implementation, when the registration information set is stored in the memory of the data collector, the intermediate device updates the registration information set based on the first-level identification information, including: the intermediate device determining multiple indices corresponding to the first-level based on multiple different algorithms of a preset data structure; wherein, the multiple different algorithms correspond one-to-one with the multiple indices; the intermediate device sending the identification information and the multiple indices to the data collector; the multiple indices are used by the data collector to determine the storage location and update the registration information set based on the first-level registration information; wherein, the multiple locations indicated by the multiple indices in the memory of the data collector include the storage location.

[0017] This application embodiment uses an intermediate device to determine multiple indices corresponding to the first stream using multiple different algorithms based on a preset data structure; so that the data collector does not need to calculate the multiple indices, but directly determines the storage location corresponding to the first stream from the multiple locations indicated by the multiple indices in the memory of the data collector; therefore, the CPU overhead of the data collector is reduced to a certain extent, thereby reducing the CPU load of the data collector.

[0018] In one possible implementation, the intermediate device updates the registration information set based on the first-level identification information, including: the intermediate device determining the storage location based on the first-level identification and the storage status of the data collector's memory; and the intermediate device updating the registration information set with the first-level registration information.

[0019] In this embodiment, the intermediate device determines the storage location corresponding to the first-level data based on the identifier of the first-level data and the storage status of the data collector's memory; and updates the registration information of the first-level data to the registration information set. The data collector does not need to determine the storage location of the first-level data or update the registration information set; therefore, the CPU overhead of the data collector is reduced, thereby lowering the CPU load of the data collector.

[0020] In one possible implementation, before the intermediate device determines the storage location based on the first-level identifier and the storage status of the memory, the method further includes: the intermediate device obtaining the storage status of the memory from the memory of the data collector through a read message of RDMA.

[0021] In this embodiment, the intermediate device reads messages based on RDMA and directly reads the storage status of the memory from the preset storage structure in the memory of the data collector. The data collector does not need to send the storage status to the intermediate device. Therefore, the CPU overhead of the data collector is reduced, thereby reducing the CPU load of the data collector.

[0022] Secondly, embodiments of this application provide a data collection device for network measurement. This data collection device is applied as an intermediate device in a network data collection system. The system includes: a data collector and multiple forwarding devices in the network topology; the intermediate device is positioned between the data collector and the multiple forwarding devices; the data collection device includes: a transceiver module and a read / write module; the transceiver module is used to acquire first-level registration information; wherein the first-level registration information includes: the identifier of the first-level device, the identifier of the first forwarding device, and the storage location of the first-level network data in the memory of the data collector; the first forwarding device is the forwarding device corresponding to the first hop of the first-level data among the multiple forwarding devices; the transceiver module is also used to acquire network data from a target forwarding device based on the identifier of the first forwarding device and the identifier of the first-level device; wherein the target forwarding device is the forwarding device on the first-level routing path among the multiple forwarding devices; the read / write module is used to write network data to the storage location via write packets of Remote Direct Memory Access (RDMA).

[0023] In one possible implementation, the read / write module is used to read first-level registration information from the registration information set in the memory of the data collector via read messages of RDMA; wherein the registration information set includes: registration information of at least one stream; or, the read / write module is used to obtain first-level registration information from the registration information set stored in the intermediate device; wherein the registration information set includes: registration information of at least one stream.

[0024] In one possible implementation, when the target forwarding device includes a first forwarding device and a second forwarding device, and both the first and second forwarding devices collect network data, the transceiver module is specifically used to send a collection message including a first-level identifier to the first forwarding device based on the identifier of the first forwarding device; the collection message is used by the first forwarding device to obtain first data based on the first-level identifier, and to trigger the second forwarding device to obtain second data based on the first-level identifier; wherein, the first data is the first-level network data collected by the first forwarding device; the second data is the first-level network data collected by the second forwarding device; the transceiver module is also specifically used to receive network data sent by the second forwarding device; wherein, the network data includes: the first data and the second data.

[0025] In one possible implementation, the transceiver module is further configured to receive first-level identification information sent by the first forwarding device; the identification information includes: the first-level identifier and the identifier of the first forwarding device; the read / write module is further configured to update the registration information set based on the identification information; wherein the registration information set includes registration information of at least one stream; the updated registration information set includes the first-level registration information.

[0026] In one possible implementation, where the registration information set is stored in the memory of the data collector, the data collection device includes: a processing module; the processing module is used to determine multiple indices corresponding to the first stream based on multiple different algorithms of a preset data structure; wherein the multiple different algorithms correspond one-to-one with the multiple indices; a transceiver module is used to send identification information and the multiple indices to the data collector; the multiple indices are used by the data collector to determine the storage location and update the registration information set based on the registration information of the first stream; wherein the multiple locations indicated by the multiple indices in memory include the storage location.

[0027] In one possible implementation, the processing module is used to determine the storage location based on the first-level identifier and the memory storage status; the read / write module is used to update the first-level registration information to the registration information set.

[0028] In one possible implementation, the read / write module is used to obtain the memory storage status from the data collector's memory via RDMA read messages.

[0029] Thirdly, embodiments of this application provide a network measurement data collection system, which includes: a data collector and multiple forwarding devices in the network topology, and an intermediate device disposed between the data collector and the multiple forwarding devices; the intermediate device is used to perform the method described in the first aspect and any one of its possible implementations.

[0030] In one possible implementation, the target forwarding device caches the network data of the stream based on the target data structure; the target forwarding device is used to query the first-level network data from the target data structure based on the first-level identifier.

[0031] In one possible implementation, the data collector is used to represent the storage status of the data collector's memory based on a preset data structure; the data collector is used to determine the first position that is not occupied by other streams among the multiple positions corresponding to multiple indices determined by the intermediate device as the storage position.

[0032] Fourthly, embodiments of this application provide a computing device, which includes a memory and a processor, the memory being coupled to the processor; the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the computing device performs the method described in the first aspect and any of its possible implementations.

[0033] Fifthly, embodiments of this application provide a computer storage medium including computer instructions that, when executed on a computing device, cause the computing device to perform the method described in the first aspect and any of its possible implementations.

[0034] In a sixth aspect, embodiments of this application provide a computer program product comprising computer instructions that, when executed on a computer, perform any one of the methods of the first aspect and its possible implementations.

[0035] In a seventh aspect, this application provides a computing device cluster including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method described in the first aspect and any of its possible implementations.

[0036] It should be understood that the beneficial effects of the technical solutions of the second to seventh aspects of this application and the corresponding possible implementations can be referred to the above-described technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0037] Figure 1 A schematic diagram of a Cuckoo hash algorithm provided in an embodiment of this application;

[0038] Figure 2 A schematic diagram of a network data collection system provided in an embodiment of this application;

[0039] Figure 3 This application provides a schematic diagram of the hardware structure of a computing device.

[0040] Figure 4 This is a schematic diagram of a flow registration method provided in an embodiment of this application;

[0041] Figure 5 A schematic flowchart illustrating a method for updating a registration information set, provided in an embodiment of this application;

[0042] Figure 6 A schematic flowchart illustrating another method for updating a registration information set provided in this application embodiment;

[0043] Figure 7 This application provides a schematic flowchart of a data collection method for network measurement.

[0044] Figure 8 A schematic flowchart of a first-class network data acquisition method provided in this application embodiment;

[0045] Figure 9 A schematic diagram of another first-class network data acquisition method provided in this application embodiment;

[0046] Figure 10 This is a schematic diagram of the structure of a data collection device for network measurement provided in an embodiment of this application. Detailed Implementation

[0047] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0048] The terms "first forwarding device" and "second forwarding device," etc., used in the specification and claims of this application are used to distinguish different forwarding devices, rather than to describe a specific order of forwarding devices. For example, "first data" and "second data," etc., are used to distinguish different data, rather than to describe a specific order of data.

[0049] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0050] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple forwarding devices means two or more forwarding devices.

[0051] First, some concepts involved in the data collection method, apparatus, system, and computing device for network measurement provided in the embodiments of this application will be explained as follows:

[0052] RDMA (Remote Direct Memory Access) is a technology that allows the memory of one computer to directly access the memory of another computer without the intervention of the operating system.

[0053] A Bloom filter is a filter used to determine whether an element belongs to a set.

[0054] The Cuckoo hash table is used to store different types of data; any two tables (i.e., storage units) within this Cuckoo hash table are used to store different types of data. The specific process of determining the storage location of a piece of data (e.g., data A) in the Cuckoo hash table includes the following:

[0055] S1: Based on multiple different hash algorithms h1 and h2, determine multiple hash values ​​corresponding to data A. These multiple hash values ​​include h1 (data A) and h2 (data A). h1 (data A) represents the hash value obtained by inputting data A into the hash algorithm corresponding to h1; h2 (data A) represents the hash value obtained by inputting data A into the hash algorithm corresponding to h2.

[0056] S2: Determine whether the position 1 indicated by the first hash value h1 (data A) in the Cuckoo hash table is occupied by other data; if not, determine position 1 as the storage location for data A.

[0057] S3: If so Figure 1 If position 1 shown in Figure (a) is occupied by data B, then data B is removed from position 1; position 1 is determined as the storage location for data A.

[0058] S4: Determine whether the position 2 indicated by the second hash value h2 (data B) among the multiple hash values ​​of the removed data B in the cuckoohash table is occupied by other data.

[0059] S5: If so Figure 1 If position 2 in Figure (b) is occupied by data C, then data C is removed from position 2, and position 2 is determined as the storage location of data B.

[0060] S6: Determine whether the position 1 indicated by the second hash value h2 (data C) among the multiple hash values ​​of the removed data C is occupied by other data.

[0061] S7: If so Figure 1 If position 1 in Figure (c) is occupied by data A, then data A is removed from position 1, and position 1 is determined as the storage location for data C.

[0062] S8: Determine whether the position 3 indicated by the second hash value h2 (data A) of the removed data A in the Cuckoo hash table is occupied by other data.

[0063] S9: If so Figure 1 If position 3 shown in Figure (d) is not occupied by other data, then position 3 will be determined as the storage location for data A.

[0064] A data structure is a specific format in which storage units are organized according to certain logical relationships and storage methods; the storage unit is the physical carrier used by the data structure to store data. Specifically, a data structure is a collection of data elements that have one or more specific relationships with each other.

[0065] Network measurement is the process of real-time or offline monitoring and analysis of the attributes (such as the number of packets in the stream), performance indicators (such as data transmission rate), and device status (such as the number of failures) of data streams obtained from the network.

[0066] Network data: This refers to the data obtained from the network during network measurement, which is to be monitored or analyzed (e.g., the number of packets in the flow, or the size of the flow).

[0067] Stateful measurement, also known as passive measurement, requires maintaining a data structure on the forwarding device to store the network data collected by the forwarding device, and then sending the data structure containing the network data to the data collector after the measurement window ends.

[0068] In data centers, network measurement is a core foundation for many network management tasks; among them, the core of network measurement technology is the collection of network data.

[0069] Common methods for collecting network data include: multiple forwarding devices cache the network data they collect in their respective maintained data structures (e.g., sketch data structures). At the end of the measurement window, these forwarding devices send their respective maintained sketch data structures to a data collector, which then queries the sketch data structure for each flow based on the identifier of each flow. The data collector then determines the storage location for each flow's network data in its memory and stores the network data for each flow in that corresponding storage location.

[0070] However, since the data collector receives the sketch data structure, queries the network data of the sketch data structure, determines the storage location of the network data in the data collector's memory, and stores the network data, the CPU overhead of the data collector increases, thus increasing the CPU load in the data collector.

[0071] Based on this, embodiments of this application provide a data collection method for network measurement; the method sets up an intermediate device between the data collector and multiple forwarding devices; the intermediate device obtains the storage location of the first-stream network data in the memory of the data collector, and writes the first-stream network data directly into the storage location based on RDMA; without the intervention of the data collector, the CPU overhead of the data collector is reduced, thus reducing the CPU load of the data collector.

[0072] This application provides a network measurement data collection method that is applied in, for example, Figure 2 The network data collection system shown includes a data collector 103 and multiple forwarding devices 101 in the network topology, as well as an intermediate device 102 disposed between the multiple forwarding devices 101 and the data collector 103.

[0073] Multiple forwarding devices 101 are used to forward data streams (hereinafter referred to as "streams") and collect network data of the streams (such as the bandwidth occupied by stream A or the number of packets in stream A). Each of the multiple forwarding devices 101 can be a network device such as a switch or router that has the ability to forward data and collect network data.

[0074] It should be noted that the forwarding device in this application can be a programmable network device or a general network device. Specifically, this application does not limit the programming capabilities of the forwarding device.

[0075] It should be understood that the metrics used in network measurements determine which types of network data the forwarding devices need to collect from the flow; that is, if the network measurements are used to measure different metrics, the types of network data collected by the forwarding devices may differ. For example, if the network measurements are used to measure the packet loss rate of a flow during transmission, the network data that the multiple forwarding devices 101 need to collect is the number of packets in each flow forwarded by each forwarding device. If the network measurements are used to measure the transmission rate of a flow, the network data that the multiple forwarding devices 101 need to collect is the timestamp of the received flow collected by each forwarding device.

[0076] For example, suppose the network measurement is used to measure the packet loss rate of a flow during transmission; further suppose that when flow A passes through forwarding device x, the network data collected by forwarding device x shows that flow A includes 10 packets; when flow A passes through forwarding device w, the next hop of forwarding device x, the network data collected by forwarding device w shows that flow A includes 8 packets. Then, based on the network data collected by forwarding device x and forwarding device w, it can be determined that the result of this network measurement is that the packet loss rate of flow A in the transmission path from forwarding device x to forwarding device w is 20%.

[0077] In this application, multiple forwarding devices 101 are connected to intermediate device 102 via communication links. Specifically, each of the multiple forwarding devices 101 may have a direct communication link with the intermediate device 102; alternatively, some of the forwarding devices 101 may have direct communication links with the intermediate device 102, while the remaining forwarding devices may have indirect communication links with the intermediate device 102 through these latter forwarding devices. This application does not limit the connection method between the multiple forwarding devices 101 and the intermediate device 102 in this specific embodiment.

[0078] In this application, multiple forwarding devices 101 are also used to send the collected network data of the stream to intermediate devices 102.

[0079] Intermediate device 102 is used to receive network data streams (e.g., first-stream network data) sent by multiple forwarding devices 101, and obtain the storage location of the first-stream network data in the memory of data collector 103. Then, intermediate device 102 directly writes the first-stream network data to the storage location based on RDMA technology.

[0080] The intermediate device 102 can be a network device with data transmission and reception and read / write capabilities, such as a switch or router; it can also be a computing device with data transmission and reception and read / write capabilities, such as a server or desktop computer; specifically, this application embodiment does not limit the device type of the intermediate device 102.

[0081] The data collector 103 is used to perform network measurement functions by analyzing the network data streams stored in memory. The data collector 103 can be a computing device with storage and computing capabilities, such as a physical server, cloud server, desktop computer, or laptop computer.

[0082] Optionally, if the intermediate device 102 is a network device, the intermediate device 102 and the data collector 103 can be integrated in the same rack.

[0083] For example, Figure 3 It can be Figure 2 The diagram shows the hardware structure of any one of the computing devices in the forwarding device 101, intermediate device 102, and data collector 103. The computing device may include a processor 201, a memory 202, and a communication interface 203. The processor 201, memory 202, and communication interface 203 can be connected to each other via a bus 204 or other means.

[0084] Processor 201 includes one or more CPUs. The CPU can be a single-core CPU or a multi-core CPU. Optionally, processor 201 may also include a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.

[0085] The memory 202 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or optical memory, disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer.

[0086] In the embodiments of this application, Figure 3 When the computing device shown is a forwarding device 101, the memory 202 can store a preset data structure for caching first-stream network data. When the computing device is an intermediate device 102, the memory 202 can store first-stream registration information. When the computing device is a data collector 103, the memory 202 can store first-stream network data.

[0087] In one possible implementation, the memory 202 may exist independently of the processor 201. The memory 202 can be connected to the processor 201 via a bus 204 and is used to store data, operating system, instructions, or program code. When the processor 201 calls and executes the instructions or program code stored in the memory 202, it can implement the relevant steps in the data collection method for network measurement provided in this application embodiment.

[0088] In another possible implementation, the memory 202 can also be integrated with the processor 201.

[0089] The communication interface 203 can be a transceiver module used to communicate with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc. The communication interface 203 can receive commands, messages, or data. The transceiver module can be a transceiver or similar device.

[0090] Optionally, the communication interface 203 can also be a transceiver circuit located within the processor 201, used to implement signal input and signal output of the processor 201. The communication interface 203 can be a wired interface (port), such as a fiber distributed data interface (FDDI) or a gigabit Ethernet (GE) interface, or the communication interface 203 can also be a wireless interface.

[0091] Bus 204 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. The bus can also be divided into serial bus and parallel bus. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0092] It should be understood that, Figure 3 The computing device mentioned is merely one example of a computing device; it can have more than Figure 3 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. For example, a computing device can also include a smart network card, such as a data processing unit (DPU).

[0093] The embodiments of this application can be based on a computing device (such as...) Figure 3 The data collection method for network measurement provided in this application can be executed on multiple computing devices (i.e., a cluster of computing devices). The specific embodiments of this application do not limit the number of computing devices.

[0094] When the network measurement data collection method provided in this application is executed based on multiple computing devices, the memory 202 of the multiple computing devices may store the same instructions for executing the network measurement data collection method, so that the multiple computing devices execute the network measurement data collection method provided in this application independently. Alternatively, the multiple computing devices may each store partial instructions for executing the network measurement data collection method, so that the combination of the multiple computing devices jointly executes the instructions for executing the network measurement data collection method.

[0095] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0096] It is worth noting that the network measurement data collection method provided in this application includes a flow registration phase and a data collection phase. The flow registration phase determines the network data of the flow to be collected (hereinafter referred to as the first flow). Figure 2 The data collection phase involves storing the data to be stored in the memory of the data collector 103 (hereinafter referred to as the data collector) and the correspondence between the data to be stored and the first-order data. The data collection phase is the phase in which the network data of the first-order data collected by the forwarding device is stored in the memory of the data collector at the location to be stored.

[0097] Since the storage location of the first-stream network data in the data collector's memory is determined and stored during the flow registration phase, the first-stream network data collected by the forwarding device can be directly stored in this storage location during the data collection phase. This avoids the problem of increased CPU overhead for the data collector caused by having to determine the storage location of the network data every time before storing the network data. Therefore, the CPU load of the data collector is reduced.

[0098] It should be noted that the network measurement data collection method provided in this application embodiment can be applied to stateful measurement scenarios. Since stateful measurement scenarios involve the forwarding device collecting first-flow network data within a measurement window and storing the first-flow network data in a locally maintained target preset structure; the data collection phase (i.e., periodically collecting network data) only begins after the measurement window ends; therefore, the flow registration phase proposed in this application is executed before the data collection phase.

[0099] The following sections will explain the stream registration phase and the data collection phase separately:

[0100] Stream registration phase

[0101] This application provides a stream registration method, which is applied to... Figure 2 The network data collection system shown; such as Figure 4 As shown, the method includes: S110-S120.

[0102] S110, The first forwarding device sends the first-level identification information to the intermediate device.

[0103] In this application, the first forwarding device can be Figure 2 The forwarding device corresponding to the first hop of the first flow among the multiple forwarding devices 101 shown (hereinafter referred to as "multiple forwarding devices") is the first forwarding device on the first flow forwarding path (or routing path) that receives the first flow; it can also be any forwarding device among all forwarding devices on the first flow forwarding path; or it can be multiple forwarding devices on the first flow forwarding path (e.g., all forwarding devices); the specific first forwarding device is specified by the user or system based on network measurement indicators (or requirements), and this application embodiment does not limit the first forwarding device.

[0104] It should be noted that the embodiments of this application are illustrated using the example of the first forwarding device being the forwarding device corresponding to the first hop of the first flow, and will not be repeated hereafter.

[0105] In this application, the first stream is the stream of network data to be collected in the network (e.g., stream A).

[0106] In this application, the first-level identification information includes: the first-level identifier and the identifier of the first forwarding device.

[0107] In this application, S110 is implemented in which the first forwarding device sends first-level identification information to the intermediate device based on the communication link between the first forwarding device and the intermediate device; correspondingly, the intermediate device receives the first-level identification information sent by the first forwarding device.

[0108] It should be noted that the forwarding device performs the flow registration action only once for a received flow (e.g., flow A) (i.e., S110-S120). Specifically, the first forwarding device may perform the flow registration action when it first receives the first flow, and not perform the flow registration action again when it receives the first flow again.

[0109] Optionally, prior to S110, the method further includes: the first forwarding device receiving the first stream and determining whether the first stream is being received for the first time. Specifically, the first forwarding device determines whether the first stream is being received for the first time by using a Bloom filter and the identifier of the first stream.

[0110] Specifically, the first forwarding device queries the Bloom filter to check if a first-order identifier exists. If the first-order identifier exists in the Bloom filter, it is determined that the first stream is not the first reception; if the first-order identifier does not exist in the Bloom filter, it is determined that the first stream is the first reception, and the first-order identifier is inserted into the Bloom filter. For specific implementation details, please refer to relevant technologies, which will not be elaborated here.

[0111] In this embodiment, the first forwarding device sends the identification information of the first stream to the intermediate device only when the first forwarding device receives the first stream for the first time; thus saving the I / O resources of the first forwarding device.

[0112] S120, the intermediate device updates the registration information set based on the first-level identification information. The registration information set includes registration information for at least one stream; the updated registration information set includes the first-level registration information.

[0113] In this application, the registration information of a flow includes: the identifier of the flow, the identifier of the forwarding device corresponding to the first hop of the flow, and the location of the network data of the flow to be stored in the memory of the data collector; that is, the registration information of the first-order flow includes: the identifier of the first-order flow, the first forwarding device, and the location of the network data of the first-order flow to be stored in the memory of the data collector.

[0114] For ease of description, the storage location of the first-stream network data in the memory of the data collector is referred to as the storage location corresponding to the first-stream, and will not be described in detail hereafter.

[0115] In this application, the storage location corresponding to the first stream can be determined by the data collector or by an intermediate device. Specifically, when the storage location corresponding to the first stream is determined by the data collector, S120 is implemented as described in method 1 below; when the storage location corresponding to the first stream is determined by an intermediate device, S120 is implemented as described in method 2 below.

[0116] Method 1: The specific implementation of S120 is as follows Figure 5 The following are included: S120A-S120D.

[0117] S120A, the intermediate device, determines multiple indices corresponding to the first stream based on multiple different algorithms of a preset data structure.

[0118] In this application, the preset data structure in S120A is the data structure of the network data of the cache stream in the memory of the data collector; wherein, the preset data structure can be a hash table, a tree, or a multidimensional array; the specific embodiments of this application do not specifically limit the preset data structure.

[0119] It should be noted that the embodiments in this application are illustrated using a hash table as the preset data structure, and will not be repeated hereafter.

[0120] It should be understood that, given a hash table as the default data structure, the multiple different algorithms refer to the multiple different hash algorithms corresponding to that hash table. Each hash algorithm corresponds one-to-one with a specific index (i.e., hash value); in other words, the multiple different algorithms in this application correspond one-to-one with multiple indices.

[0121] The implementation method of S120A in this application includes: inputting the first-level identifier into multiple different algorithms to obtain the index (e.g., hash value) of the output of each of the multiple different algorithms; that is, the multiple indices are composed of the index output by each algorithm.

[0122] For example, suppose there are multiple different algorithms, namely hash algorithm h1 and hash algorithm h2. Then, the intermediate device inputs the first-level identifier into h1 and obtains index 1 output by h1; the intermediate device inputs the first-level identifier into h2 and obtains index 2 output by h2; at this time, the multiple indices include index 1 and index 2.

[0123] S120B, the intermediate device sends first-level identification information and multiple indexes to the data collector.

[0124] The implementation of S120B in this application can be as follows: The intermediate device, based on RDMA, writes the first-level identification information and multiple indexes directly into the registration cache in the data collector's memory, enabling the data collector to obtain the first-level identification information and multiple indexes. Alternatively, the intermediate device can send the first-level identification information and multiple indexes to the data collector based on Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) messages, enabling the data collector to obtain the first-level identification information and multiple indexes by parsing the TCP or UDP messages. Specific implementations of S120B in this application do not limit the specific implementation method of S120B.

[0125] When the intermediate device writes the first-level identification information and multiple indexes directly into the memory of the data collector based on RDMA, the CPU of the data collector does not need to receive and parse the message; therefore, the CPU overhead of the data collector is reduced, thereby reducing the CPU load of the data collector.

[0126] The S120C data collector determines the storage location corresponding to the first stream based on multiple indexes.

[0127] It should be understood that the data structure used to cache the network data of the stream in the memory of the data collector is a preset data structure (e.g., a hash table) in S120A. The multiple indexes in this application are determined based on multiple different hash algorithms of this hash table, so each index indicates a position in the hash table.

[0128] It should be noted that, since the preset data structure in this application is stored in the memory of the data collector, each storage unit in the preset data structure used to store network data corresponds to a storage area in the memory of the data collector; therefore, the storage location for storing network data of a certain stream determined from the preset data structure is the storage location for storing the network data of that stream in the memory.

[0129] In this application, the multiple locations indicated by the multiple indexes in the memory of the data collector (i.e., the hash table in memory) include the storage location corresponding to the first stream; that is, S120C determines the storage location corresponding to the first stream from the multiple locations indicated by the multiple indexes in the memory of the data collector.

[0130] This application uses the Cuckoo hash table as an example to illustrate the embodiments, as follows:

[0131] In this application, the data collector can determine the storage location corresponding to the first stream based on the commonly used Cuckoo hash algorithm, as shown in method 11 below; or it can determine the storage location corresponding to the first stream based on the improved Cuckoo hash algorithm, as shown in method 12 below.

[0132] Method 11: The S120C is implemented as follows: the data collector determines the storage location corresponding to the first stream from the cuckoo hash table based on multiple indexes; for specific implementation details, please refer to the relevant technologies, which will not be elaborated here.

[0133] Method 12: The implementation of S120C is as follows: The data collector determines the first position that is not occupied by other streams among the multiple positions indicated by multiple indices in the cuckoo hash table as the storage position corresponding to the first stream.

[0134] For example, suppose there are four indexes, and these four indexes indicate four positions in the Cuckoo hash table: position 1, position 2, position 3, and position 4. The intermediate device first determines whether position 1 is occupied by another stream. If position 1 is not occupied by another stream, then position 1 is determined as the storage location corresponding to the first stream. If position 1 is occupied by another stream, then it determines whether position 2 is occupied by another stream. If position 2 is not occupied by another stream, then position 2 is determined as the storage location corresponding to the first stream. This process is repeated until the storage location corresponding to the first stream is determined.

[0135] In this embodiment, the first position not occupied by other streams among the multiple positions indicated by multiple indexes in the Cuckoo hash table is determined as the storage position corresponding to the first stream. This avoids the problem of increased CPU overhead of the data collector caused by the frequent removal (i.e., kick-out) rules of the commonly used Cuckoo hash algorithm. Therefore, the CPU overhead of the data collector is reduced, thereby reducing the CPU load of the data collector.

[0136] S120D, the data collector updates the registration information set based on first-level registration information;

[0137] In this application, the updated registration information set based on S120D includes: first-level registration information; that is, the updated registration information set includes: the identifier of the first-level device, the identifier of the first forwarding device, and the correspondence between the storage locations corresponding to the first-level device.

[0138] This application embodiment uses an intermediate device to determine multiple indices corresponding to the first stream using multiple different algorithms based on a preset data structure; so that the data collector does not need to calculate the multiple indices, but directly determines the storage location corresponding to the first stream from the multiple locations indicated by the multiple indices in the memory of the data collector; therefore, the CPU overhead of the data collector is reduced to a certain extent, thereby reducing the CPU load of the data collector.

[0139] Method 2: For the specific implementation of S120, please refer to S120a-S120b below, which will not be repeated here.

[0140] This application embodiment saves the storage location corresponding to the first stream by registering the information set, which avoids the problem of reduced data collection efficiency caused by having to determine the storage location corresponding to the first stream every time the network data of the first stream is collected during the data collection stage.

[0141] It should be noted that determining the storage location corresponding to the first stream and updating the registration information set both increase the CPU overhead of the data collector, thus increasing the CPU load of the data collector.

[0142] Based on this, the embodiments of this application provide another implementation of S120, such as... Figure 6 As shown, the method includes: S120a-S120b.

[0143] S120a, the intermediate device determines the storage location corresponding to the first-level identifier and the memory storage status of the data collector.

[0144] In this application, the memory storage status of the data collector is used to indicate which locations in the memory are occupied by network data and which locations are not occupied by network data. Specifically, when the data collector caches data based on a preset data structure in memory, the memory storage status is used to indicate the occupancy of each storage unit in the preset data structure.

[0145] In this application, the storage status of the data collector's memory is obtained by the intermediate device from a preset storage structure in the data collector's memory based on RDMA read messages. Specifically, the intermediate device can read the storage status from the data collector's memory each time S120a is executed; alternatively, the intermediate device can read the storage status from the data collector's memory only once and store it. Specifically, this application embodiment does not limit the number of times the intermediate device obtains the storage status.

[0146] It should be understood that if the intermediate device reads the storage information from the data collector's memory only once and stores the storage information, and subsequently determines the storage location corresponding to a stream based on the storage information, the intermediate device will mark the storage location in the storage information as the storage location occupied by network data.

[0147] In this embodiment, the intermediate device reads messages based on RDMA and directly reads the storage status of the memory from the preset storage structure in the memory of the data collector. The data collector does not need to send the storage status to the intermediate device. Therefore, the CPU overhead of the data collector is reduced, thereby reducing the CPU load of the data collector.

[0148] It should be noted that the implementation of S120a in this application is similar to that of S120A-S120C. For a detailed description of S120a, please refer to the relevant descriptions of S120A-S120C above, which will not be repeated here.

[0149] S120b, the intermediate device updates the first-level registration information to the registration information set.

[0150] In this application, the updated registration information set based on S120b includes: first-level registration information; that is, the updated registration information set includes: the identifier of the first-level device, the identifier of the first forwarding device, and the correspondence between the storage locations corresponding to the first-level device.

[0151] The registration information set in S120b of this application can be stored (or maintained) in the memory of the data collector or in an intermediate device. Wherein, if the registration information set is stored in the memory of the data collector, S120b is implemented as described in Method 1 below. Where the registration information set is stored in an intermediate device, S120b is implemented as described in Method 2 below.

[0152] Method 1: The intermediate device writes the first-level registration information into the registration information set in the memory of the data collector based on the RDMA write message, so that the written registration information set includes the first-level registration information.

[0153] In this embodiment of the application, when the registration information set is stored in the memory of the data collector, the intermediate device directly writes the first-level registration information into the registration information set in the memory of the data collector based on RDMA, without the need for the intervention of the data collector. Therefore, the CPU overhead of the data collector is reduced, thereby reducing the CPU load of the data collector.

[0154] Method 2: The intermediate device inserts the first-level registration information into the registration information set stored in the intermediate device, so that the inserted registration information set includes the first-level registration information.

[0155] In this embodiment, the intermediate device determines the storage location corresponding to the first-level data based on the identifier of the first-level data and the storage status of the data collector's memory; and updates the registration information of the first-level data to the registration information set. The data collector does not need to determine the storage location of the first-level data or update the registration information set; therefore, the CPU overhead of the data collector is reduced, thereby lowering the CPU load of the data collector.

[0156] Data collection phase

[0157] This application provides a data collection method for network measurement, which is applied in... Figure 2 The network data collection system shown; such as Figure 7 As shown, the method includes: S210-S230.

[0158] S210, intermediate devices obtain first-class registration information.

[0159] It should be noted that the registration information set and first-level registration information in S210 of this application are consistent with the registration information set and first-level registration information in S120. For details, please refer to the relevant description in S120, which will not be repeated here.

[0160] It should be understood that since the first-level registration information has been updated to the registration information set during the flow registration phase, the first-level registration information already exists in the registration information set during the data collection phase.

[0161] In this application, S210 is implemented by an intermediate device obtaining first-level registration information from a registration information set. Specifically, when the registration information set is stored in the memory of the data collector, S210 is implemented as described in method 1 below. When the registration information set is stored in an intermediate device, S210 is implemented as described in method 2 below.

[0162] Method 1: The intermediate device reads the first-level registration information from the registration information cached in the memory of the data collector through the read message of RDMA.

[0163] In this embodiment, the intermediate device directly reads the first-level registration information from the registration information set cached in the memory of the data collector via RDMA, without requiring the data collector to send the first-level registration information to the intermediate device; thus, the CPU overhead of the data collector is reduced, thereby lowering the CPU load of the data collector.

[0164] Method 2: The intermediate device obtains the first-level registration information from the registration information set stored in the intermediate device.

[0165] It should be noted that the registration information set can be cached in the memory of the intermediate device or stored in the hard disk of the intermediate device. The specific embodiment of this application does not limit the location of the registration information set in the intermediate device.

[0166] This application embodiment stores the registration information set in the intermediate device, so that the intermediate device can directly obtain the first-level registration information from the local device; therefore, it is not necessary to maintain the registration information set in the data collector, and the data collector does not need to intervene in the process of obtaining the first-level registration information; thus, while saving the storage resources of the data collector, the CPU load of the data collector is also reduced.

[0167] S220: The intermediate device obtains first-level network data from the target forwarding device based on the identifier of the first forwarding device and the identifier of the first-level device.

[0168] In this application, the target forwarding device is the forwarding device on the first-order routing path among a plurality of forwarding devices. The target forwarding device includes: a first forwarding device.

[0169] It should be understood that when the network measurement metric or requirement is to measure the first forwarding device, the target forwarding device is the first forwarding device. When the network measurement metric or requirement is to measure the entire routing path of the first-order system, the target forwarding device is multiple forwarding devices on the first-order routing path (e.g., all forwarding devices on the routing path); specifically, this application does not limit the target forwarding device in its embodiments.

[0170] It should be noted that each forwarding device in the target forwarding device stores the network data of the stream based on the target data structure within the measurement window. This target data structure can be a linked list, a sketch data structure, or a tree; specifically, this application does not limit the target data structure used by the forwarding device to cache network data.

[0171] This application uses the sketch data structure as an example for illustration, and will not be repeated hereafter.

[0172] The implementation of S220 in this application is as follows: the intermediate device sends a first-level identifier to the first forwarding device in the target forwarding device based on the identifier of the first forwarding device; so that the target forwarding device queries the first-level network data from the target data structure based on the first-level identifier, and sends the first-level network data to the intermediate device.

[0173] Specifically, when the target device is the first forwarding device, S220 is implemented as described in Method 1 below; when the target device is any of the forwarding devices on the first-level routing path, S220 is implemented as described in Method 2 below; as follows:

[0174] Method 1: When the target device is the first forwarding device, the specific implementation of S220 is as follows: Figure 8 The following are included: S220A-S220C.

[0175] S220A: The intermediate device sends a collection message, including the first-level identifier, to the first forwarding device based on the identifier of the first forwarding device.

[0176] It should be noted that the implementation of S220A in this application is similar to that of S110. For a detailed description of S220A, please refer to the above description of S110, which will not be repeated here.

[0177] S220B, the first forwarding device, responds to the collected message and obtains first-level network data based on the first-level identifier.

[0178] The implementation of S220B in this application is that the intermediate device queries the first-level network data from the target data structure based on the first-level identifier.

[0179] It should be understood that the process of storing network data of a stream (e.g., stream A) in a sketch data structure includes: calculating multiple hash values ​​corresponding to the identifier of stream A based on multiple hash algorithms; and then repeatedly storing the network data of stream A in multiple locations corresponding to these multiple hash values ​​in the sketch data structure.

[0180] Based on this, the implementation process of S220B will be explained below, taking the target data structure as a sketch data structure as an example:

[0181] Sa and the first forwarding device calculate multiple hash values ​​corresponding to the first-level identifier based on multiple hash algorithms corresponding to the sketch data structure.

[0182] Sb and the first forwarding device respectively obtain multiple network data cached in multiple locations indicated by the multiple hash values ​​in the sketch data structure.

[0183] Sc. The first forwarding device determines the first-order network data from multiple network data that meets the conditions (e.g., the minimum number of recorded packets).

[0184] For example, if the first-stream network data is the count of packets in the first-stream, suppose the multiple network data are: 100 packets, 200 packets, and 300 packets. Since the network data with a large number of packets may have caused a hash collision, resulting in the count of packets from other streams being included in the first-stream packet count, the network data with the smallest number of packets (i.e., 100 packets) is determined as the first-stream network data.

[0185] In this embodiment, the first forwarding device obtains the first-level network data based on the first-level identifier; the data collector does not need to query the first-level network data from the target data structure, thus reducing the CPU overhead of the data collector; therefore, the CPU load of the data collector is reduced.

[0186] S220C: The first forwarding device sends the first stream of network data to the intermediate device.

[0187] It should be noted that the implementation of S220C in this application is similar to that of S110. For a detailed description of S220C, please refer to the above description of S110, which will not be repeated here.

[0188] Method 2: When the target device is one of multiple forwarding devices on a first-class routing path, the implementation of S220 is as follows: S220a-S220e, which will not be repeated here.

[0189] S230: The intermediate device writes the first-stream network data into the storage location in the memory of the data collector through the write message of RDMA.

[0190] This embodiment obtains the identifier of the first-level network, the first forwarding device corresponding to the first hop of the first-level network, and the storage location of the first-level network data in the memory of the data collector through an intermediate device; and obtains the first-level network data from the forwarding devices on the first-level routing path based on the identifier of the first forwarding device and the identifier of the first-level network. Then, the first-level network data is written to the storage location in the memory of the data collector via RDMA. Since the process of storing the first-level network data in the storage location does not require the intervention of the data collector, the CPU overhead of the data collector is reduced, thus lowering the CPU load of the data collector.

[0191] In the case where the target device is one of multiple forwarding devices on a first-class routing path, and these multiple forwarding devices include a first forwarding device and a second forwarding device (i.e., the target device includes both a first forwarding device and a second forwarding device), this application embodiment provides an implementation method for S220, such as... Figure 9 As shown, the method includes: S220a-S220e.

[0192] S220a, The intermediate device sends a collection message including the first-level identifier to the first forwarding device based on the identifier of the first forwarding device.

[0193] It should be noted that the implementation of S220a in this application is similar to that of S110. For a detailed description of S220a, please refer to the above description of S110, which will not be repeated here.

[0194] S220b, the first forwarding device obtains the first data based on the first-level identifier.

[0195] In this application, the first data is the first stream of network data collected by the first forwarding device.

[0196] It should be noted that the implementation of S220b in this application is similar to that of S220B. For a detailed description of S220b, please refer to the above description of S220B, which will not be repeated here.

[0197] S220c, the first forwarding device sends the first data and the first-order identifier to the second forwarding device.

[0198] The implementation of S220c in this embodiment includes: a first forwarding device determining first-level routing information based on a first-level identifier; then, the first forwarding device determining a second forwarding device based on the first-level routing information; and finally, the first forwarding device sending first data and the first-level identifier to the second forwarding device. For a detailed implementation of S220c, please refer to related technologies, which will not be elaborated here.

[0199] S220d, the second forwarding device, obtains second data based on the first-level identifier.

[0200] In this application, the second data is the first stream of network data collected by the second forwarding device.

[0201] It should be noted that the implementation of S220d in this application is similar to that of S220B. For a detailed description of S220d, please refer to the above description of S220B, which will not be repeated here.

[0202] It should be understood that S220c-S220d are the specific implementations that trigger the second forwarding device to obtain the second data based on the first-level identifier.

[0203] S220e, the second forwarding device sends the first stream of network data to the intermediate device; the first stream of network data includes: first data and second data.

[0204] It should be noted that the implementation of S220e in this application is similar to that of S110. For a detailed description of S220e, please refer to the above description of S110, which will not be repeated here.

[0205] In this embodiment, an intermediate device sends a collection message including the identifier of a first-level network to a first-level forwarding device based on the identifier of the first-level forwarding device, thereby triggering the first-level forwarding device to acquire the network data (i.e., first data) of the first-level network collected by the first-level forwarding device. Since the first-level forwarding device is the forwarding device corresponding to the first hop of the first-level network, it can trigger other forwarding devices (i.e., second forwarding devices) on the routing path of the first-level network to acquire the network data (i.e., second data) of the first-level network collected by the second device. This allows the intermediate device to acquire the network data (including first data and second data) of the first-level network along the entire routing path of the first-level network, without requiring the intermediate device to send collection messages to multiple forwarding devices along the entire routing path of the first-level network separately, thus saving the transmission resources of the intermediate device.

[0206] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the network measurement data collection device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0207] This application embodiment can, based on the above method, exemplarily divide the network measurement data collection device into functional modules. For example, the network measurement data collection device may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0208] When dividing each function into modules according to its corresponding function. Figure 10 This diagram illustrates a possible structure of the data collection device for network measurements involved in the above embodiments. For example... Figure 10 As shown, the data collection device for network measurement includes a transceiver module 1010 and a read / write module 1020.

[0209] The transceiver module 1010 is used to obtain first-level registration information; for example, by performing step S210 in the above method embodiment.

[0210] The transceiver module 1010 is also used to obtain first-level network data from the target forwarding device based on the identifier of the first forwarding device and the identifier of the first-level device; for example, by performing step S220 in the above method embodiment.

[0211] The read / write module 1020 is used to write the first stream of network data to the storage location via RDMA write messages; for example, by executing step S230 in the above method embodiment.

[0212] Optionally, the read / write module 1020 is used to read first-level registration information from the registration information set in the memory of the data collector via RDMA read messages.

[0213] or,

[0214] The read / write module 1020 is used to obtain first-level registration information from the registration information set stored in the intermediate device.

[0215] Optionally, when the target forwarding device includes a first forwarding device and a second forwarding device, and both the first forwarding device and the second forwarding device collect first-stream network data, the transceiver module 1010 is used to send a collection message including the first-stream identifier to the first forwarding device based on the identifier of the first forwarding device; for example, performing step S220A in the above method embodiment.

[0216] The transceiver module 1010 is also used to receive first stream network data sent by the second forwarding device; for example, to perform step S220C in the above method embodiment.

[0217] Optionally, the transceiver module 1010 is used to receive first-level identification information sent by the first forwarding device; for example, by performing step S110 in the above method embodiment.

[0218] The read / write module 1020 is used to update the registration information set based on the first-level identification information; for example, by performing step S120 in the above method embodiment.

[0219] Optionally, if the registration information set is stored in the memory of the data collector, the network measurement data collection device includes a processing module 1030.

[0220] The processing module 1030 is used to determine multiple indices corresponding to the first stream based on multiple different algorithms of a preset data structure; for example, it executes step S120A in the above method embodiment.

[0221] The transceiver module 1010 is used to send first-level identification information and multiple indexes to the data collector; for example, to perform step S120B in the above method embodiment.

[0222] Optionally, the processing module 1030 is used to determine the storage location based on the first-class identifier and the storage status of the data collector's memory; for example, by performing step S120a in the above method embodiment.

[0223] The read / write module 1020 is used to update the first-level registration information to the registration information set; for example, by executing step S120b in the above method embodiment.

[0224] Optionally, the read / write module 1020 is used to obtain the memory storage status from the data collector's memory via RDMA read messages.

[0225] Optionally, the read / write module 1020 is used to write the first-level registration information into the registration information set in the memory of the data collector via RDMA write messages.

[0226] This application provides a network measurement data collection system, which includes: a data collector and multiple forwarding devices in the network topology, and an intermediate device disposed between the data collector and the multiple forwarding devices; wherein, the data collector is used to perform any action performed by the data collector in the above embodiments; the intermediate device is used to perform any action performed by the intermediate device in the above embodiments; and the forwarding device is used to perform any action performed by the forwarding device in the above embodiments.

[0227] This application provides a computing device including a memory and at least one processor connected to the memory. The memory is used to store computer program code, which includes computer instructions. When the computer instructions are executed by the at least one processor, the computing device performs the method described in the above embodiments.

[0228] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a network device, such as a switch or router. In some embodiments, the computing device can also be a terminal device such as a desktop computer, laptop computer, or smartphone.

[0229] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data collection method for network measurements.

[0230] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform a data collection method for network measurements.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data collection method for network measurement, characterized in that, The method is applied to a network data collection system, the system comprising: a data collector and multiple forwarding devices in a network topology, and an intermediate device disposed between the data collector and the multiple forwarding devices; the method includes: The intermediate device acquires the registration information of the first-level flow; wherein, the registration information includes: the identifier of the first-level flow, the identifier of the first forwarding device, and the storage location of the network data of the first-level flow in the memory of the data collector; the first forwarding device is the forwarding device corresponding to the first hop of the first-level flow among the plurality of forwarding devices; The intermediate device obtains the network data from the target forwarding device based on the identifier of the first forwarding device and the identifier of the first flow; wherein, the target forwarding device is the forwarding device on the routing path of the first flow among the plurality of forwarding devices; The intermediate device writes the network data to the storage location by remotely accessing the write message of RDMA.

2. The method according to claim 1, characterized in that, The intermediate device obtains first-level registration information, including: The intermediate device reads the registration information of the first stream from the registration information set in the memory of the data collector via an RDMA read message; wherein, the registration information set includes: registration information of at least one stream; or, The intermediate device obtains the registration information of the first stream from the registration information set stored in the intermediate device; wherein, the registration information set includes: registration information of at least one stream.

3. The method according to claim 1 or 2, characterized in that, When the target forwarding device includes the first forwarding device and the second forwarding device, and both the first forwarding device and the second forwarding device collect the network data, The intermediate device obtains the network data from the target forwarding device based on the identifier of the first forwarding device and the identifier of the first flow, including: The intermediate device sends a collection message including the identifier of the first stream to the first forwarding device based on the identifier of the first forwarding device; the collection message is used by the first forwarding device to obtain first data based on the identifier of the first stream, and triggers the second forwarding device to obtain second data based on the identifier of the first stream; wherein, the first data is network data of the first stream collected by the first forwarding device; the second data is network data of the first stream collected by the second forwarding device; The intermediate device receives the network data sent by the second forwarding device; wherein the network data includes: the first data and the second data.

4. The method according to any one of claims 1-3, characterized in that, Before the intermediate device obtains the first-level registration information, the method further includes: The intermediate device receives the identification information of the first stream sent by the first forwarding device; the identification information includes: the identifier of the first stream and the identifier of the first forwarding device; The intermediate device updates the registration information set based on the identification information; wherein the registration information set includes registration information of at least one stream; and the updated registration information set includes the registration information of the first stream.

5. The method according to claim 4, characterized in that, When the registration information set is stored in the memory of the data collector, the intermediate device updates the registration information set based on the identification information, including: The intermediate device determines multiple indices corresponding to the first stream based on multiple different algorithms with a preset data structure; wherein, the multiple different algorithms correspond one-to-one with the multiple indices; The intermediate device sends the identification information and the plurality of indexes to the data collector; the plurality of indexes are used by the data collector to determine the storage location and update the registration information set based on the registration information of the first stream; wherein the plurality of indexes include the storage location among the plurality of locations indicated in the memory.

6. The method according to claim 4, characterized in that, The intermediate device updates the registration information set based on the identification information, including: The intermediate device determines the storage location based on the identifier of the first stream and the storage status of the memory; The intermediate device updates the registration information of the first stream to the registration information set.

7. The method according to claim 6, characterized in that, Before the intermediate device determines the storage location based on the identifier of the first stream and the storage status of the memory, the method further includes: The intermediate device obtains the storage status of the memory from the memory of the data collector through the read message of RDMA.

8. A data collection device for network measurement, characterized in that, The data collection device is used as an intermediate device in a network data collection system, the system comprising: a data collector and multiple forwarding devices in a network topology; the intermediate device is disposed between the data collector and the multiple forwarding devices; the data collection device comprises: a transceiver module and a read / write module. The transceiver module is used to acquire the registration information of the first stream; wherein, the registration information includes: the identifier of the first stream, the identifier of the first forwarding device, and the storage location of the network data of the first stream in the memory of the data collector; the first forwarding device is the forwarding device corresponding to the first hop of the first stream among the plurality of forwarding devices; The transceiver module is further configured to obtain the network data from the target forwarding device based on the identifier of the first forwarding device and the identifier of the first flow; wherein the target forwarding device is the forwarding device on the routing path of the first flow among the plurality of forwarding devices; The read / write module is used to write the network data to the storage location by accessing the write message of RDMA via remote direct memory.

9. The data collection device according to claim 8, characterized in that, The read / write module is used to read the registration information of the first stream from the registration information set in the memory of the data collector via RDMA read messages; wherein, the registration information set includes: registration information of at least one stream; or, The read / write module is used to obtain the registration information of the first stream from the registration information set stored in the intermediate device; wherein, the registration information set includes: registration information of at least one stream.

10. The data collection apparatus according to claim 8 or 9, characterized in that, When the target forwarding device includes the first forwarding device and the second forwarding device, and both the first forwarding device and the second forwarding device collect the network data, The transceiver module is specifically configured to send a collection message including the identifier of the first stream to the first forwarding device based on the identifier of the first forwarding device; the collection message is used by the first forwarding device to obtain first data based on the identifier of the first stream, and to trigger the second forwarding device to obtain second data based on the identifier of the first stream; wherein, the first data is network data of the first stream collected by the first forwarding device; the second data is network data of the first stream collected by the second forwarding device; The transceiver module is further configured to receive the network data sent by the second forwarding device; wherein the network data includes the first data and the second data.

11. The data collection apparatus according to any one of claims 8-10, characterized in that, The transceiver module is further configured to receive the identification information of the first stream sent by the first forwarding device; the identification information includes: the identifier of the first stream and the identifier of the first forwarding device; The read / write module is further configured to update the registration information set based on the identification information; wherein the registration information set includes registration information of at least one stream; and the updated registration information set includes the registration information of the first stream.

12. The data collection apparatus according to claim 11, characterized in that, When the registration information set is stored in the memory of the data collector, the data collection device further includes: a processing module; The processing module is used to determine multiple indices corresponding to the first stream based on multiple different algorithms of a preset data structure; wherein, the multiple different algorithms correspond one-to-one with the multiple indices; The transceiver module is further configured to send the identification information and the plurality of indexes to the data collector; the plurality of indexes are used by the data collector to determine the storage location and update the registration information set based on the registration information of the first stream; wherein the storage location is included among the plurality of locations indicated by the plurality of indexes in the memory.

13. The data collection apparatus according to claim 11, characterized in that, The processing module is specifically used to determine the storage location based on the identifier of the first stream and the storage status of the memory; The read / write module is further configured to update the registration information of the first stream to the registration information set.

14. The data collection apparatus according to claim 13, characterized in that, The read / write module is specifically used to obtain the storage status of the memory from the memory of the data collector through the read message of RDMA.

15. A data collection system for network measurement, characterized in that, The system includes: a data collector and a plurality of forwarding devices in the network topology, and an intermediate device disposed between the data collector and the plurality of forwarding devices; the intermediate device is used to perform the actions performed by the intermediate device in any one of claims 1 to 7.

16. The system according to claim 15, characterized in that, The target forwarding device caches the network data of the stream based on the target data structure; The target forwarding device is used to query the network data of the first stream from the target data structure based on the identifier of the first stream.

17. The system according to claim 15 or 16, characterized in that, The data collector represents the storage status of its memory based on the preset data structure; The data collector is used to determine the first location that is not occupied by other streams among the multiple locations corresponding to the multiple indices determined by the intermediate device as the location to be stored.

18. A computing device, characterized in that, The device includes a memory and a processor, the memory being coupled to the processor; the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the computing device causes the device to perform the method as described in any one of claims 1 to 7.

19. A computer storage medium, characterized in that, Includes computer instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of claims 1 to 7.

20. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 7.

Images