An information processing method, apparatus, system and equipment for terminal-network convergence

Abstract

A converged information transmission method, information processing method, apparatus, and device for end-to-end network integration are disclosed. The method includes: receiving throughput information at the granularity of queue pairs (qp) sent from multiple end devices in the network, as well as qp information transmitted from end-to-end devices; obtaining data packet traffic forwarding information recorded by switches in the network; analyzing the traffic between end-to-end devices in the network based on the throughput information, qp information, and data packet traffic forwarding information to obtain global network traffic information; converting the global traffic information into a traffic matrix of the global network topology and displaying the traffic matrix, which includes the traffic forwarding situation between multiple end devices in the network and the load situation on the forwarding path, thereby clearly reflecting the forwarding process and scale of different traffic in the network within the model; from the network perspective, it also reflects the load situation on different paths, providing convenience for further model analysis, operation troubleshooting, and network operation and maintenance.

Description

Technical Field

[0001] This disclosure relates to the field of network communication, and in particular to an information processing method, apparatus, system and equipment for end-to-end network convergence. Background Technology

[0002] In recent years, as the scale of Distributed Deep Learning (DDL) training models has grown larger, the requirements for the size of Graphics Processing Unit (GPU) clusters in data centers have also increased. DDL mainly includes data communication and data computation processes. Among them, Remote Direct Memory Access (RDMA), due to its kernel bypass feature, can directly transfer data between the memory of different hosts, thereby reducing the CPU usage of data communication processes and allowing more CPU resources to be used for data computation. Therefore, RDMA technology is mainly used for communication in current GPU clusters.

[0003] Most data centers currently use architectures like Fat-Tree and Spine-Leaf based on CLOS (Common Lisp Object System, a network programming framework). These architectures use a combination of multi-layer switches for traffic forwarding, maximizing the performance of RDMA network cards with minimal construction costs. In these architectures, traffic forwarding is primarily achieved through the Equal-cost Multi-path (ECMP) routing algorithm, which uses 5-tuple hashing. While this algorithm generally ensures traffic is forwarded to multiple parallel links, as the DDL framework grows larger and the volume of traffic increases, the ECMP routing algorithm struggles to achieve complete load balancing. Consequently, during DDL model training, network administrators have a poor understanding of how the model actually operates on the network. Although the end-to-end transmission method can be obtained from the model itself, it remains unknown when traffic is forwarded within the CLOS network, hindering model analysis and troubleshooting. Summary of the Invention

[0004] In view of this, this disclosure provides an information processing method, apparatus and device for end-to-end network convergence to solve the problem that network administrators have poor perception of the specific operation mode of the model on the network during the training of distributed deep learning models.

[0005] Firstly, this disclosure provides an information processing method for end-to-end network convergence, the method comprising:

[0006] It receives throughput information at the granularity of queue pairs (qp) sent from multiple end devices in the network, as well as qp information transmitted from end devices. Multiple end devices form multiple qp, and each qp consists of two end devices: a sending end and a receiving end.

[0007] Obtain packet traffic forwarding information recorded by switches between end-to-end devices in the network;

[0008] Based on throughput information, QP information, and packet traffic forwarding information, the traffic between end-to-end devices in the network is analyzed to obtain global network traffic information;

[0009] The global traffic information is converted into a traffic matrix of the global network topology and displayed. The traffic matrix contains the traffic forwarding information between multiple end devices in the network and the load information on the forwarding path.

[0010] The method provided in this paper, when evaluating and analyzing global network traffic, acquires throughput information at the quantile of qp (quantity per second) from network-side terminal devices and end-to-end device qp information. Combining the qp-level throughput information, qp information, and packet forwarding information from the network switch side, it obtains global network traffic information. Finally, this global traffic information is represented by a traffic matrix, which clearly reflects the forwarding process and scale of different traffic flows within the network. From a network perspective, it also reflects the load conditions on different paths, facilitating further model analysis, troubleshooting, and network maintenance. For example, this method assists in data center fault diagnosis and load balancing, enabling overall control of the communication model and specific location of congestion points in scenarios where model size and complexity continuously increase, thereby optimizing communication.

[0011] Secondly, this disclosure provides another end-to-end converged information processing method, applied to any end device in a network, wherein the any end device includes a first end device, and the method includes:

[0012] The kernel function is called to establish a queue pair (qp) with the second-end device, and the extended Berkeley Packet Filter (eBPF) technology is used to obtain the qp information between the first-end device and the second-end device. The qp information includes: qp identifier and qp establishment timestamp.

[0013] Based on the qp identifier, timestamp, and packet sequence number (PSN) counted by the network interface counter, calculate the number of packets sent by qp within a preset time period;

[0014] Calculate the throughput information of QP based on the number of packets sent, and send the throughput information and QP information to the data center.

[0015] The method provided in this paper, on the end side, can obtain QP information between end devices through eBPF technology, including QP identifiers and QP establishment timestamps. Based on this QP information and the packet sequence number (PSN) counted by the network interface card counter, the number of packets sent by the QP within a preset time period is calculated, and then the throughput information of the QP is calculated. Finally, this throughput information of the QP is sent to the data center on the network side, so that the data center can perform global network traffic analysis based on the throughput information of the QP, and generate a traffic matrix that shows the global traffic and load status, which facilitates further model analysis, troubleshooting and network operation and maintenance.

[0016] Thirdly, this disclosure provides an information processing apparatus for end-to-end network convergence, the apparatus comprising:

[0017] The receiving module is used to receive throughput information at the granularity of queue pairs (qp) sent by multiple end devices in the network, as well as qp information transmitted from end to end devices. Multiple end devices form multiple qp, and each qp consists of two end devices: a sending end and a receiving end.

[0018] The acquisition module is used to acquire packet traffic forwarding information recorded by switches between end-to-end devices in the network;

[0019] The processing module is used to analyze the traffic between end-to-end devices in the network based on throughput information, QP information, and packet traffic forwarding information to obtain global network traffic information.

[0020] The conversion module is used to convert global traffic information into a traffic matrix of the global network topology;

[0021] The display module is used to display the traffic matrix, which contains information on traffic forwarding between multiple end devices in the network and the load on the forwarding path.

[0022] Fourthly, this disclosure also provides another end-to-end converged information processing apparatus, which includes:

[0023] The calling module is used to call kernel functions to establish a queue pair (qp) with the second-end device, and to use the extended Berkeley Packet Filter (eBPF) technology to obtain qp information between the first-end device and the second-end device. The qp information includes: qp identifier and qp establishment timestamp.

[0024] The calculation module is used to calculate the number of packets sent by a qp within a preset time period based on the qp identifier, timestamp, and packet sequence number (PSN) counted by the network interface card counter.

[0025] The calculation module is also used to calculate the throughput information of qp based on the number of packets sent;

[0026] The sending module is used to send the throughput information and QP information of QP to the data center.

[0027] Fifthly, this disclosure provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the end-to-end converged information processing method described in the first aspect or any corresponding embodiment.

[0028] Furthermore, the electronic device is also used to perform the end-to-end converged information processing method described in the second aspect or any of its corresponding embodiments.

[0029] In addition, this disclosure provides a computer-readable storage medium storing computer instructions, which are used to cause a computer to perform the method of the first aspect or any of its corresponding embodiments, or to perform the end-to-end converged information processing method of the second aspect or any of its corresponding embodiments.

[0030] Sixthly, this disclosure provides an end-to-end converged system, which includes at least two end devices and a data center, wherein the two or more end devices are connected to each other through at least one switch;

[0031] Furthermore, the data center is used to execute the end-to-end converged information processing method described in the first aspect above; each end device is used to execute the end-to-end converged information processing method described in the second aspect above. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 This is a system architecture diagram of an end-to-end convergence model provided in an embodiment of this disclosure;

[0034] Figure 2 This is a flowchart of a general method for end-to-end network convergence provided in an embodiment of this disclosure;

[0035] Figure 3 This is a flowchart illustrating an end-to-end converged information processing method provided in an embodiment of this disclosure;

[0036] Figure 4 This is a flowchart illustrating another end-to-end converged information processing method provided in this embodiment of the disclosure;

[0037] Figure 5 This is a schematic diagram illustrating a network traffic matrix provided in an embodiment of this disclosure;

[0038] Figure 6 This is a structural block diagram of an end-to-end converged information processing device provided in an embodiment of this disclosure;

[0039] Figure 7 This is a structural block diagram of another end-to-end converged information processing device provided in this embodiment of the disclosure;

[0040] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0042] The technical solution of this disclosure embodiment can be based on the CLOS network architecture. It uses the Equal Cost Multipath (ECMP) routing algorithm based on five-tuple hashes for traffic forwarding and uses switch detection to determine the load status on the path. Although this method determines whether ECMP has achieved load balancing by measuring the throughput of different switch ports, this method of detection on the network side can only be used as a single means and cannot be combined with services to perform targeted optimization based on the communication mode of the DDL framework.

[0043] DDL frameworks are typically composed of one or more communication primitives. Currently, some of the mainstream distributed communication primitives include Broadcast, AllGather, AllReduce, and AllToAll. The main differences between these primitives lie in their data transmission and processing methods. For example, AllReduce involves each server transmitting data to the next server, while AllToAll allows one server to send data to all other servers simultaneously. Different communication primitives can impose varying loads on parallel links in the network. Excessive load on a single link can severely impact the quality of framework communication. Therefore, performing traffic model analysis on the endpoint is a more business-oriented approach.

[0044] In edge traffic model analysis, the Berkeley Packet Filter (BPF) is an architecture used to filter network packets. It exposes raw interfaces at the data link layer to encapsulate and send / receive data packets, while simultaneously sending fixed-format packets to the application layer. Extended Berkeley Packet Filter (eBPF) applies BPF technology to performance tuning, detection, and flow control, providing technical support for edge-side model analysis.

[0045] In RDMA scenarios, queue pairs (qp) are typically used for communication. eBPF is used to detect the two key kernel functions ib_modify_qp and ib_destroy_qp to detect the creation and destruction of qp between the ends.

[0046] This disclosure provides a distributed deep learning communication model analysis method that integrates endpoints and networks, specifically for data center scenarios. Its system architecture is as follows: Figure 1 As shown.

[0047] See Figure 1 This is a system architecture diagram of an end-to-end converged model provided in this embodiment. The system includes at least one end device and a network-side device.

[0048] At least one end device can be a server or a server cluster, such as a high-performance server. In this embodiment, each end device can be referred to as a "host".

[0049] Network-side devices include data centers, data managers, servers, server clusters, etc. Additionally, they may include switches or switching equipment used to connect end-to-end devices. In this embodiment, a data center is used as an example of a network-side device.

[0050] like Figure 1 As shown, the end-side devices include eight devices, host1 to host8. Hosts 1 to 4 are located on one side, and the four end devices, host5 to host8, are located on the other side. The end devices on both sides are connected through multiple switches, such as switch S0, switch S1_0 to switch S1_7, and switch S2.

[0051] Additionally, it also includes data centers (in Figure 1 (Not shown in the image), this data center connects every host and every switch in the entire network, and is used to receive information sent by other devices and to coordinate and control the devices in the entire network.

[0052] In the scenario of Remote Direct Memory Access (RDMA) communication within a data center, this disclosure presents an edge-side QP information detection method. Based on the detection results, the DDL model is analyzed to address the problem of poor model perception capability in data centers, thereby facilitating further assistance in load balancing and other tasks.

[0053] Specifically, the technical solution of this embodiment involves multi-terminal interaction, namely, information interaction between hosts (end-side devices), such as queue pair (QP) information; and information interaction between hosts and data centers (network-side devices), such as hosts reporting QP information and computing throughput to the data center. Additionally, it involves information interaction between intermediate switches and the data center when transmitting data packets end-to-end, thereby realizing converged information transmission and analysis from end-to-end devices to network devices (referred to as "end-to-network").

[0054] Specifically, see Figure 2 This is a flowchart illustrating a general method for end-to-end network convergence provided in this embodiment. The flowchart includes data interaction and network detection between end-side devices and network-side devices.

[0055] On the endpoint, the entire lifecycle of a qp can be detected using eBPF (extended BPF) technology. By counting the packet sequence number (PSN) in each data packet and combining it with the counter of the RDMA network card, the throughput information of each qp can be calculated, and the qp throughput information calculated on the endpoint is reported to the data center on the network side.

[0056] On the network side, the server or data center uses packet sampling and analysis technology to determine the paths taken by packets with different 5-tuples (source IP address, destination IP address, source port number, destination port number, and network protocol number). By combining the data from the terminal side and the network side, the hash status is analyzed, a traffic matrix is ​​drawn, and the traffic matrix is ​​used to determine whether congestion has occurred in the network and to determine which path(s) of traffic caused the phenomenon.

[0057] The following is a detailed explanation of each step in the overall process described above, starting with the method flow for the end-side equipment.

[0058] Example 1

[0059] Example 1 is a method embodiment for an end-side device. See also... Figure 3 This embodiment provides an information processing method for end-to-end network convergence. This method can be applied to any end device (any host) in the network, whereby the any end device includes a first end device (e.g., host1). The method includes:

[0060] Step S101: Call the kernel function to establish a queue pair qp with the second end device, and use the extended Berkeley packet filter eBPF technology to obtain the qp information between the first end device and the second end device.

[0061] The qp information includes: qp identifier, timestamp of qp establishment, and each qp corresponds to a qp identifier.

[0062] Specifically, the end-side detection behavior of any host based on eBPF technology includes: the first end device first calls the device's kernel function to obtain the host status table in real time. The host status table includes: the first end device number, the first end device's IP address, and the communication port number occupied by the qp, etc. Then, based on the contents of the host status table and eBPF technology, the qp information is obtained.

[0063] For example, information such as the five-tuple information, process, creation time (create_time), and qp identifier (qpn) can be stored in JSON format, and this host status table can be stored on the local storage resources on the client side.

[0064] Optionally, a type of QP information can be represented as qp_info, and further, this qp_info is:

[0065] qp_info={src_qpn:{src_ip,dst_ip,src_port,dst_port,dst_qpn,comm,create_time}}

[0066] Wherein, src_qpn represents the source host qpn, src_ip represents the source host IP address, dst_ip represents the destination host IP address, src_port represents the source host port number, dis_port represents the destination host port number, dst_qpn represents the destination host qpn, comm is the process ID, and create_time represents the creation timestamp.

[0067] Additionally, the kernel function called in this step is the ib_modify_qp function.

[0068] Step S102: Calculate the number of packets sent by the qp within a preset time period based on the qp identifier, the timestamp, and the packet sequence number (PSN) counted by the network card counter.

[0069] The network card counter is pre-set on the first end device, and the number of network cards on each end device can be one or more.

[0070] Specifically, in step S102 above, the first end device, such as host1, can obtain the latest N packet sequence numbers (PSNs) sent and received by N qp based on the qpn, the packet establishment timestamp, and the RDMA network card counter.

[0071] Step S103: Calculate the throughput information of the qp based on the number of packets sent. The throughput information (or "throughput") refers to the volume of traffic over a period of time, and this throughput information can be used to measure the traffic volume of each qp.

[0072] By querying the network interface card (NIC) counter information, the traffic sent by each QP within a preset time period can be determined, and the throughput information of each QP can be calculated based on the traffic information. By querying the counter information at fixed intervals, the amount of traffic sent by each QP in each preset time period can be statistically determined, thereby calculating the throughput information at the QP granularity.

[0073] Optionally, the obtained throughput information for QP is represented as psn_info. And the psn_info is:

[0074] psn_info={src_qpn:{timestamp:{psn_counter}}}

[0075] Where timestamp is the timestamp (i.e., the aforementioned establishment time) and psn_counter is the network card's counter value.

[0076] Step S104: Send the throughput information of the qp and the qp information to the data center.

[0077] The throughput information of the qp is the psn_info obtained in step S303. The qp information is the information obtained when establishing the qp pair in step S301. These two parts of information can be sent to the data center simultaneously or separately; this embodiment does not impose any restrictions on this.

[0078] Each host (e.g., host1) periodically uploads edge-side detection data, such as throughput information at the qp level, and psn_info, to the data center responsible for data processing, so that the data center can receive the data and analyze and coordinate the overall network situation based on it.

[0079] It should be noted that this embodiment uses a single end device, host1, as an example to calculate QP information and PSN_info related to host1 and report them to the data center. Other end devices, such as any of the host devices from host2 to host8, can use the same method to report their own detection data, such as throughput information and QP information at the QP granularity, and report this information to the network-side devices. The process of other devices detecting and reporting data is not detailed in this embodiment.

[0080] The method provided in this embodiment allows end-side devices to obtain QP information between end-to-end devices through eBPF technology, including QP identifiers and QP establishment timestamps. Based on this QP information and the packet sequence number (PSN) counted by the network interface card counter, the number of packets sent by the QP within a preset time period is calculated, thereby calculating the QP throughput information and sending it to the data center on the network side. This enables the data center to perform global network traffic analysis based on the QP throughput information, generating a traffic matrix that displays global traffic and load status, facilitating further model analysis, troubleshooting, and network maintenance.

[0081] Additionally, after the first-end device completes the transmission process, it continuously sends qp information. When it detects that another preset function is called on the kernel of the first-end device, it stops sending the qp throughput information (psn_info) and the qp information to the data center.

[0082] The function to be called is ib_destroy_qp.

[0083] In this embodiment, the edge device uses eBPF technology to detect key kernel functions. One kernel function is ib_modify_qp, which is used to detect the creation of a qp pair, and the other kernel function is ib_destroy_qp, which is used to determine the destruction of the qp pair.

[0084] Example 2

[0085] Example 2 is a method embodiment for network-side devices. See also... Figure 4 This disclosure provides an end-to-end converged information transmission method, which can be applied to network-side devices, such as data centers. The method includes:

[0086] Step S201: Receive throughput information at the granularity of queue pairs (qp) sent from multiple end devices in the network, as well as qp information transmitted from end-to-end devices.

[0087] In this system, multiple end devices form multiple qp (Queue Pairs), and each qp consists of two end devices: a sending end and a receiving end. For example... Figure 1As shown, host1 and host5 form one QP pair, and host1 and host7 form another QP pair. Communication between any host on the host side (host1 to host4) and any host on the other side (host5 to host8) will form a QP pair.

[0088] Optionally, based on the aforementioned hosts host1 to host8, it can receive QP information and throughput information for establishing a QP between any two hosts on both sides. This embodiment assumes that it receives N QP information and N throughput information sent by multiple end devices. N ≥ 1 and is a positive integer.

[0089] Step S202: Obtain the packet traffic forwarding information recorded by the switches between end-to-end devices in the network.

[0090] One implementation involves the data center receiving data packet traffic forwarding information sent by a network-side device (such as a switch). This data packet traffic forwarding information is generated by the network-side device (such as a switch) based on data packet records between end-to-end devices, for example, by the switch sampling and analyzing data packets, and then sending it to the data center.

[0091] For example, on the network side, a switch (such as switch s0) periodically samples RDMA packets. The RDMA packets can be all RDMA packets sent by hosts host1 to host4 within a time period. By configuring ACL (Access Control List), the packets are mirrored to a server responsible for packet processing, such as server i.

[0092] Server i acquires these RDMA data packets and parses their content to obtain a 5-tuple information. This 5-tuple information includes the source device IP address, destination device IP address, source device port, destination device port, and transport protocol. Server i uses this 5-tuple information to determine the forwarding result for each QP (Quick Password). For example, it determines which device the data packet sent by the QP will ultimately be forwarded to. Finally, the parsed data packet traffic forwarding information and the QP forwarding result are sent to the data center. The data packet traffic forwarding information can be represented as "switch_info". Furthermore...

[0093] switch_info={switch,[src_ip,dst_ip,src_port,dst_port]}

[0094] Step S203: Analyze the traffic between end-to-end devices in the network based on the throughput information, the qp information, and the data packet traffic forwarding information to obtain the global traffic information of the network.

[0095] Specifically, one implementation includes:

[0096] Step 1: Find a unique identifier that is common to both the throughput information and the QP information. For example, the unique identifier is src_qpn.

[0097] Step 2: Based on the common unique identifier, merge the entries in the throughput information and qp information to generate the first merged entry. That is, perform a merge operation on entries with the same unique identifier, such as src_ip, dst_ip, src_port, timestamp, and psn_counter.

[0098] Step 3: According to the network architecture, match the switches through which the data packets sent by the qp flow in each layer according to the table entry information of each qp in the qp information, and integrate them according to the order in which the data packets pass to obtain multiple traffic paths;

[0099] Specifically, according to the internal layering of the network architecture, the data packets sent by each QP are matched layer by layer based on the table entry information (src_ip, dst_ip, src_port) to find the unique switch through which each layer flows, and the traffic path is obtained by integrating them according to the order of passage.

[0100] Step 4: Add the multiple traffic paths to the table information corresponding to each qp contained in the first merged table entry to obtain the global traffic information.

[0101] Optionally, the global traffic information is represented as "flow_info". Combining the above information such as qp_info, psn_info, and switch_info, the calculated global traffic information is as follows:

[0102] flow_info={src_qpn:{src_ip,dst_ip,src_port,path:[s0,s1_1,s2],timestamp:psn_counter}}

[0103] Step S204: Convert the global traffic information into a global network topology traffic matrix and display the traffic matrix, which includes the traffic forwarding status between the multiple end devices in the network and the load status on the forwarding path.

[0104] Specifically, converting the aforementioned global traffic information into a global network topology traffic matrix and displaying the traffic matrix includes: inputting the global traffic information into a model analysis system for aggregated communication pattern matching processing, outputting the global network topology traffic matrix, and displaying the traffic matrix through a webpage.

[0105] The display of the traffic matrix includes: displaying the traffic matrix in the form of a traffic matrix diagram, such as... Figure 5 As shown, the traffic matrix diagram includes at least one path for end-to-end traffic transmission, and the load status on each path, which is represented by three states: idle, normal, and congested.

[0106] The idle state represents low flow, which can be displayed in green; the normal state represents average flow, which can be displayed in yellow; and the congested state represents high flow, which can be displayed in red.

[0107] Furthermore, step S204 above also includes: receiving query information input by the user, the query information including: the query task ID, the start time and end time of the task; obtaining a time interval based on the task ID, the start time and the end time, and polling the timestamps corresponding to each qp in the first merged table entry within the time interval; counting all timestamps within the time interval, and analyzing the global traffic information of the network based on the qp information corresponding to all timestamps.

[0108] The query process can be performed by network administrators. Taking a training exercise as an example, the user enters the task ID (query information), and the system automatically retrieves all machine IPs and the task's start and end times. Combining the IPs and start and end times, a final traffic matrix is ​​obtained. For example, based on the start time t1 and end time t2, a time interval (t1~t2) is obtained. N timestamps are iterated within the time interval (t1~t2). If there are m timestamps within (t1~t2), these M timestamps are counted; M≤N; the m timestamps correspond to m end-to-end flow paths. In response to the user's task ID, the global traffic information is processed through a collective communication mode, such as NCCL (Nvidia Collective multi-GPU Communication Library), to implement collective communication and point-to-point communication, resulting in a global network topology traffic matrix, which is then displayed on the front end. For high-load traffic / transmission paths, different colors can be used to distinguish them.

[0109] The method provided in this embodiment establishes a QP on the end device side, collects QP information to calculate the throughput information at the QP granularity, and sends this information to the data center. This allows the data center to incorporate throughput information (including PSN information) when analyzing the business model, thereby obtaining the traffic information of each QP on the end side in each time period. This generates a more granular traffic matrix and communication pattern for analyzing the business model, displaying global traffic and load status, and facilitating further model analysis, troubleshooting, and network operation and maintenance.

[0110] Example 3

[0111] This embodiment is a specific implementation based on a combination of Embodiment 1 and Embodiment 2 described above. Specifically, according to the above... Figure 1 The network architecture shown is based on ECMP using 5-tuple hashing, randomly selecting forwarding paths from multiple equivalent paths. Assume the current network topology is as follows: Figure 1 As shown, the direction and quantity of data flow between hosts in the system are unknown. Assuming the starting time is when the Unix timestamp is 1685083856 (the timestamp is represented by a string in this embodiment), each host has only one queue pair qp, with the corresponding relationship as host1→qp1, ..., host8→qp8. The current status of the host is shown in Table 1.

[0112] As shown in Table 1, in step S101 above, the host status table is obtained on the terminal side.

[0113] Table 1. Real-time Host Status Table

[0114] host1 192.168.56.101 50001 host2 192.168.56.102 50002 host3 192.168.56.103 50003 host4 192.168.56.104 50004 host5 192.168.56.105 50005 host6 192.168.56.106 50006 host7 192.168.56.107 50007 host8 192.168.56.108 50008

[0115] Assuming that the hosts within the cluster use the allreduce communication method, the specific detection process is as follows:

[0116] In step S101 above, on the end side based on eBPF technology, each host detects the establishment of qp and records its corresponding qp information (qp_info), as shown in Table 2 (taking the qp for sending information as an example).

[0117] Table 2, qp_info diagram

[0118]

[0119] Each host has a network interface card (NIC), such as an RNIC (RDMA NIC). Each host has multiple NICs. The host obtains the latest sent and received PSNs of QP based on the internal NIC counter.

[0120] Assuming the RNIC records corresponding data every 0.1 seconds, the traffic volume sent by each QP in each time period can be obtained by querying the counter information at fixed intervals, thus obtaining the throughput information at the QP granularity. This corresponds to step S102 above. Taking the psn_info recorded at a certain moment as an example, the results are shown in Table 3.

[0121] Table 3. PSN_Info Illustrated Table

[0122]

[0123]

[0124] Table 3 above corresponds to step S103 of the aforementioned embodiment. After obtaining the throughput information of qp1 to qp8, each host device periodically uploads the detection data (including psn_info and qp information) from the end side to the server responsible for data processing, such as a data center, which is to execute step S104.

[0125] On the network side, based on the sampling of data packets by the switch, the switch periodically samples the passing data packets and mirrors them to the server responsible for data packet processing, such as server i, or the aforementioned data center, by configuring ACLs. This embodiment uses the data center as an example, corresponding to the method flow of Embodiment 2. Specifically, the data center parses the content of the received data packets, determines the hash result to obtain the data packet traffic forwarding information, corresponding to the aforementioned step S202. The data packet traffic forwarding information is switch_info, as shown in Table 4, and the data is uploaded to server 1.

[0126] Table 4. Switch_info Diagram Table

[0127]

[0128]

[0129] After receiving the relevant data from the data processing and packet processing server, the data center analyzes the existing data and calculates the global flow information flow_info (corresponding to steps S203 and S204 of the aforementioned embodiment 2) by integrating information such as qp_info, psn_info, and switch_info.

[0130] In this embodiment, taking the stream sent by qp1 on host1 as an example, the corresponding integrated record results are as follows:

[0131] The `flow_info` statement contains the following information: `{src_qpn=qp1:{src_ip=192.168.56.101,dst_ip=192.168.56.105,src_port=50001,path:[s0,s1_3,s2],timestamp=1685083856:psn_counter=8834903}`. This `flow_info` statement reveals the data packet transmission path between the source device (host1) and the destination device (host5), passing through switches s0, s1_3, and s2.

[0132] Similarly, other transmission paths between end-to-end devices can also be obtained, but this embodiment will not list them all.

[0133] After synthesizing the data entries in Tables 1 to 4, the overall network traffic information (flow_info) of the entire system is obtained. Analysis of this flow_info reveals that the internal cluster communication follows an allreduce pattern, with the communication path being host1→host5→host2→host6→host3→host7→host4→host8. In this embodiment, the identifiers "1~8" for each host are abbreviated, where the arrow "→" indicates travel from one end to the other. The forwarding path of each data flow at the intermediate switches is shown in Table 5.

[0134] Table 5. Flow_info Diagram Table

[0135]

[0136]

[0137] The global traffic information (flow_info) is shown in Table 5 above. During the generation of Table 5, the first merged entry is generated using the common unique identifier (src_qpn) from Tables 2 and 3. Then, according to the network architecture, the system matches the switches (switch_info in Table 4) through which the data packets sent by each qp flow within each layer, based on the entry information of each qp in the qp information. Multiple traffic paths are then integrated according to the order in which the data packets pass, resulting in the global traffic information flow_info, as shown in Table 5. A model analysis based on the traffic matrix is ​​presented. If the user inputs the corresponding task ID, the system automatically queries all machine IPs and the start and end times of the task. Combining the IPs and times, the final traffic matrix is ​​obtained based on flow_info and displayed on the front end. High-load traffic is differentiated by color. Specific effects are shown below. Figure 5 As shown.

[0138] This embodiment provides an end-to-end converged information processing method. On the network side, by adding PSN information, traffic information for each QP (query point per second) on the end side can be obtained within each time period. This allows for fine-grained analysis of the traffic matrix and communication patterns of the service model. From the model's perspective, the forwarding process and scale of different traffic within the model in the network can be clearly seen. From the network's perspective, the load on different paths can be observed, facilitating subsequent model optimization and network maintenance. For example, this method assists in data center fault diagnosis and load balancing, enabling overall control of the communication model and specific location of congestion points in scenarios where model scale and complexity continuously increase, thereby optimizing communication.

[0139] Example 4

[0140] This embodiment also provides an end-to-end converged information processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0141] This embodiment provides an information processing device for end-to-end network convergence, such as... Figure 6 As shown, this device is used to implement the functions of the aforementioned network equipment, such as a data center. The device includes:

[0142] The receiving module 610 is used to receive throughput information in queuing pairs (qp) granularity sent by multiple end devices in the network, as well as qp information transmitted from end-to-end devices. The multiple end devices form multiple qp, and each qp consists of two end devices: a sending end and a receiving end.

[0143] The acquisition module 620 is used to acquire packet traffic forwarding information recorded by switches between end-to-end devices in the network.

[0144] The processing module 630 is used to analyze the traffic between end-to-end devices in the network based on throughput information, qp information and packet traffic forwarding information to obtain global network traffic information.

[0145] The conversion module 640 is used to convert global traffic information into a traffic matrix of the global network topology.

[0146] The display module 650 is used to display the traffic matrix, which contains the traffic forwarding information between multiple end devices in the network and the load information on the forwarding path.

[0147] Optionally, in one possible implementation, the throughput information, qp information, and packet traffic forwarding information are used.

[0148] The processing module 630 is specifically used to find the common unique identifier in the throughput information and qp information; based on the common unique identifier, it merges the table entries in the throughput information and qp information to generate a first merged table entry; according to the network architecture, it matches the data packets sent by each qp in the qp information layer by layer through the switches in each layer, and integrates multiple traffic paths according to the order of data packet passage, and adds the multiple traffic paths to the table entry information corresponding to each qp contained in the first merged table entry to obtain global traffic information.

[0149] Optionally, in another possible implementation, the conversion module 640 is specifically used to input global traffic information into the model analysis system for aggregated communication pattern matching processing, output a traffic matrix of the global network topology, and display the traffic matrix through the display module 650.

[0150] Optionally, in another possible implementation, the processing module 630 is further configured to receive query information input by the user, including: the task ID, the start time and end time of the task; obtain a time interval based on the task ID, start time and end time, and poll the timestamps corresponding to each qp in the first merged table entry within the time interval; count all timestamps within the time interval, and analyze the global network traffic information based on the qp information corresponding to all timestamps.

[0151] Optionally, in another possible implementation, the acquisition module 620 is specifically used to receive data packet traffic forwarding information sent by the network-side device, the data packet traffic forwarding information being generated by the network-side device based on the records of the switches between the end-to-end devices.

[0152] Optionally, in another possible implementation, the display module 650 is specifically used to display a traffic matrix in the form of a traffic matrix diagram. The traffic matrix diagram includes: at least one path for end-to-end traffic transmission, and the load status on each path, represented by three states: idle, normal, and congested.

[0153] In addition, this embodiment also provides another end-to-end converged information processing device, such as... Figure 7 As shown, the device has the functionality of the aforementioned end device, such as any host. The device includes:

[0154] Module 710 is called to call kernel functions to establish a queue pair (qp) with the second-end device, and uses the extended Berkeley Packet Filter (eBPF) technology to obtain qp information between the first-end device and the second-end device. The qp information includes: qp identifier and qp establishment timestamp.

[0155] The calculation module 720 is used to calculate the number of packets sent by qp within a preset time period based on qp identifier, timestamp, and packet sequence number PSN counted by network card counter.

[0156] In addition, the calculation module 720 is also used to calculate the throughput information of qp based on the number of packets sent.

[0157] The sending module 730 is used to send the throughput information and QP information of QP to the data center.

[0158] Optionally, in one possible implementation of this embodiment, the calling module 710 is specifically used to call a kernel function to obtain the host status table in real time, and obtain qp information based on the contents of the host status table and eBPF technology. The host status table includes: the first-end device number, the IP address of the first-end device, and the communication port number occupied by the qp.

[0159] Optionally, in another possible implementation of this embodiment, the above-mentioned device further includes a stop module, which is used to stop sending the throughput information and qp information to the data center when the calling module 710 detects that another preset function is called on the kernel of the first end device.

[0160] It should be noted that the above-mentioned device in this embodiment is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0161] In addition, the above-mentioned device may include other more or fewer units / modules, and the further functional descriptions of each of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0162] Example 5

[0163] This disclosure also provides an electronic device having the above-described features. Figure 6 or Figure 7 The information processing device shown.

[0164] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of this disclosure, such as... Figure 8As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces).

[0165] In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory units, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0166] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0167] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0168] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0169] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0170] The electronic device also includes input and output devices. The input devices can receive input numerical or character information and generate key signal inputs related to user settings and function control of the electronic device, such as touchscreens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output devices may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0171] The electronic device also includes a communication interface 30 for communicating with other devices or communication networks.

[0172] In addition, the electronic device may include one or more network cards / RDMA network cards.

[0173] Optionally, the aforementioned electronic device can be a network-side device, such as a data center or a data center controller; or it can be an end device, such as a single-end server or server cluster. Furthermore, the electronic device can also be a switch used to implement end-to-end packet forwarding functionality.

[0174] In addition, this embodiment also provides an end-to-end converged system, which includes at least two end devices and a data center. Furthermore, the system may also include other network devices, such as switches.

[0175] In this system, two or more end devices are connected through at least one switch, and the data center is connected to each end device via the TCP / IP protocol.

[0176] The structure of each of the above devices can be the same as described above. Figure 8 The electronic devices shown have the same or similar structures. Furthermore, the aforementioned functions can be performed within this system. Figure 2 The information processing method for end-to-end network convergence shown; each end device is used to perform the aforementioned... Figure 3 The method flow shown above, the data center is used to execute the aforementioned... Figure 4 The method described above obtains global network traffic information and a global network topology traffic matrix, thereby enabling global network traffic detection, such as RDMA flow-level detection. From a network perspective, the global network topology traffic matrix reflects the load on different paths, facilitating further model analysis, troubleshooting, and network maintenance.

[0177] Furthermore, this solution utilizes network path probing tools, such as eBPF technology, to construct and send UD-type probe packets with the same five-tuple as the service on the endpoint side. Simultaneously, it configures ACLs on the network side for mirroring analysis, thus implementing a path probing tool. Compared to traceroute, the technical solution provided in this embodiment analyzes faster and is not limited by the switch's CPU, supporting simultaneous probing of a large number of paths.

[0178] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware, firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and to be stored on a local storage medium after being downloaded over a network, so that the methods described herein can be stored on such software processing on a storage medium using a general-purpose computer, a special-purpose processor, or programmable or special-purpose hardware.

[0179] The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; furthermore, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the end-to-end converged information processing method shown in the above embodiments is implemented.

[0180] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An information processing method for terminal-network convergence, characterized in that, The method includes: The system receives throughput information at the granularity of queue pairs (qp) sent from multiple end devices in the network, as well as qp information transmitted from end-to-end devices, wherein the multiple end devices form multiple qp, and each qp consists of two end devices: a sending end and a receiving end. Obtain packet traffic forwarding information recorded by switches between end-to-end devices in the network; Based on the throughput information, the qp information, and the packet traffic forwarding information, the traffic between end-to-end devices in the network is analyzed to obtain the global traffic information of the network; The global traffic information is converted into a global network topology traffic matrix and the traffic matrix is ​​displayed. The traffic matrix contains the traffic forwarding information between the multiple end devices in the network and the load information on the forwarding path. The step of analyzing the traffic between end-to-end devices in the network based on the throughput information, the qp information, and the packet traffic forwarding information to obtain the global traffic information of the network includes: Find the common unique identifier in the throughput information and the qp information; Based on the common unique identifier, the table entries in the throughput information and the qp information are merged to generate a first merged table entry; According to the network architecture, the data packets sent by each QP in the QP information are matched layer by layer to the switches through which they flow in each layer, and multiple traffic paths are obtained by integrating them according to the order in which the data packets pass through. The multiple traffic paths are added to the table information corresponding to each qp contained in the first merged table entry to obtain the global traffic information.

2. The method according to claim 1, characterized in that, The step of converting the global traffic information into a traffic matrix representing the global network topology and displaying the traffic matrix includes: The global traffic information is input into the model analysis system for aggregated communication pattern matching, outputting a traffic matrix of the global network topology, and then displaying the traffic matrix on a webpage.

3. The method according to claim 1, characterized in that, The analysis of traffic between end-to-end devices in the network to obtain global traffic information of the network includes: Receive query information input by the user, including: the task ID, the start time and end time of the task; Based on the task ID, the start time, and the end time, a time interval is obtained, and the timestamp corresponding to each qp in the first merged table entry is polled within the time interval. All timestamps within the stated time interval are counted, and the global traffic information of the network is obtained by analyzing the qp information corresponding to all timestamps.

4. The method according to any one of claims 1-3, characterized in that, The step of obtaining the packet traffic forwarding information recorded by the switches between end-to-end devices in the network includes: The network-side device receives the data packet traffic forwarding information sent by the network-side device, which is generated by the network-side device based on the switch records between the end-to-end devices.

5. The method according to any one of claims 1-3, characterized in that, The display of the traffic matrix includes: The traffic matrix is ​​displayed in the form of a traffic matrix diagram, which includes at least one path for end-to-end traffic transmission and the load status of each path, which is represented by three states: idle, normal, and congested.

6. An information processing method for terminal-network convergence, characterized in that, The method is applied to any end device in a network, wherein the any end device includes a first end device, and the method includes: A kernel function is called to establish a queue pair (qp) with the second end device, and the extended Berkeley Packet Filter (eBPF) technology is used to obtain the qp information between the first end device and the second end device. The qp information includes: qp identifier and timestamp of qp establishment. Based on the qp identifier, the timestamp, and the packet sequence number (PSN) counted by the network card counter, calculate the number of packets sent by the qp within a preset time period; The throughput information of the qp is calculated based on the number of packets sent; The throughput information of the QP and the QP information are sent to the data center; The step of calculating the number of packets sent by the qp within a preset time period based on the qp identifier, the timestamp, and the packet sequence number (PSN) counted by the network interface card counter includes: For any qp corresponding to the qp identifier, according to the timestamp, the difference between the packet sequence number PSN counted by the network card counter within the preset time period is calculated as the number of packets sent by the qp within the preset time period; The step of calculating the throughput information of the QP based on the number of packets sent includes: The throughput information of the qp is determined based on the ratio between the number of packets sent and the duration of the preset time period.

7. The method according to claim 6, characterized in that, The process of obtaining the QP information between the first end device and the second end device using the Extended Berkeley Packet Filter (eBPF) technology includes: The kernel function is invoked to obtain the host status table in real time. The host status table includes: the first terminal device number, the IP address of the first terminal device, and the communication port number occupied by qp. The qp information is obtained based on the contents of the host status table and the eBPF technology.

8. The method according to claim 6 or 7, characterized in that, The method further includes: When another preset function is called on the kernel of the first terminal device, the transmission of qp throughput information and the qp information to the data center is stopped.

9. An information processing device integrating terminal network, characterized in that, The device includes: The receiving module is used to receive throughput information at the granularity of queue pairs (qp) sent by multiple end devices in the network, as well as qp information transmitted from end to end devices, wherein the multiple end devices form multiple qp, and each qp consists of two end devices: a sending end and a receiving end. The acquisition module is used to acquire packet traffic forwarding information recorded by switches between end-to-end devices in the network; The processing module is used to analyze the traffic between end-to-end devices in the network based on the throughput information, the qp information and the data packet traffic forwarding information, to obtain the global traffic information of the network; The conversion module is used to convert the global traffic information into a traffic matrix of the global network topology; The display module is used to display the traffic matrix, which includes the traffic forwarding information between the multiple end devices in the network and the load information on the forwarding path; The step of analyzing the traffic between end-to-end devices in the network based on the throughput information, the qp information, and the packet traffic forwarding information to obtain the global traffic information of the network includes: Find the common unique identifier in the throughput information and the qp information; Based on the common unique identifier, the table entries in the throughput information and the qp information are merged to generate a first merged table entry; According to the network architecture, the data packets sent by each QP in the QP information are matched layer by layer to the switches through which they flow in each layer, and multiple traffic paths are obtained by integrating them according to the order in which the data packets pass through. The multiple traffic paths are added to the table information corresponding to each qp contained in the first merged table entry to obtain the global traffic information.

10. An information processing device integrating terminal network, characterized in that, The device includes: The calling module is used to call kernel functions to establish a queue pair (qp) with the second end device, and to use the extended Berkeley Packet Filter (eBPF) technology to obtain qp information between the first end device and the second end device. The qp information includes: qp identifier and timestamp of qp establishment. The calculation module is used to calculate the number of packets sent by the qp within a preset time period based on the qp identifier, the timestamp, and the packet sequence number (PSN) counted by the network card counter. The calculation module is also used to calculate the throughput information of the qp based on the number of packets sent; The sending module is used to send the throughput information of the qp and the qp information to the data center; The step of calculating the number of packets sent by the qp within a preset time period based on the qp identifier, the timestamp, and the packet sequence number (PSN) counted by the network interface card counter includes: For any QP corresponding to the QP identifier, the counter information of the network card is queried at preset time intervals to obtain the packet sequence number (PSN) corresponding to the start time and the packet sequence number (PSN) corresponding to the end time of the preset time interval; the number of packets sent by the QP within the preset time interval is determined based on the difference between the packet sequence number (PSN) corresponding to the end time and the packet sequence number (PSN) corresponding to the start time. The step of calculating the throughput information of the QP based on the number of packets sent includes: The throughput information of the qp is determined based on the ratio between the number of packets sent and the duration of the preset time period.

11. A terminal-network converged system, characterized in that, The system includes at least two end devices and a data center, and the two end devices are connected to each other via at least one switch. The data center is used to execute the end-to-end converged information processing method as described in any one of claims 1 to 5; Each of the terminal devices is configured to perform the information processing method for end-to-end network convergence as described in any one of claims 6 to 8.

12. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory and the processor are connected to each other; The memory stores computer instructions; The processor executes the computer instructions to perform the information processing method for end-to-end convergence as described in any one of claims 1 to 5, or the information processing method for end-to-end convergence as described in any one of claims 6 to 8.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the information processing method for end-to-end convergence as described in any one of claims 1 to 5, or the information processing method for end-to-end convergence as described in any one of claims 6 to 8.

Images