Data stream transmission method, device and equipment and computer readable storage medium

Abstract

The invention relates to the technical field of communication, in particular to a data stream transmission method, device and equipment and a computer readable storage medium. According to the invention, the unique target uplink path is determined for each first network card through the serial number of the connection interface of the first network card on the rack top switch and the number of the outgoing interfaces of the rack top switch, so that each equivalent uplink path between the rack top switch and the convergence switch is only occupied by one first network card. Therefore, the same number of data streams in each path is ensured, and path congestion is avoided.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technologies, and in particular, to a data stream transmission method, apparatus, device, and computer-readable storage medium. Background Art

[0002] During the training process of large models, usually thousands or even tens of thousands of network card devices are required for data transmission, which has extremely high requirements for network communication performance.

[0003] Traditional large model training relies on the ECMP (Equal-Cost Multi-Path) network load balancing technology of switches. Hash calculation is performed based on the five-tuple of data packets, and the forwarding path is determined according to the hash result. In large-scale distributed systems (such as large model training clusters), the number of data streams is huge and complex. Some five-tuples may be hashed to the same path, resulting in the traffic on these paths far exceeding their carrying capacity, while the traffic on other paths is very small, causing congestion on individual paths. Since the cluster communication of large models needs to be completed after all nodes have communicated, the deceleration of individual nodes caused by network congestion will cause all the remaining nodes in the cluster to wait, resulting in a large number of GPU devices being idle, thus causing resource waste.

[0004] Therefore, the path load balancing ability of the current data stream transmission method is still insufficient, which is likely to cause path congestion. Summary of the Invention

[0005] In order to solve the above technical problems, the present disclosure provides a data stream transmission method, apparatus, device, and computer-readable storage medium to ensure that the number of data streams on each path is the same and avoid path congestion.

[0006] In a first aspect, an embodiment of the present disclosure provides a data stream transmission method, including:

[0007] The method is applied to a communication cluster, which includes a plurality of first network cards, a top-of-rack switch, and an aggregation switch. The first network cards are connected to the top-of-rack switch, and there are multiple equivalent upstream paths between the top-of-rack switch and the aggregation switch. The method includes:

[0008] For each first network card, obtain the connection interface number of the first network card at the top-of-rack switch;

[0009] Obtain the number of outgoing interfaces of the top-of-rack switch, and each outgoing interface corresponds to an aggregation switch;

[0010] Determine a target upstream path in the multiple equivalent upstream paths according to the connection interface number and the number of outgoing interfaces, and the upstream path corresponds to the outgoing interface one by one;

[0011] Allocate the target uplink path to the first network card, so that the data stream sent by the first network card is transmitted to the aggregation switch via the target uplink path.

[0012] In some embodiments, the determining the target uplink path from the multiple equivalent uplink paths according to the connection interface number and the number of outgoing interfaces includes:

[0013] Take the remainder of the connection interface number divided by the number of outgoing interfaces to obtain the number of the target outgoing interface;

[0014] Determine the uplink path corresponding to the target outgoing interface as the target uplink path.

[0015] In some embodiments, the allocating the target uplink path to the first network card, so that the data stream sent by the first network card is transmitted to the aggregation switch via the target uplink path includes:

[0016] Control the first network card to send a data stream carrying the target uplink path information to the top-of-rack switch;

[0017] Control the top-of-rack switch to forward the data stream to the aggregation switch via the target uplink path according to the target uplink path information.

[0018] In some embodiments, the communication cluster includes a first server and a second server, the multiple first network cards are located in the first server, the second server includes multiple second network cards, and the multiple second network cards are connected to the top-of-rack switch. The method further includes:

[0019] For each first network card, determine multiple equivalent downlink paths between the aggregation switch and the top-of-rack switch, and the equivalent downlink paths correspond one-to-one to the downlink interfaces on the aggregation switch;

[0020] Determine the target downlink path from the multiple equivalent downlink paths according to the uplink interface number and the number of downlink interfaces of the target uplink path on the aggregation switch;

[0021] Allocate the target downlink path to the first network card.

[0022] In some embodiments, the determining the target downlink path from the multiple equivalent downlink paths according to the uplink interface number and the number of downlink interfaces of the target uplink path on the aggregation switch includes:

[0023] Take the remainder of the uplink interface number divided by the number of downlink interfaces to obtain the number of the target downlink interface;

[0024] Determine that the downlink path corresponding to the target downlink interface is the target downlink path.

[0025] In some embodiments, after allocating the target downlink path to the first network card, the method further includes:

[0026] Control the first network card in the network card pair to carry the target uplink path information and the target downlink path information when sending a data stream to the second network card;

[0027] After the data stream sent by the first network card is transmitted to the aggregation switch via the target uplink path, control the aggregation switch to transmit the data stream to the top-of-rack switch via the target downlink path according to the target downlink path information.

[0028] In a second aspect, an embodiment of the present disclosure provides a data stream transmission device, which is applied to a communication cluster. The communication cluster includes multiple first network cards, a top-of-rack switch, and an aggregation switch. The first network cards are connected to the top-of-rack switch, and there are multiple equivalent uplink paths between the top-of-rack switch and the aggregation switch. The device includes:

[0029] A first acquisition module, configured to, for each first network card, acquire the connection interface number of the first network card at the top-of-rack switch;

[0030] A second acquisition module, configured to acquire the number of outgoing interfaces of the top-of-rack switch, and each of the outgoing interfaces corresponds to an aggregation switch;

[0031] A determination module, configured to determine a target uplink path from the multiple equivalent uplink paths according to the connection interface number and the number of outgoing interfaces, and the uplink path corresponds to the outgoing interface one by one;

[0032] An allocation module, configured to allocate the target uplink path to the first network card, so that the data stream sent by the first network card is transmitted to the aggregation switch via the target uplink path.

[0033] In some embodiments, the communication cluster includes a first server and a second server. The multiple first network cards are located in the first server, and the second server includes multiple second network cards. The multiple second network cards are connected to the top-of-rack switch; the data stream transmission device further includes a third determination module;

[0034] A third determination module, configured to, for each first network card, determine multiple equivalent downlink paths between the aggregation switch and the top-of-rack switch, and the equivalent downlink paths correspond to the downlink interfaces on the aggregation switch one by one;

[0035] The determining module is further configured to determine a target downlink path from the multiple equivalent downlink paths according to the target uplink path, the uplink interface number of the aggregation switch, and the number of downlink interfaces.

[0036] The allocation module is further configured to allocate the target downlink path to the first network card.

[0037] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0038] A memory;

[0039] A processor; and

[0040] A computer program;

[0041] Wherein, the computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0042] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the method as described in the first aspect.

[0043] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instruction, and when the computer program or instruction is executed by a processor, it implements the data stream transmission method as described above.

[0044] The data stream transmission method, apparatus, device, and computer-readable storage medium provided by the embodiments of the present disclosure determine a unique target uplink path for each first network card according to the connection interface number of the first network card at the top-of-rack switch and the number of outgoing interfaces of the top-of-rack switch, so that each equivalent uplink path between the top-of-rack switch and the aggregation switch is only occupied by one first network card, thereby ensuring that the number of data streams on each path is the same and avoiding path congestion. Description of the Drawings

[0045] The drawings here are incorporated into the specification and form a part of this specification, showing the embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure.

[0046] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0047] Figure 1Flowchart of the data stream transmission method provided by the embodiments of the present disclosure;

[0048] Figure 2 Schematic diagram of an application scenario provided by the embodiments of the present disclosure;

[0049] Figure 3 Schematic structural diagram of the data stream transmission device provided by the embodiments of the present disclosure;

[0050] Figure 4 Schematic structural diagram of the electronic device provided by the embodiments of the present disclosure. Detailed implementation manners

[0051] In order to be able to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0052] Many specific details are set forth in the following description in order to fully understand the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all of the embodiments.

[0053] The five-tuple refers to five parameters used to uniquely identify a network connection in a computer network: source IP address, destination IP address, source port number, destination port number, and protocol type. The five-tuple plays a key role in the routing and forwarding process of data packets, and it determines which network interface the data packet should be sent to.

[0054] ECMP (Equal-Cost Multi-Path) is a network load balancing technology that allows data packets to be transmitted through multiple equivalent paths, thereby improving the utilization rate of network bandwidth and the overall network throughput. ECMP determines the transmission path of data packets through hash calculation to ensure that packets of the same data stream can take the same path to maintain the order of data packets.

[0055] Among them, hash calculation is a process of converting an input (or called "key") into a fixed-length output through a specific algorithm, and the output is usually called a "hash value" or "hash code". In computer science, hash functions are widely used in fields such as data retrieval, data storage, and data security.

[0056] The Graphics Processing Unit (GPU) was originally designed to process computer graphics and image rendering, but with the development of technology, the GPU has gradually evolved into a powerful parallel computing processor. In the fields of deep learning and scientific computing, the GPU is widely used because it can handle a large number of parallel computing tasks.

[0057] In a large model training cluster, there are numerous data streams. Due to the non-uniformity of ECMP, after the five-tuples of a large number of data streams are calculated by hashing, they may be randomly forwarded by the switch to the same transmission path. This process is uncontrollable, resulting in too many data streams on some paths and too few on other paths.

[0058] For traditional data center networks, since the amount of data transmitted by each data stream is small and transmissions are not initiated simultaneously, the impact of network congestion on overall performance is controllable. However, in large model training, cluster communication initiates a large number of data transmissions at the same time point. During the model training process, there is a commonly used all2all communication form, that is, each network card in the training cluster communicates with all other network cards in the cluster simultaneously. The characteristic of this communication form is short-term bursts and an extremely large number of data streams (referred to as elephant flows), which will exert great pressure on the forwarding ability of the network switch, resulting in network congestion and a decrease in network transmission speed. This characteristic is different from the previous online service network communication forms, and the network communication forms of traditional services usually appear discretely with fewer data streams (referred to as mouse flows).

[0059] In such sudden large-scale data transmissions, the non-uniformity of ECMP causes some data paths to need to transmit far more traffic than their carrying capacity, triggering the traffic control and speed reduction mechanism of the switch. Since the cluster communication of large models requires all nodes to complete communication before it can end, the speed reduction of individual nodes caused by network congestion will cause all the remaining nodes in the cluster to wait, resulting in a large number of GPU devices being idle, thus causing resource waste.

[0060] In response to the above problems, the embodiments of the present disclosure provide a data stream transmission method, which will be introduced below in combination with specific embodiments.

[0061] Figure 1 It is a flowchart of the data stream transmission method provided by the embodiments of the present disclosure. This method can be applied to Figure 2 the application scenario shown. This application scenario includes a communication cluster, which includes a first server 21, a second server 22, a task scheduler 23, multiple top-of-rack switches (Torswitch1-8), and multiple aggregation switches (AGG switch1-4). Multiple first network cards (Server1 and its NIC1-8) are set on the first server 21, and multiple second network cards (Server2 and its NIC1-8) are set on the second server 22. The data stream starting from the first network card reaches the specified second network card after being routed through the top-of-rack switch and the aggregation switch.

[0062] Among them, the first network card or the second network card can be a device such as a GPU.

[0063] S101. For each first network card, obtain the connection interface number of the first network card on the top-of-rack switch.

[0064] There are multiple interfaces on the top-of-rack switch, and some of these interfaces are used to connect to the first network cards, which are called connection interfaces.

[0065] As Figure 2 shown, each first network card is respectively connected to the top-of-rack switch. Each first network card corresponds to a top-of-rack switch, and a top-of-rack switch can be connected to multiple first network cards.

[0066] That is, for each first network card, the connection interface number of its corresponding top-of-rack switch is unique.

[0067] S102. Obtain the number of outgoing interfaces of the top-of-rack switch. Each of the outgoing interfaces corresponds to a aggregation switch respectively.

[0068] Among the multiple interfaces of the top-of-rack switch, there are also some interfaces used to connect to the aggregation switches, which are called outgoing interfaces. Each outgoing interface is respectively connected to an aggregation switch.

[0069] As Figure 2 shown, taking Torswitch1 as an example, the first network card Server1-NIC1 accesses Torswitch1 through the connection interface, and Tor switch1 is respectively connected to 4 aggregation switches AGGswitch1-4 through different outgoing interfaces.

[0070] The number of outgoing interfaces of the top-of-rack switch is the number of aggregation switches. The communication paths between these outgoing interfaces and the aggregation switches are all equivalent paths, which are called upstream paths. That is, there are multiple equivalent upstream paths between the top-of-rack switch and the aggregation switches.

[0071] S103. According to the connection interface number and the number of outgoing interfaces, determine a target upstream path among the multiple equivalent upstream paths. The upstream paths correspond to the outgoing interfaces one by one.

[0072] Since the connection interface number is unique, a unique target upstream path can be determined according to the connection interface number and the number of outgoing interfaces. This target upstream path is dedicated to transmitting the data stream from the corresponding connection interface, that is, the data stream of the first network card corresponding to this connection interface.

[0073] Among them, the number of connection interfaces is the same as the number of outgoing interfaces, that is, the number of first network cards is the same as the number of equivalent upstream paths between the top-of-rack switch and the aggregation switches.

[0074] S104. Assign the target upstream path to the first network card so that the data stream sent by the first network card is transmitted to the aggregation switch via the target upstream path.

[0075] And so on, determine the target upstream path corresponding to each first network card respectively, and assign it to the corresponding first network card, so that each first network card can exclusively occupy its target upstream path.

[0076] In the embodiment of the present disclosure, for each first network card, obtain the connection interface number of the first network card at the top-of-rack switch; obtain the number of outgoing interfaces of the top-of-rack switch, and each of the outgoing interfaces corresponds to an aggregation switch; according to the connection interface number and the number of outgoing interfaces, determine the target upstream path among the multiple equivalent upstream paths, and the upstream path corresponds to the outgoing interface one by one; assign the target upstream path to the first network card so that the data stream sent by the first network card is transmitted to the aggregation switch via the target upstream path; through the connection interface number of the first network card at the top-of-rack switch and the number of outgoing interfaces of the top-of-rack switch, determine the unique target upstream path for each first network card, so that each equivalent upstream path between the top-of-rack switch and the aggregation switch is exclusively occupied by only one first network card, thereby ensuring that the number of data streams in each path is the same and avoiding path congestion.

[0077] It should be noted that Figure 2 The application scenario shown is only an example of a possible implementation. In actual situations, in addition to the two-layer switch (top-of-rack switch, aggregation switch) architecture shown Figure 2 The data stream transmission method in the embodiment of the present disclosure can also be applied to the scenario of multi-layer switches, so as to determine the upstream path and downstream path between adjacent two-layer switches.

[0078] For example, there is at least one intermediate layer switch between the top-of-rack switch and the aggregation switch. Based on the method described in the above embodiment, determine the upstream path between the top-of-rack switch and the intermediate layer switch, and further determine the upstream path between the intermediate layer switch and the aggregation switch.

[0079] Based on the above embodiment, the step of determining the target upstream path among the multiple equivalent upstream paths according to the connection interface number and the number of outgoing interfaces includes: taking the remainder of the number of outgoing interfaces by the connection interface number to obtain the serial number of the target outgoing interface; determining the upstream path corresponding to the target outgoing interface as the target upstream path.

[0080] See Figure 2, taking the first network card Server1-NIC1 as an example, assuming that the connection interface number of the first network card Server1-NIC1 to TORswitch1 is 1, use this connection interface number to take the remainder of the 4 equivalent uplink paths between TOR switch1 and AGGswitch1-4, and obtain the serial number 1 of the target outgoing interface, that is, select the uplink path between the corresponding TOR switch1-AGGswitch1 as the target uplink path of Server1-NIC1.

[0081] In some embodiments, the step of allocating the target uplink path to the first network card so that the data stream sent by the first network card is transmitted to the aggregation switch via the target uplink path includes: controlling the first network card to send a data stream carrying the target path information to the top-of-rack switch; controlling the top-of-rack switch to forward the data stream to the aggregation switch via the target path according to the target path information.

[0082] The task scheduler 23 assigns a unique target uplink path to each first network card and allocates the target uplink path to the first network card.

[0083] In the communication cluster, the communication path and the packet five-tuple information form a packet five-tuple mapping table, so that the task scheduler 23 can manage the data streams of different network cards and the corresponding paths. The packet five-tuple information contains srcport, which is used to identify the IP address of a certain first network card; after determining the target uplink path corresponding to any first network card, use this target uplink path information to search the packet five-tuple mapping table, and modify the srcport in the packet five-tuple corresponding to the target uplink path in the packet five-tuple mapping table to the IP address of this first network card, specifying that the data stream starting from this first network card carries the target uplink path information. When the data packets in the data stream reach the top-of-rack switch, control the top-of-rack switch to forward the data packets based on the target uplink path information, so that the data stream triggered by the first network card reaches the aggregation switch through the target path.

[0084] See Figure 2, taking the communication path between Server1-NIC1, TOR switch1, and AGG switch1 as an example. Since the uplink path between TOR switch1 and AGG switch1 is specified as the exclusive target uplink path for Server1-NIC1, the srcport in the five-tuple information corresponding to the corresponding path in the packet five-tuple mapping table is modified to the IP address of Server1-NIC1. It is specified that the data stream starting from Server1-NIC1 carries the information of the uplink path between TOR switch1 and AGG switch1 (i.e., the target uplink path information of Server1-NIC1). When the data stream starting from Server1-NIC1 arrives at TOR switch1, TOR switch1 forwards the data stream to AGG switch1 based on the carried target uplink path information.

[0085] In the embodiment of the present disclosure, by specifying that the data stream starting from the first network card carries the corresponding target uplink path information, it is controlled that the data sent by the first network card can all be transmitted through the target uplink path, so that each first network card exclusively occupies the target uplink path, alleviating the non-uniformity of the ECMP hashing algorithm, and without the support of a specific switch. It only needs to specify that the data stream starting from the first network card on the server side carries the target uplink path information, reducing the configuration difficulty of the communication cluster.

[0086] Based on any of the above embodiments, further, each second network card is also respectively connected to the corresponding top-of-rack switch. There is a downlink path between the top-of-rack switch and the aggregation switch, which is used to forward the data stream from the first network card to the top-of-rack switch corresponding to the second network card, and finally reach the specified second network card. Optionally, in the communication cluster, each second network card corresponds to a unique data stream receiving path.

[0087] Among them, the data stream from the first network card also carries the information of the specified second network card. After the data stream from the first network card arrives at the aggregation switch, it is controlled that the aggregation switch determines the top-of-rack switch through which the unique data stream receiving path corresponding to the second network card passes according to the carried information of the specified second network card, and forwards the data stream to the top-of-rack switch, and finally reaches the specified second network card.

[0088] Such as Figure 2As shown in the figure, assume that the data stream starting from the first network card Server1-NIC1 needs to be sent to the second network card Server2-NIC4. After the data stream starting from the first network card Server1-NIC1 reaches the AGG switch1, since the data stream reception path corresponding to the second network card Server2-NIC4 only passes through the Tor switch4, the control aggregation switch forwards the data stream to the Tor switch4 according to the information of the specified second network card Server2-NIC4 carried therein, and finally reaches the second network card Server2-NIC4.

[0089] In the embodiments of the present disclosure, by specifying the target upstream path of the first network card, and further constructing a unique data forwarding path between each pair of the first network card and the second network card based on the connection relationship between the top-of-rack switch and the second network card, it is ensured that the number of data streams flowing through each path is the same and the traffic is within the path carrying capacity range, thereby ensuring that network congestion does not occur in all paths.

[0090] In some other embodiments, the multiple second network cards are connected to the top-of-rack switch, and the method further includes: for each first network card, determining multiple equivalent downstream paths between the aggregation switch and the top-of-rack switch, where the equivalent downstream paths correspond one-to-one to the downstream interfaces on the aggregation switch; determining a target downstream path from the multiple equivalent downstream paths according to the upstream interface number of the target upstream path on the aggregation switch and the number of downstream interfaces; and allocating the target downstream path to the first network card.

[0091] Wherein, determining the target downstream path from the multiple equivalent downstream paths according to the upstream interface number of the target upstream path on the aggregation switch and the number of downstream interfaces includes: taking the remainder of the number of downstream interfaces by the upstream interface number to obtain the number of the target downstream interface; and determining the downstream path corresponding to the target downstream interface as the target downstream path.

[0092] The aggregation switch includes multiple downstream interfaces, respectively corresponding to the equivalent downstream paths between different top-of-rack switches. During the communication process between the first server and the second server, it is necessary to send the data stream starting from any one of the first network cards to the second server. After the data stream reaches the aggregation switch through the target upstream path, the method of taking the remainder is further used to select one of the multiple equivalent downstream paths as the target downstream path.

[0093] Further, after allocating the target downlink path to the first network card, the method further includes: controlling the first network card in the network card pair to carry the target uplink path information and the target downlink path information when sending a data stream to the second network card; after the data stream sent by the first network card is transmitted to the aggregation switch via the target uplink path, controlling the aggregation switch to transmit the data stream to the top-of-rack switch via the target downlink path according to the target downlink path information.

[0094] The data stream sent from the first network card carries the target uplink path information and the target downlink path information. When the data stream sent from the first network card reaches the top-of-rack switch, control the top-of-rack switch to forward the data stream to the corresponding aggregation switch according to the target uplink path information; then control the aggregation switch to forward the data stream to the corresponding top-of-rack switch according to the target downlink path information carried in the data stream; after receiving the data stream from the aggregation switch, the top-of-rack switch further forwards it to the second network card, thereby realizing the data stream transmission between the first server and the second server.

[0095] In the embodiments of the present disclosure, the target downlink path is determined from multiple equivalent downlink paths by using the uplink interface number and the downlink interface quantity and allocated to the first network card, so that each data stream sent from each first network card in the first server corresponds to an exclusive target uplink path and downlink path, making the number of data streams on each data transmission path between the first server and the second server the same, effectively avoiding speed reduction caused by path congestion.

[0096] Figure 3 It is a schematic structural diagram of the data stream transmission device provided by the embodiments of the present disclosure. The data stream transmission device is applied to a communication cluster and may be the task scheduler as described in the above embodiments, or the data stream transmission device may be a component or assembly in the task scheduler; the communication cluster includes multiple first network cards, a top-of-rack switch, and an aggregation switch. The first network cards are connected to the top-of-rack switch, and there are multiple equivalent uplink paths between the top-of-rack switch and the aggregation switch. The data stream transmission device provided by the embodiments of the present disclosure can execute the processing flow provided by the data stream transmission method embodiments, such as Figure 3As shown in the figure, the data stream transmission device 30 includes: a first acquisition module 31, a second acquisition module 32, a determination module 33, and an allocation module 34; the first acquisition module 31 is used to acquire, for each first network card, the connection interface number of the first network card at the top-of-rack switch; the second acquisition module 32 is used to acquire the number of outgoing interfaces of the top-of-rack switch, and each of the outgoing interfaces corresponds to a convergence switch; the determination module 33 is used to determine a target uplink path among the multiple equivalent uplink paths according to the connection interface number and the number of outgoing interfaces, and the uplink path corresponds to the outgoing interface one by one; the allocation module 34 is used to allocate the target uplink path to the first network card, so that the data stream sent by the first network card is transmitted to the convergence switch via the target uplink path.

[0097] Optionally, the determination module 33 includes a remainder unit 331 and a first determination unit 332; the remainder unit 331 is used to obtain the number of the target outgoing interface by taking the remainder of the connection interface number divided by the number of outgoing interfaces; the first determination unit 332 is used to determine the uplink path corresponding to the target outgoing interface as the target uplink path.

[0098] Optionally, the allocation module 34 includes a first control unit 341 and a second control unit 342; the first control unit 341 is used to control the first network card to send a data stream carrying the target uplink path information to the top-of-rack switch; the second control unit 342 is used to control the top-of-rack switch to forward the data stream to the convergence switch via the target uplink path according to the target uplink path information.

[0099] Optionally, the communication cluster includes a first server and a second server, the multiple first network cards are located in the first server, the second server includes multiple second network cards, and the multiple second network cards are connected to the top-of-rack switch; the data stream transmission device 30 further includes a third determination module 35, which is used to determine, for each first network card, multiple equivalent downlink paths between the convergence switch and the top-of-rack switch, and the equivalent downlink paths correspond to the downlink interfaces on the convergence switch one by one; the determination module 33 is further used to determine a target downlink path among the multiple equivalent downlink paths according to the uplink interface number and the number of downlink interfaces of the convergence switch of the target uplink path; the allocation module 34 is further used to allocate the target downlink path to the first network card.

[0100] Optionally, the determination module 33 includes a remainder unit 331 and a second determination unit 333; the remainder unit 331 is used to obtain the number of the target downlink interface by taking the remainder of the uplink interface number divided by the number of downlink interfaces; the second determination unit 333 is used to determine the downlink path corresponding to the target downlink interface as the target downlink path.

[0101] Optionally, the allocation module 34 includes a first control unit 341 and a second control unit 342. The first control unit 341 is configured to control the first network card in the network card pair to carry the target upstream path information and the target downstream path information when sending a data stream to the second network card. The second control unit 342 is configured to control the aggregation switch to transmit the data stream through the target downstream path to the top-of-rack switch according to the target downstream path information after the data stream sent by the first network card is transmitted to the aggregation switch through the target upstream path.

[0102] Figure 3 The data stream transmission device of the illustrated embodiment can be used to execute the technical solutions of the above method embodiments. The implementation principles and technical effects are similar and will not be elaborated here.

[0103] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. The electronic device can be the task scheduler as described in the above embodiment. The electronic device provided by the embodiment of the present disclosure can execute the processing flow provided by the data stream transmission method embodiment, as Figure 4 shown, the electronic device 40 includes: a memory 41, a processor 42, a computer program, and a communication interface 43. Among them, the computer program is stored in the memory 41 and is configured to be executed by the processor 42 to perform the data stream transmission method as described above.

[0104] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the data stream transmission method described in the above embodiment.

[0105] In addition, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instruction, and when the computer program or instruction is executed by a processor, it implements the data stream transmission method as described above.

[0106] It should be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0107] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data stream transmission method, characterized in that: The method is applied in a communication cluster, the communication cluster includes a plurality of first network cards, a rack top switch and an aggregation switch, the first network card is connected to the rack top switch, and a plurality of equal-cost uplink paths are included between the rack top switch and the aggregation switch, the method includes: For each first network card, obtain the serial number of the connection interface of the first network card on the switch at the top of the rack; Obtain the number of outbound interfaces of the switch at the top of the rack, each of the outbound interfaces corresponding to an aggregation switch; Determine a target uplink path among the multiple equal-cost uplink paths according to the connection interface sequence number and the number of the outbound interfaces, wherein the uplink path corresponds to the outbound interface in a one-to-one manner; The target uplink path is allocated to the first network card, so that the data flow sent by the first network card is transmitted to the aggregation switch via the target uplink path.

2. The method according to claim 1, characterized in that: The determining, according to the connection interface sequence number and the number of the outgoing interfaces, a target uplink path among the multiple equal-cost uplink paths comprises: Using the connection interface sequence number to calculate the modulus of the number of outgoing interfaces, a sequence number of a target outgoing interface is obtained; An uplink path corresponding to the target outgoing interface is determined as the target uplink path.

3. The method according to claim 1, characterized in that The allocating the target uplink path to the first network card so that the data stream sent by the first network card is transmitted to the aggregation switch via the target uplink path includes: Controlling the first network card to send a data stream carrying the target uplink path information to the rack top switch; The rack top switch is controlled to forward the data flow to the aggregation switch via the target uplink path according to the target uplink path information.

4. The method according to claim 1, characterized in that: The communication cluster includes a first server and a second server, the plurality of first network cards are located in the first server, the second server includes a plurality of second network cards, and the plurality of second network cards are connected to the rack top switch, and the method further includes: For each first network card, determine a plurality of equal-cost downstream paths between the aggregation switch and the top-of-rack switch, wherein the equal-cost downstream paths correspond one-to-one to downstream interfaces on the aggregation switch; Determine a target downstream path from among the multiple equal-cost downstream paths according to the upstream interface sequence number and the number of downstream interfaces of the target upstream path on the aggregation switch; The target downlink path is allocated to the first network card.

5. The method according to claim 4, characterized in that The step of determining the target downlink path from the multiple equal-cost downlink paths according to the uplink interface sequence number and the number of downlink interfaces of the target uplink path on the aggregation switch includes: Using the sequence number of the upstream interface to obtain the sequence number of the target downstream interface, the modulus of the number of the downstream interfaces is obtained; A downlink path corresponding to the target downlink interface is determined as the target downlink path.

6. The method according to claim 4, characterized in that After allocating the target downlink path to the first network card, the method further includes: Controlling the first network card in the network card pair to carry the target uplink path information and the target downlink path information when sending a data stream to the second network card; After the data stream sent by the first network card is transmitted to the aggregation switch via the target uplink path, the aggregation switch is controlled to transmit the data stream to the rack top switch via the target downlink path according to the target downlink path information.

7. A data stream transmission device, characterized in that: Applied in a communication cluster, the communication cluster includes a plurality of first network cards, a rack top switch and an aggregation switch, the first network card is connected to the rack top switch, and a plurality of equal-cost uplink paths are provided between the rack top switch and the aggregation switch, the device includes: A first acquisition module, configured to acquire, for each first network card, a serial number of a connection interface of the first network card on the switch at the top of the rack; A second acquisition module is used to acquire the number of outbound interfaces of the switch on the top of the rack, each of which corresponds to an aggregation switch; A determination module, configured to determine a target uplink path among the multiple equivalent uplink paths according to the connection interface sequence number and the number of the outbound interfaces, wherein the uplink path corresponds to the outbound interface in a one-to-one manner; The allocation module is used to allocate the target uplink path to the first network card, so that the data flow sent by the first network card is transmitted to the aggregation switch via the target uplink path.

8. The device according to claim 7, characterized in that The communication cluster includes a first server and a second server, the plurality of first network cards are located in the first server, the second server includes a plurality of second network cards, and the plurality of second network cards are connected to the rack top switch; the data stream transmission device also includes a third determination module; A third determination module is used to determine, for each first network card, a plurality of equivalent downstream paths between the aggregation switch and the rack top switch, wherein the equivalent downstream paths correspond one-to-one to the downstream interfaces on the aggregation switch; The determination module is further used to determine the target downlink path among the multiple equal-cost downlink paths according to the uplink interface sequence number and the number of downlink interfaces of the target uplink path on the aggregation switch; The allocation module is further used to allocate the target downlink path to the first network card.

9. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images