Multi-tenant oriented multi-path reliable transmission control method and system

Abstract

This invention relates to the field of network communication technology, specifically proposing a multi-tenant multi-path reliable transmission control method and system. The method first achieves tenant isolation in the control plane by generating and distributing a tenant policy table containing an authorized path pool for each tenant. Then, it initializes the transmission status of each tenant on each path according to the policy table, providing a data foundation for fine-grained scheduling. Next, in response to a data packet to be sent, it queries the tenant path mapping table using "tenant identifier + path selection identifier" to determine the tenant's dedicated segmented route IPv6 target path, achieving tenant-level isolation for data plane routing. Finally, it sends the data packet through the target path. Thus, by introducing a tenant-aware mechanism in both the control plane and data plane, each tenant can perform efficient and stable data transmission on an independent, controlled network resource slice.

Description

Technical Field

[0001] This invention belongs to the field of network communication technology, specifically relating to a multi-tenant multi-path reliable transmission control method and system. Background Technology

[0002] As the scale of large-scale model training continues to expand, a single training task typically requires synchronous computation and gradient exchange across a large number of GPUs (Graphics Processing Units) and nodes, placing high demands on network transmission throughput, tail latency, and packet loss recovery capabilities. To address this, the industry has proposed the MRC (Multipath Reliable Connection) mechanism, which improves the reliability and fault recovery capabilities of training networks by distributing data packets across multiple paths and combining SRv6 source routing and selective retransmission.

[0003] However, in multi-tenant large model training scenarios, multiple tenants share network interface card ports, switch links, queue buffers, and path resources. Existing MRC solutions are mainly designed for single large-scale training clusters, using multi-path resources as a unified resource pool, and lack a tenant-oriented mechanism for allocating ports, network planes, and path pools. Training tasks of different tenants simultaneously contend for the same path resources, causing performance to affect each other among tenants, which cannot meet the path resource isolation requirements in multi-tenant shared networks.

[0004] In other words, existing technologies struggle to effectively isolate and allocate multi-path resources of different tenants in a multi-tenant shared training network, leading to performance interference between training tasks of different tenants and reducing the overall training efficiency and reliability of the network. Summary of the Invention

[0005] This invention aims to at least address the technical problem in related technologies of effectively isolating and allocating multi-path resources of different tenants on demand in multi-tenant shared training networks. Therefore, the purpose of this invention is to propose a multi-tenant multi-path reliable transmission control method, system, electronic device, and storage medium.

[0006] In a first aspect, the present invention proposes a multi-tenant, multi-path reliable transmission control method, comprising the following steps: generating and distributing a tenant policy table for each tenant, wherein the tenant policy table includes a tenant identifier, a tenant level, and an authorized path pool, the authorized path pool containing multiple paths, and the authorized path pools associated with the tenant identifiers of different tenants being isolated from each other; initializing the transmission state of each tenant on each path in the corresponding authorized path pool according to the tenant policy table; responding to a data packet to be sent, querying a preset tenant path mapping table according to the tenant identifier and path selection identifier of the tenant to which the data packet belongs, and determining the corresponding segmented routing IPv6 path as the target path from the authorized path pool associated with the tenant identifier, wherein the tenant path mapping table is used to record the segmented routing IPv6 paths corresponding to combinations of different tenant identifiers and different path selection identifiers, and the same path selection identifier corresponds to different forwarding paths in different network planes under different tenants; and sending the data packet through the target path.

[0007] According to an embodiment of the present invention, the multi-tenant multi-path reliable transmission control method first achieves tenant isolation in the control plane by generating and distributing a tenant policy table containing an authorized path pool for each tenant; then, it initializes the transmission status of each tenant on each path according to the policy table, providing a data foundation for fine-grained scheduling; next, in response to the data packet to be sent, it uses "tenant identifier + path selection identifier" to query the tenant path mapping table to determine the tenant's exclusive segmented route IPv6 target path, achieving tenant-level isolation for data plane routing; finally, it sends the data packet through the target path; thus, by introducing a tenant-aware mechanism in both the control plane and the data plane, it fundamentally solves the problem of disordered contention for path resources and performance interference in multi-tenant shared networks, enabling each tenant to perform efficient and stable data transmission on an independent, controlled network resource slice.

[0008] In addition, the multi-tenant-oriented multi-path reliable transmission control method according to embodiments of the present invention may also have the following additional technical features: Furthermore, before generating and distributing the tenant policy table for each tenant, the multi-tenant multi-path reliable transmission control method further includes: tenant resource slicing and path pool construction, comprising the following steps: obtaining the tenant level and / or tenant weight of each tenant; allocating port resources, network plane resources, and path pool resources to each tenant according to the tenant level and / or the tenant weight, wherein the allocation ratio of the port resources, the allocation ratio of the network plane resources, and the allocation ratio of the path pool resources are positively correlated with the tenant weight, and under the same tenant weight, the tenant with the higher tenant level is given priority in resource allocation; based on the allocated port resources, the network plane resources, and the path pool resources, constructing the authorized path pool for each tenant, and dividing a portion of the paths in the authorized path pool into an active path pool, dividing another portion of the paths into a backup path pool, and using the unallocated reserved resources as a global backup resource pool.

[0009] Furthermore, the transmission status includes the number of bytes sent, the number of acknowledged bytes, the number of unacknowledged bytes, the basic round-trip time, the effective path rate, the path weight, and the path health score. During the transmission of the data packet through the target path, the multi-tenant-oriented multi-path reliable transmission control method further includes: path state maintenance and congestion awareness, comprising the following steps: calculating the basic bandwidth-delay product based on the effective path rate and the basic round-trip time; determining the amount of data in transit that can naturally exist on the path without queuing based on the basic bandwidth-delay product; calculating the in-transit excess based on the number of unacknowledged bytes and the basic bandwidth-delay product; calculating a congestion delay estimate based on the in-transit excess and the effective path rate; and determining the current congestion queuing delay level based on the congestion delay estimate.

[0010] Furthermore, after calculating the congestion delay estimate based on the in-transit excess and the effective path rate, the multi-tenant-oriented multi-path reliable transmission control method further includes: queue-aware path weight adjustment, comprising the following steps: obtaining the path's congestion delay estimate, explicit congestion notification count, negative acknowledgment count, selective acknowledgment gap count, historical success rate, and path health score; when the congestion delay estimate is lower than a preset congestion delay threshold, and the explicit congestion notification count, the negative acknowledgment count, and the selective acknowledgment gap count are all zero, increasing the path weight of the path to prioritize new data packets. The path; when the congestion delay estimate increases or the explicit congestion notification count increases, the path weight of the path is reduced so that new data packets use the path less, but the path is not removed from the active path pool; when the negative acknowledgment count is greater than zero, selective retransmission is performed on the data packets corresponding to the negative acknowledgment, and the path weight of the path is reduced within a preset time window; when the selective acknowledgment gap count on the same path reaches a preset gap count threshold, the path health score of the path is reduced; when the path health score is lower than a preset health threshold, the path is removed from the active path pool.

[0011] Furthermore, after calculating the congestion delay estimate based on the in-transit excess and the effective path rate, the multi-tenant-oriented multi-path reliable transmission control method further includes: queue-aware dynamic timeout threshold calculation and timeout processing, comprising the following steps: calculating a dynamic timeout threshold based on the basic round-trip timeout, the congestion delay estimate, delay jitter, and tenant level; when a data packet sent through the path exceeds the dynamic timeout threshold and still has not received an acknowledgment, triggering timeout event processing, including: obtaining the congestion delay estimate of the path and the tenant level of the tenant; when the congestion delay estimate is higher than the congestion delay threshold, determining that the current timeout is caused by queuing delay, extending the waiting time of the data packet, reducing the path weight of the path, and reducing the selection of the path for new data packets; when the congestion delay estimate is not higher than the congestion delay threshold, determining that the current timeout is caused by actual packet loss or path anomaly, performing selective retransmission on the data packet, and reducing the path health score of the path.

[0012] Furthermore, after sending the data packet through the target path, the multi-tenant-oriented multi-path reliable transmission control method further includes: hierarchical packet loss recovery, comprising the following steps: when the negative acknowledgment count is greater than zero, determining the data packet to be recovered based on the negative acknowledgment, selectively retransmitting the data packet, reducing the path weight of the original transmission path of the data packet, and clearing the negative acknowledgment count to zero after the retransmission is completed, keeping the path in the active path pool, and not removing the path directly due to a single negative acknowledgment; when the selective acknowledgment gap count is greater than zero, starting a short out-of-order waiting window; after the short out-of-order waiting window ends, if the selective acknowledgment gap has been filled by a subsequent acknowledgment, then clearing the selective acknowledgment gap count to zero; if the selective acknowledgment gap still exists, then selectively retransmitting the missing data packet; When the cumulative count of selective acknowledgment gaps on the same path reaches a preset gap count threshold, the path health score of the path is reduced. When the path health score is lower than a preset health threshold, the path is removed from the active path pool, and a path is selected from the backup path pool to be added to the active path pool. When a data packet exceeds the dynamic timeout threshold and is still not acknowledged, and the count of negative acknowledgment is zero, and the count of selective acknowledgment gaps is zero, the congestion delay estimate of the path and the tenant's tenant level are read. When the congestion delay estimate is higher than the congestion delay threshold, it is determined that queuing delay has slowed down the acknowledgment, the waiting time is extended, and the path weight of the path is reduced. When the congestion delay estimate is not higher than the congestion delay threshold, it is determined that there is a real packet loss or path anomaly, selective retransmission is performed, and the path health score of the path is reduced.

[0013] Furthermore, after removing the path from the active path pool, the multi-tenant-oriented multi-path reliable transmission control method further includes: dynamic replacement of the active path pool and the backup path pool, including the following steps: selecting a backup path from the tenant's backup path pool and adding it to the active path pool to maintain the number of paths in the active path pool; performing background probing on the abnormal path that has been removed from the active path pool; when the background probing successfully reaches a preset recovery threshold consecutively, adding the abnormal path back to the backup path pool and waiting for subsequent scheduling; when the background probing does not reach the recovery threshold, keeping the abnormal path in an inactive state.

[0014] Furthermore, during the transmission of the data packet through the target path, the multi-tenant-oriented multi-path reliable transmission control method further includes: tenant-level retransmission budget control, comprising the following steps: setting a retransmission budget for each tenant, wherein the retransmission budget represents the retransmission traffic allowance allowed for the tenant within a preset rolling time window, and the retransmission traffic allowance is expressed in any one of the following measurement methods: retransmission bytes, retransmission packets, retransmission requests, retransmission bandwidth percentage, or retransmission tokens; including both normal transmission traffic and retransmission traffic in the tenant's resource consumption, and updating the actual retransmission traffic count of the corresponding tenant each time a retransmission occurs. When the actual retransmission traffic count of any tenant reaches the retransmission budget, a budget control operation is performed, including: limiting new retransmissions triggered by ordinary timeout events for the tenant; retaining necessary retransmissions triggered by negative acknowledgment events and continuous selective acknowledgment gap events for the tenant; increasing the dynamic timeout threshold for subsequent ordinary timeout events on each path of the tenant; reducing the path weight of the tenant on congested paths to reduce the selection of new data packets on congested paths; and when there are available backup paths for the tenant, removing abnormal paths from the active path pool and selecting paths from the backup path pool to add to the active path pool.

[0015] Furthermore, after sending the data packet through the target path, the multi-tenant-oriented multi-path reliable transmission control method further includes tenant connection release and resource reclamation, including the following steps: stopping the transmission of new data packets for all paths in the authorized path pool corresponding to the tenant; performing cleanup retransmission or discard processing on the currently unacknowledged data packets of the tenant according to the hierarchical retransmission strategy; and after confirming that the transmission status of the tenant has stabilized, reclaiming the tenant's port resources, network plane resources, path pool resources and retransmission budget, and re-incorporating the reclaimed resources into the allocable resource set.

[0016] Secondly, based on the same inventive concept, this invention also proposes a multi-tenant, multi-path reliable transmission control system, comprising: a tenant control module, used to generate and distribute a tenant policy table for each tenant, wherein the tenant policy table includes a tenant identifier, a tenant level, and an authorized path pool, the authorized path pool containing multiple network paths, and the authorized path pools associated with the tenant identifiers of different tenants are isolated from each other; and a segmented routing IPv6 path pool module, used to store a preset tenant path mapping table, wherein the tenant path mapping table records the segmented routing IPv6 paths corresponding to combinations of different tenant identifiers and different path selection identifiers, and the same path selection identifier is not... Different forwarding paths corresponding to different network planes under the same tenant; a queue-aware scheduling module, used to initialize the transmission status of each tenant on each network path in the corresponding authorized path pool according to the tenant policy table, responding to the data packet to be sent, querying the tenant path mapping table according to the tenant identifier and path selection identifier of the tenant to which the data packet belongs, determining the corresponding segmented route IPv6 path from the authorized path pool associated with the tenant identifier as the target path, and sending the data packet through the target path; a reliability recovery module, used to perform hierarchical packet loss recovery; a multi-plane physical network module, used to provide physical transmission resources for multiple ports and multiple network planes.

[0017] The multi-tenant-oriented multi-path reliable transmission control system provided in the second aspect has the same technical features as the multi-tenant-oriented multi-path reliable transmission control method provided in the first aspect, and therefore also has similar technical effects as the first aspect, which will not be described in detail here.

[0018] Thirdly, based on the same inventive concept, the present invention also proposes an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor, when executing the computer program stored in the memory, implements the steps of the multi-tenant-oriented multi-path reliable transmission control method as described in the above embodiments of the present invention.

[0019] The electronic equipment provided in the third aspect has the same technical features as the multi-tenant multi-path reliable transmission control method provided in the first aspect, and therefore also has similar technical effects as the first aspect, which will not be described in detail here.

[0020] Fourthly, based on the same inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multi-tenant, multi-path reliable transmission control method as described in the above embodiments of the present invention.

[0021] The computer-readable storage medium provided in the fourth aspect has the same technical features as the multi-tenant-oriented multi-path reliable transmission control method provided in the first aspect, and therefore also has similar technical effects as the first aspect, which will not be described in detail here.

[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The above and additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a multi-tenant-oriented multi-path reliable transmission control method according to an embodiment of the present invention; Figure 2 This is an overall flowchart of a multi-tenant, multi-path reliable transmission control method according to a specific embodiment of the present invention. Figure 3 This is a structural block diagram of a multi-tenant, multi-path reliable transmission control system according to an embodiment of the present invention; Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present invention.

[0024] Figure label: 100 - Multi-tenant-oriented multi-path reliable transmission control system; 110 - Tenant control module; 120 - Segmented routing IPv6 path pool module; 130 - Queue-aware scheduling module; 140 - Reliability recovery module; 150 - Multi-plane physical network module. Detailed Implementation

[0025] The embodiments of the present invention are described in detail below, and the embodiments described with reference to the accompanying drawings are exemplary.

[0026] To address the problem in related technologies of effectively isolating and allocating multi-path resources of different tenants on demand in multi-tenant shared training networks, this invention provides a multi-tenant multi-path reliable transmission control method, system, electronic device, and storage medium, as described below. Figures 1-4 This invention describes a multi-tenant, multi-path reliable transmission control method, system, electronic device, and storage medium according to embodiments of the present invention.

[0027] Figure 1 This is a flowchart of a multi-tenant, multi-path reliable transmission control method according to an embodiment of the present invention. Figure 1 As shown, a multi-tenant-oriented multi-path reliable transmission control method according to an embodiment of the present invention includes the following steps: Step S1: Generate and distribute a tenant policy table for each tenant. The tenant policy table includes a tenant ID, a tenant level, and an authorized path pool. The authorized path pool contains multiple paths, and the authorized path pools associated with the tenant IDs of different tenants are isolated from each other.

[0028] In a specific embodiment, after receiving a tenant's access request or obtaining tenant description information from the scheduling system, the tenant controller generates a unique tenant policy table for each tenant based on a pre-configured scheduling policy. The tenant policy table includes a Tenant ID (Tenant Identifier), a tenant priority, and an authorized path pool. The authorized path pool contains multiple network paths that the tenant is allowed to use, and the authorized path pools associated with the tenant identifiers of different tenants are logically isolated from each other; that is, paths in a tenant's authorized path pool will not be allocated to other tenants by default.

[0029] Specifically, the embodiments of the present invention first generate and distribute a tenant policy table containing tenant identifier, tenant level and authorized path pool for each tenant, thereby realizing tenant awareness in the control plane; by allocating logically isolated authorized path pools to different tenants, the training traffic of different tenants is fundamentally avoided from competing for the same path resources; thus effectively solving the resource isolation and performance interference problems in multi-tenant shared networks, laying the foundation for subsequent tenant-based scheduling and differentiated services.

[0030] Step S2: Initialize the transmission status of each tenant on each path in the corresponding authorized path pool according to the tenant policy table.

[0031] In a specific embodiment, after receiving the tenant policy table, the sending end (such as the network interface card or smart network interface card of a computing node) initializes a series of protocol state variables (transmission states) for that tenant on each path in its authorized path pool. Specifically, these state variables include, but are not limited to: transmission sequence number ( send seq ), Confirm Serial Number ( ack seq ), Number of bytes sent ( sent bytes ), Number of confirmed bytes ( acked bytes ), number of unconfirmed bytes ( inflightb ytes Basic round-trip time ( base RTT ), path effective rate ( C ), path weight ( path weight ) and path health score ( healthscore ).

[0032] Specifically, the embodiments of the present invention then initialize fine-grained transmission states for each path of each tenant, transforming global, undifferentiated network transmission into stateful and controllable transmission oriented towards tenants and paths; thereby providing a real-time and accurate data foundation for subsequent implementation of precise congestion awareness, queue-aware scheduling, and hierarchical retransmission, enabling the transmission behavior of each tenant to be independently monitored and adjusted.

[0033] Step S3: In response to the data packet to be sent, based on the tenant identifier and path selection identifier of the tenant to which the data packet belongs, query the preset tenant path mapping table, and determine the corresponding segmented route IPv6 path as the target path from the authorized path pool associated with the tenant identifier. The tenant path mapping table is used to record the segmented route IPv6 paths corresponding to different combinations of tenant identifiers and different path selection identifiers. The same path selection identifier corresponds to different forwarding paths in different network planes under different tenants.

[0034] In a specific embodiment, when the sending end needs to send a data packet, it first determines the tenant to which the data packet belongs; then, the sending end selects a path selection identifier (EV) (Entropy Value) according to a scheduling strategy (such as round-robin, random, or probability selection based on path weight); next, the sending end queries a preset tenant path mapping table using "tenant identifier + path selection identifier" as an index. This mapping table expands the traditional global mapping of "path selection identifier - path" into a tenant-aware mapping of "(tenant identifier, path selection identifier) ​​- SRv6 (Segment Routing over IPv6 (Internet Protocol Version 6) path"; the query result determines the SRv6 path that the data packet should use as the target path.

[0035] The key is that the same path selection identifier corresponds to different forwarding paths on different network planes under different tenants; for example, tenant A's EV-001 is mapped to a path on network plane 0, while tenant B's EV-001 is mapped to a completely different path on network plane 4.

[0036] Specifically, this embodiment of the invention then introduces a tenant-aware path mapping mechanism to strongly bind data plane routing behavior to tenant identity; the sending end must query the path based on the combination of tenant identifier and path selection identifier, ensuring that data packets can only be forwarded within the path pool authorized to their respective tenants; compared with the globally unified EV mapping in the prior art, this invention achieves tenant-level strong isolation for data plane path selection, effectively preventing conflicts and interference between tenants caused by misuse or preemption of paths, and helping to enhance the controllability and security of multi-tenant networks.

[0037] Step S4: Send data packets through the target path.

[0038] In a specific embodiment, the sending end encodes the queried SRv6 path information into the IPv6 extension header of the data packet, and then sends the data packet out; the data packet will be forwarded strictly in the order of the nodes specified by the SRv6 path.

[0039] Specifically, the embodiments of the present invention utilize SRv6 source routing technology, enabling the sending end to precisely control the forwarding path of data packets in the network without relying on the complex routing protocols of intermediate switches; thereby enabling the isolation policy of the tenant path pool to be precisely executed on the physical network, ensuring that data packets are always transmitted on the dedicated path resources allocated to their tenants.

[0040] Therefore, the multi-path reliable transmission control method for multi-tenants according to embodiments of the present invention first achieves tenant isolation in the control plane by generating and distributing a tenant policy table containing an authorized path pool for each tenant; then, it initializes the transmission status of each tenant on each path according to the policy table, providing a data foundation for fine-grained scheduling; next, in response to the data packet to be sent, it uses "tenant identifier + path selection identifier" to query the tenant path mapping table to determine the tenant's exclusive segmented route IPv6 target path, achieving tenant-level isolation for data plane routing; finally, it sends the data packet through the target path; thus, by introducing a tenant-aware mechanism in both the control plane and the data plane, it fundamentally solves the problem of disordered competition for path resources and performance interference in multi-tenant shared networks, enabling each tenant to perform efficient and stable data transmission on an independent, controlled network resource slice.

[0041] In one embodiment of the present invention, before generating and issuing a tenant policy table for each tenant, the multi-tenant multi-path reliable transmission control method further includes: tenant resource slicing and path pool construction, comprising the following steps: obtaining the tenant level and / or tenant weight of each tenant; allocating port resources, network plane resources and path pool resources to each tenant according to the tenant level and / or tenant weight, wherein the allocation ratio of port resources, the allocation ratio of network plane resources and the allocation ratio of path pool resources are positively correlated with the tenant weight, and under the same tenant weight, tenants with higher tenant levels are given priority in resource allocation; based on the allocated port resources, network plane resources and path pool resources, constructing an authorized path pool for each tenant, and dividing a portion of the paths in the authorized path pool into an active path pool and another portion of the paths into a backup path pool, and using the unallocated reserved resources as a global backup resource pool.

[0042] In a specific embodiment, the tenant controller first obtains the tenant level and / or tenant weight of each tenant; then, based on these parameters, it allocates port resources (e.g., dividing an 800G network card into multiple 100G logical ports), network plane resources (e.g., 8 parallel network planes), and path pool resources (including the size of the active path pool and the backup path pool) to each tenant; the allocation ratio is positively correlated with the tenant weight, and under the same weight, tenants with higher levels are allocated priority; finally, based on the allocated resources, an authorized path pool is built for each tenant and divided into active and backup parts, with unallocated reserved resources serving as a global backup resource pool.

[0043] Specifically, this embodiment of the invention uses a resource slicing mechanism based on tenant level and weight to pool physical network resources (ports, planes, paths) into logically isolated tenant resource slices. This enables high-priority tenants to obtain more path resources and larger bandwidth quotas, thereby ensuring the quality of service for their critical training tasks. At the same time, by independently dividing active and backup path pools for each tenant and setting up a global backup pool, a resource foundation is provided for subsequent rapid fault recovery and elastic scaling, which helps to further improve the manageability and stability of multi-tenant networks.

[0044] In one embodiment of the present invention, the transmission state includes the number of bytes sent, the number of acknowledged bytes, the number of unacknowledged bytes, the basic round-trip time, the effective path rate, the path weight, and the path health score. During the transmission of data packets through the target path, the multi-tenant multi-path reliable transmission control method further includes: path state maintenance and congestion awareness, comprising the following steps: calculating the basic bandwidth-delay product based on the effective path rate and the basic round-trip time; determining the amount of data in transit that can naturally exist on the path without queuing based on the basic bandwidth-delay product; calculating the in-transit excess based on the number of unacknowledged bytes and the basic bandwidth-delay product; calculating a congestion delay estimate based on the in-transit excess and the effective path rate; and determining the current congestion queuing delay level based on the congestion delay estimate.

[0045] In a specific embodiment, the sending end continuously monitors the transmission status on each path; the sending end determines the effective rate of the path ( ) and basic round-trip time ( ) Calculate the base bandwidth delay product ( This value represents the amount of data that should be in transit on the path without queuing; then, based on the real-time number of unacknowledged bytes ( The product of bandwidth and delay is used to calculate the excess amount in transit. Finally, the congestion delay estimate is calculated based on the excess amount in transit and the effective path rate. ).

[0046] Specifically, this invention proposes a congestion awareness mechanism based on self-calculation of the sender's protocol state. Unlike existing technologies that rely on switch feedback (such as ECN (Explicit Congestion Notification)) or passively measuring RTT (Round-Trip Time), this invention proactively calculates "in-transit excess" and "congestion delay estimate" by maintaining the sending and acknowledgment status of each tenant and each path at the sender end. These two parameters can reflect the queuing congestion level on the current path in real time and quantitatively, without being affected by the traffic of other tenants (because the state is maintained by tenant and path). This provides a more direct, accurate, and timely congestion signal for subsequent path scheduling and timeout control than traditional methods, and is a key technical foundation for distinguishing between queuing delay and actual packet loss.

[0047] In one embodiment of the present invention, after calculating the congestion delay estimate based on the in-transit excess and the effective path rate, the multi-tenant multi-path reliable transmission control method further includes: queue-aware path weight adjustment, comprising the following steps: obtaining the path's congestion delay estimate, explicit congestion notification count, negative acknowledgment count, selective acknowledgment gap count, historical success rate, and path health score; when the congestion delay estimate is lower than a preset congestion delay threshold, and the explicit congestion notification count, negative acknowledgment count, and selective acknowledgment gap count are all zero, increasing the path weight of the path so that new data packets preferentially select the path; when the congestion delay estimate increases or the explicit congestion notification count increases, decreasing the path weight of the path so that new data packets use the path less frequently, but without removing the path from the active path pool; when the negative acknowledgment count is greater than zero, performing selective retransmission on the data packets corresponding to the negative acknowledgment, and decreasing the path weight of the path within a preset time window; when the selective acknowledgment gap count on the same path reaches a preset gap count threshold, decreasing the path health score of the path; when the path health score is lower than a preset health threshold, removing the path from the active path pool.

[0048] In a specific embodiment, the sending end obtains the path's congestion delay estimate, explicit congestion notification (ECN) count, NACK (Negative Acknowledgement) count, SACK (Selective Acknowledgement) gap count, historical success rate, and path health score. When the congestion delay estimate is below a threshold and there is no negative feedback, the path weight of the path is increased, making it more likely to be selected. When the congestion delay increases or the ECN count increases, the path weight of the path is decreased, reducing its usage, but not immediately removing it. When a NACK is received, selective retransmission is performed on the corresponding data packet, and the path weight is temporarily reduced. When the SACK gap count of the same path reaches a threshold, its health score is reduced. If the health score is below the health threshold, the path is removed from the active path pool.

[0049] Specifically, this embodiment of the invention achieves differentiated processing of instantaneous congestion and persistent anomalies through a queue-aware path weight adjustment mechanism. Path weight is a short-term scheduling parameter used for load balancing, which responds quickly to instantaneous queue deepening or NACK events, reducing the injection of new traffic to avoid exacerbating congestion. Path health score, on the other hand, is a long-term indicator used to evaluate path quality, and it is only affected by serious problems such as persistent SACK gaps. This two-layer design of "weight and health separation" can effectively prevent a healthy path from being prematurely and incorrectly removed from the active pool due to brief network fluctuations (such as sudden traffic from other tenants), thereby ensuring rapid response to congestion while maintaining the overall stability of the path pool.

[0050] In one embodiment of the present invention, after calculating the congestion delay estimate based on the in-transit excess and the effective path rate, the multi-tenant multi-path reliable transmission control method further includes: queue-aware dynamic timeout threshold calculation and timeout processing, comprising the following steps: calculating a dynamic timeout threshold based on the basic round-trip timeout, the congestion delay estimate, delay jitter, and tenant level; when a data packet sent through the path exceeds the dynamic timeout threshold and has not received an acknowledgment, triggering timeout event processing, including: obtaining the congestion delay estimate of the path and the tenant level of the tenant; when the congestion delay estimate is higher than the congestion delay threshold, determining that the current timeout is caused by queuing delay, extending the waiting time of the data packet, reducing the path weight of the path, and reducing the selection of the path for new data packets; when the congestion delay estimate is not higher than the congestion delay threshold, determining that the current timeout is caused by actual packet loss or path anomaly, performing selective retransmission of the data packet, and reducing the path health score of the path.

[0051] In a specific embodiment, the sending end determines the base round-trip time (RTT) based on the base round-trip time (RTT). ), congestion delay estimate ( ), latency jitter ( ) and the timeout bias corresponding to the tenant level ( ), calculate a dynamic timeout threshold ( In the formula, This represents the congestion delay waiting coefficient. This is the latency jitter tolerance factor. (This is a tenant priority adjustment factor). When a data packet exceeds this threshold and is still not acknowledged, a timeout event is triggered. At this time, the sender checks the congestion delay estimate of the path: if the congestion delay estimate is higher than the congestion delay threshold, it indicates that the current network queue is deep, and the timeout is determined to be caused by queuing delay. The operation at this time is to extend the waiting time and reduce the path weight, but not to trigger retransmission; if the congestion delay estimate is not higher than the congestion delay threshold, it indicates that the queue is not deep, and the timeout is more likely to be caused by actual packet loss or path anomaly. Only then will selective retransmission be performed on the data packet, and the path health score will be reduced.

[0052] Specifically, this invention effectively solves the problem of "false retransmissions" through a queue-aware dynamic timeout threshold calculation and timeout handling mechanism. The core of this mechanism is that the calculation of the timeout threshold uses the estimated congestion delay of the current path as a key input, enabling the timeout threshold to adaptively scale with the degree of congestion. More importantly, after a timeout occurs, the system does not blindly retransmit, but makes a second decision based on the "congestion delay estimate": when it is determined to be a queuing delay, the system chooses to "wait" instead of "retransmit," which fundamentally eliminates false retransmissions caused by queuing delays. This mechanism can effectively break the vicious cycle of "congestion-delay-timeout-retransmission-more severe congestion," greatly saving valuable network bandwidth and queue resources, thereby helping to improve the effective throughput of multi-tenant networks under high load.

[0053] In one embodiment of the present invention, after sending data packets through the target path, the multi-tenant-oriented multi-path reliable transmission control method further includes: hierarchical packet loss recovery, comprising the following steps: when the negative acknowledgment count is greater than zero, determining the data packets to be recovered based on the negative acknowledgment, selectively retransmitting the data packets, reducing the path weight of the original transmission path of the data packets, and clearing the negative acknowledgment count to zero after the retransmission is completed, keeping the path in the active path pool, and not removing the path directly due to a single negative acknowledgment; when the selective acknowledgment gap count is greater than zero, starting a short out-of-order waiting window; after the short out-of-order waiting window ends, if the selective acknowledgment gap has been filled by subsequent acknowledgment, then clearing the selective acknowledgment gap count to zero; if the selective acknowledgment gap still exists, then processing the missing data... Selective retransmission is performed on the data packets. When the cumulative count of selective acknowledgment gaps on the same path reaches a preset gap count threshold, the path health score of the path is reduced. When the path health score is lower than a preset health threshold, the path is removed from the active path pool, and a path is selected from the backup path pool to be added to the active path pool. When a data packet exceeds the dynamic timeout threshold and is still not acknowledged, and the count of negative acknowledgment gaps is zero, and the count of selective acknowledgment gaps is zero, the congestion delay estimate of the path and the tenant's tenant level are read. When the congestion delay estimate is higher than the congestion delay threshold, it is determined that the queuing delay has slowed down the acknowledgment, the waiting time is extended, and the path weight of the path is reduced. When the congestion delay estimate is not higher than the congestion delay threshold, it is determined that there is a real packet loss or path anomaly, selective retransmission is performed, and the path health score of the path is reduced.

[0054] In a specific embodiment, the system executes differentiated recovery strategies for three different packet loss / missing events: 1. NACK event: After receiving a NACK, the sender immediately performs selective retransmission on the indicated data packet, reducing the path weight of the original path, but keeping the path in the active pool; this reflects the fast response characteristic of NACK as a "strong signal", but it is believed that a single NACK does not represent a complete path failure.

[0055] 2. SACK Gap Event: After the sender detects a SACK gap, it first starts a short out-of-order waiting window; if the gap still exists after the window ends, it performs retransmission and accumulates the SACK gap count of the path; when the count reaches a preset threshold, the path is removed from the active path pool and a new path is selected from the backup pool to be added; this reflects the prudent judgment and final handling of the "persistent" problem.

[0056] 3. Normal timeout events: The judgment is made in combination with the congestion delay estimate. If the queue is deep, wait; if the queue is shallow, retransmit and may be downgraded.

[0057] Specifically, this invention establishes a three-tiered processing mechanism—NACK, SACK gap, and normal timeout—to achieve fine-grained control over packet loss recovery. Each event corresponds to a different network state and recovery strategy: NACK indicates that the receiver has clearly lost packets and needs to respond immediately; SACK gaps may be caused by out-of-order delivery or packet loss, and a short waiting window is given before making a judgment, avoiding invalid retransmissions caused by out-of-order delivery; normal timeouts are intelligently judged based on congestion status. This hierarchical design avoids a "one-size-fits-all" approach, ensuring rapid recovery in emergency situations while preventing overreaction and path oscillations caused by network jitter, thereby helping to improve the robustness and bandwidth utilization of the system.

[0058] In one embodiment of the present invention, after a path is removed from the active path pool, the multi-tenant multi-path reliable transmission control method further includes: dynamic replacement of the active path pool and the backup path pool, including the following steps: selecting a backup path from the tenant's backup path pool and adding it to the active path pool to maintain the number of paths in the active path pool; performing background probing on the abnormal path that has been removed from the active path pool; when the background probing successfully reaches a preset recovery threshold consecutively, adding the abnormal path back to the backup path pool and waiting for subsequent scheduling; when the background probing does not reach the recovery threshold, keeping the abnormal path in an inactive state.

[0059] In a specific embodiment, when an active path is removed due to reasons such as a health score falling below a threshold, the system will immediately select a backup path from the tenant's backup path pool and add it to the active path pool to maintain the number of active paths. At the same time, background probing is performed on the removed abnormal paths (such as periodically sending a small number of probe packets). When the background probing successfully reaches a preset recovery threshold consecutively, the path will be added back to the backup path pool to wait for subsequent scheduling; if the threshold is not reached, it will remain inactive.

[0060] Specifically, this embodiment of the invention achieves path resource management and rapid self-healing through a dynamic replacement mechanism between the active path pool and the backup path pool. The existence of the backup path pool allows the replacement process of faulty paths to be completed instantly, minimizing the impact on upper-layer training tasks. The background detection mechanism ensures that removed paths can be reintegrated into the available resource pool after a period of recovery, improving the overall utilization of network resources. This mechanism forms a complete closed loop for path lifecycle management, which helps to further enhance the stability of multi-tenant networks in complex environments.

[0061] In one embodiment of the present invention, during the transmission of data packets through the target path, the multi-tenant multi-path reliable transmission control method further includes: tenant-level retransmission budget control, comprising the following steps: setting a retransmission budget for each tenant, wherein the retransmission budget represents the retransmission traffic quota allowed for the tenant within a preset rolling time window, and the retransmission traffic quota is expressed in any one of the following measurement methods: retransmission bytes, retransmission packets, retransmission requests, retransmission bandwidth percentage, or retransmission tokens; including both normal transmission traffic and retransmission traffic in the tenant's resource consumption, and updating the corresponding tenant each time a retransmission occurs. The system counts the actual retransmission traffic of each tenant. When the actual retransmission traffic count of any tenant reaches the retransmission budget, budget control operations are performed, including: limiting new retransmissions triggered by ordinary timeout events; retaining necessary retransmissions triggered by negative acknowledgment events and persistent selective acknowledgment gap events; increasing the dynamic timeout threshold for subsequent ordinary timeout events on each path of the tenant; reducing the path weight of the tenant on congested paths to reduce the selection of congested paths for new data packets; and removing abnormal paths from the active path pool and selecting paths from the backup path pool to add to the active path pool when there are available backup paths for the tenant.

[0062] In a specific embodiment, the system sets a retransmission budget for each tenant (such as the number of retransmission bytes allowed within a rolling time window). Both normal traffic and retransmission traffic are included in the tenant's resource consumption. When the actual retransmission traffic of any tenant reaches the budget threshold, the system will perform a series of budget control operations: restrict new retransmissions triggered by normal timeout events for that tenant; retain necessary retransmissions triggered by NACK and persistent SACK gaps; increase the dynamic timeout threshold for subsequent normal timeout events for that tenant; reduce the weight of that tenant on congested paths; and, if necessary, remove its abnormal paths from the active pool and replace them from the standby pool.

[0063] Specifically, this embodiment of the invention introduces a tenant-level retransmission budget control mechanism, treating retransmission traffic as a limited, managed tenant resource. This is of paramount importance in a multi-tenant environment: it effectively prevents a single tenant from triggering a "retransmission storm" due to misconfiguration, code defects, or network anomalies, thereby consuming a large amount of shared network resources and affecting all other tenants. By limiting unnecessary ordinary timeout retransmissions and retaining critical necessary retransmissions, this mechanism establishes a security barrier while ensuring the basic reliability of each tenant, thus helping to ensure the fairness and overall stability of the entire multi-tenant network.

[0064] In one embodiment of the present invention, after sending data packets through the target path, the multi-tenant multi-path reliable transmission control method further includes tenant connection release and resource reclamation, including the following steps: stopping the transmission of new data packets for all paths in the authorized path pool corresponding to the tenant; performing cleanup and retransmission or discarding processing on the currently unacknowledged data packets of the tenant according to the hierarchical retransmission strategy; and after confirming that the tenant's transmission status has stabilized, reclaiming the tenant's port resources, network plane resources, path pool resources and retransmission budget, and re-incorporating the reclaimed resources into the allocable resource set.

[0065] In a specific embodiment, when a tenant's training task ends or the connection is closed, the sender first stops sending new data packets to all paths in the authorized path pool corresponding to that tenant; then, it performs cleanup retransmission or discarding of its currently unacknowledged data packets according to the hierarchical retransmission strategy; after confirming that the transmission status has stabilized, the tenant controller reclaims the resources occupied by the tenant, such as ports, network planes, path pools, and retransmission budgets, and re-includes them in the set of allocable resources for subsequent tenants to use.

[0066] Specifically, the embodiments of the present invention ensure the clean release and recycling of resources through a clear tenant connection release and resource reclamation process; thereby avoiding resource residue and memory leaks caused by abnormal tenant exits, which helps to improve the long-term stability and resource utilization of the entire system and is an important part of building a production-grade multi-tenant network system.

[0067] The advantages and benefits of the multi-tenant-oriented multi-path reliable transmission control method of the present invention will be described below with reference to a specific embodiment. In this specific embodiment, a multi-tenant-oriented multi-path reliable transmission control method is provided.

[0068] Figure 2 This is an overall flowchart of a multi-tenant, multi-path reliable transmission control method according to a specific embodiment of the present invention. In this specific embodiment, as... Figure 2 As shown, the multi-tenant-oriented multi-path reliable transmission control method includes the following steps: Step S10: Tenant Policy Table Generation and Distribution: The tenant controller generates and distributes a tenant policy table for the tenant based on the tenant access request or the tenant description information distributed by the scheduling system. The tenant policy table includes tenant identifier, tenant priority, network plane quota, path pool size, bandwidth quota, retransmission budget, timeout configuration, and backup path pool. The sending end allocates ports, network plane, active path pool, and backup path pool for the tenant based on the tenant policy table, and initializes the protocol status of each tenant and each path, including the sending sequence number, acknowledgment sequence number, number of bytes sent, number of bytes acknowledged, number of bytes unacknowledged, basic round-trip time, path effective rate, path weight, and path health score.

[0069] Step S20: Tenant Path Mapping Table Generation and Distribution: The tenant controller generates a mapping table for each tenant from "tenant identifier + path selection identifier" to segmented route IPv6 path, and distributes it to the sending end or relevant network devices.

[0070] Step S30: Packet Path Selection and SRv6 Encoding: When sending a packet, the sender selects a path selection identifier based on the tenant identifier, path weight, and path status, and encodes the corresponding SRv6 (Segmented Routing IPv6) path into the packet header.

[0071] Step S40: Protocol state maintenance and congestion parameter calculation: The sender maintains the number of bytes sent, the number of acknowledged bytes, and the number of unacknowledged bytes according to the tenant and path dimensions, and calculates the in-transit excess and congestion delay estimate based on the base bandwidth-delay product.

[0072] Step S50: Receiver feedback processing and status update; Receiver returns acknowledgment, selective acknowledgment, or negative acknowledgment; Sender updates acknowledgment status, gap status, path weight, health score, and retransmission budget consumption based on feedback.

[0073] Step S60: Tiered packet loss recovery processing: When the sender receives a negative acknowledgment, it immediately performs selective retransmission and reduces the path weight; when the sender detects a selective acknowledgment gap, it waits for a short out-of-order window and then determines whether to retransmit and remove the path; when the sender detects a normal timeout event, it combines the congestion delay estimate and tenant level to decide whether to extend the wait, reduce the path weight, or perform retransmission.

[0074] Step S70: Dynamic replacement between active path pool and backup path pool: When a path is removed from the active path pool, the system selects a new path from the tenant's backup path pool to add to the active path pool, and performs background detection on abnormal paths; after the detection is successful for several consecutive times and the recovery threshold is reached, the path enters the backup path pool; if the detection does not reach the recovery threshold, the path remains inactive.

[0075] Specifically, when a path is deemed abnormal due to a continuous selective confirmation response gap, shallow queue timeout, or health score below a threshold, the system removes the path from the active path pool and selects an alternative path from the tenant's backup path pool to add to the active path pool. At the same time, background probing is performed on the removed path. After the probing is successful for several consecutive times and the recovery threshold is reached, the path returns to the backup path pool to wait for subsequent scheduling.

[0076] Step S80: Tenant Connection Release and Resource Reclamation: When a tenant's training task ends, the tenant connection is closed, or the scheduling system issues a resource release instruction, the sending end performs necessary cleanup and retransmission or discarding of the tenant's currently unacknowledged data according to the hierarchical retransmission strategy, and stops the transmission of new data packets for all paths in the active path pool and backup path pool corresponding to the tenant; after confirming that the relevant transmission status has stabilized, the tenant controller reclaims the tenant's port, network plane, active path pool, backup path pool, and retransmission budget, and re-includes them in the controller's allocable resource set for subsequent tenants to use.

[0077] In summary, the multi-path reliable transmission control method for multi-tenants provided by this invention constitutes a complete reliable transmission solution for multi-tenant large-scale model training scenarios through a series of interconnected innovative technologies, including tenant resource slicing, tenant-aware path mapping, sender-end congestion self-computation, queue-aware weight adjustment, dynamic timeout and graded retransmission, tenant-level retransmission budget, and dynamic path pool replacement. This effectively addresses core pain points in multi-tenant shared networks such as resource isolation, performance interference, congestion misjudgment, and retransmission storms, significantly improving the overall stability, fairness, and transmission efficiency of the multi-tenant network environment.

[0078] In other words, the multi-tenant-oriented multi-path reliable transmission control method provided by this invention has at least the following advantages compared with the prior art: 1. Supports path resource isolation in multi-tenant shared training networks: Existing MRC mainly focuses on reliable multi-path transmission within a unified training cluster, with path resources used by the system as a whole; This invention allocates ports, network planes, active path pools, and backup path pools based on tenant level and tenant weight, and determines the tenant's authorized path through a combination of tenant identifier and path selection identifier. Therefore, different tenants can only send and retransmit data within their respective authorized path pools, reducing disorderly competition among tenants for the same set of path resources.

[0079] 2. Improve the tenant controllability of multi-path scheduling: The existing MRC path selection identifier to path mapping is mainly oriented towards the global path set; the present invention extends this mapping relationship to a tenant-aware mapping, so that the sending end is constrained by both tenant identity and path pool permissions when selecting a path. Therefore, the system can perform path scheduling according to the resource quota and service level of different tenants, instead of using the same global path pool for all tenants.

[0080] 3. Improve the accuracy of path congestion judgment: Existing MRC mainly uses feedback such as ECN, SACK, and NACK for congestion avoidance and packet loss recovery; this invention further calculates the in-transit excess amount by the sending end based on the number of bytes sent, the number of acknowledgments, the basic round-trip time, and the effective path rate, and obtains the current congestion delay estimate of the path accordingly. Therefore, the system can more accurately judge whether there is queuing congestion on the path, providing a basis for subsequent path weight adjustment and timeout judgment.

[0081] 4. Reduce false retransmissions caused by misjudging queuing delays as packet loss: In multi-tenant scenarios, a sudden surge in traffic from one tenant may cause the queue on the shared path to lengthen, thus delaying data packets or acknowledgments from other tenants. Existing solutions, which rely solely on timeouts or missing packets to determine packet loss, are prone to triggering false retransmissions. This invention introduces a congestion delay estimate into the normal timeout judgment. When the congestion delay estimate is higher than the congestion delay threshold, the system extends the waiting time and reduces the path weight. When the congestion delay estimate is not higher than the congestion delay threshold, the system selectively retransmits and reduces the path health score. Therefore, this invention can distinguish between "acknowledgment delays caused by queueing" and "real packet loss or path anomalies," reducing unnecessary retransmissions.

[0082] 5. More refined handling of NACK, SACK gaps and ordinary timeout events: Although existing MRC has SACK, NACK and selective retransmission capabilities, different packet loss events have different meanings for path status in multi-tenant expansion scenarios. This invention divides packet loss recovery into three levels: NACK triggers immediate selective retransmission and reduces path weight but does not clear the path; SACK gaps, if they occur repeatedly after passing through a short out-of-order window, are retransmitted and the path is removed; ordinary timeouts are judged based on congestion delay estimates and tenant level. Therefore, this invention can avoid excessive punishment for congestion-type packet loss and can handle continuous abnormal paths in a timely manner.

[0083] 6. Preventing premature path removal due to short-term congestion: Existing multi-path reliable transmission mechanisms may directly reduce path availability or even stop using a path when packet loss or anomalies are detected. This invention maintains path weight and path health score separately. Queue depth, increased ECN, and NACK mainly affect path weight, reducing the use of the path by subsequent new packets. Only persistent SACK gaps, shallow queue timeouts, and repeated anomalies affect the path health score. Once the health score falls below a threshold, the path is removed from the active path pool. Therefore, this invention can avoid misjudging short-term queuing congestion as path failure and improve the stability of the path pool.

[0084] 7. Provide differentiated reliable transmission capabilities according to tenant level: Existing MRC main services unified training system does not emphasize the service level differences between different tenants; this invention sets different timeout thresholds, retransmission sensitivity and path switching strategies according to tenant level. Therefore, high-level tenants can obtain faster packet loss recovery and path switching, while low-level tenants adopt a more conservative retransmission strategy to reduce the additional occupation of shared network resources.

[0085] 8. Reduce the congestion amplification of retransmission traffic on the shared network: In a multi-tenant network, retransmission traffic will share link and queue resources with normal training traffic; this invention sets a retransmission budget for each tenant and includes retransmission traffic in tenant resource management. Therefore, when the retransmission traffic of a certain tenant reaches the budget threshold, the system can limit retransmissions triggered by normal timeouts, avoiding the retransmission storm of a single tenant from affecting other tenants.

[0086] 9. Improve the overall stability and predictability of multi-tenant training networks: This invention forms a complete closed loop by integrating tenant resource allocation, path pool management, queue-aware scheduling, dynamic timeout, hierarchical retransmission, and retransmission budget control. Therefore, compared with the existing MRC scheme, this invention is more suitable for multi-tenant large model training networks and can improve tenant isolation, path scheduling controllability, packet loss recovery accuracy, and shared network stability.

[0087] Further embodiments of the present invention disclose a multi-tenant, multi-path reliable transmission control system. Figure 3 This is a structural block diagram of a multi-tenant, multi-path reliable transmission control system according to an embodiment of the present invention. Figure 3 As shown, in one embodiment of the present invention, the multi-tenant-oriented multi-path reliable transmission control system 100 includes: a tenant control module 110, a segmented routing IPv6 path pool module 120, a queue-aware scheduling module 130, a reliability recovery module 140, and a multi-plane physical network module 150.

[0088] Specifically, the tenant control module 110 is used to generate and distribute a tenant policy table for each tenant. The tenant policy table includes a tenant identifier, a tenant level, and an authorized path pool. The authorized path pool contains multiple network paths, and the authorized path pools associated with the tenant identifiers of different tenants are isolated from each other.

[0089] The segmented routing IPv6 path pool module 120 is used to store a preset tenant path mapping table. The tenant path mapping table records the segmented routing IPv6 paths corresponding to combinations of different tenant identifiers and different path selection identifiers. The same path selection identifier corresponds to different forwarding paths in different network planes under different tenants.

[0090] The queue-aware scheduling module 130 is used to initialize the transmission status of each tenant on each network path in the corresponding authorized path pool according to the tenant policy table. In response to the data packet to be sent, it queries the tenant path mapping table according to the tenant identifier and path selection identifier of the tenant to which the data packet belongs, determines the corresponding segmented route IPv6 path from the authorized path pool associated with the tenant identifier as the target path, and sends the data packet through the target path.

[0091] The reliability recovery module 140 is used to perform graded packet loss recovery (including but not limited to handling NACK events, SACK gap events and ordinary timeout events, and maintaining the health status of the path).

[0092] The multi-plane physical network module 150 is used to provide physical transmission resources with multiple ports and multiple network planes.

[0093] It should be noted that the specific implementation of the multi-tenant-oriented multi-path reliable transmission control system 100 in this embodiment of the invention is similar to the specific implementation of the multi-tenant-oriented multi-path reliable transmission control method described in the above embodiment of the invention, and therefore has similar technical effects. For details, please refer to the description of the multi-tenant-oriented multi-path reliable transmission control method section. To reduce redundancy, it will not be repeated here.

[0094] Specifically, according to embodiments of the present invention, the multi-tenant multi-path reliable transmission control system implements control plane policy distribution through a tenant control module; implements tenant-aware path mapping and storage through a segmented routing IPv6 path pool module; implements tenant-aware routing and state maintenance of the data plane through a queue-aware scheduling module; implements hierarchical and intelligent packet loss recovery through a reliability recovery module; and provides underlying hardware resource support through a multi-plane physical network module. This multi-tenant multi-path reliable transmission control system is the physical carrier of the aforementioned multi-tenant multi-path reliable transmission control method. The modules work together to achieve the technical effects of path resource isolation, congestion-aware scheduling, false retransmission suppression, and rapid recovery in a multi-tenant large-scale model training network.

[0095] Further embodiments of the present invention also disclose an electronic device, Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Figure 4 As shown, in one embodiment of the present invention, the electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; when the processor executes the computer programs stored in the memory, it implements the multi-tenant-oriented multi-path reliable transmission control method as described in any of the above embodiments of the present invention.

[0096] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0097] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0098] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0099] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0100] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0101] It should be noted that the specific implementation of the electronic device in the embodiments of the present invention is similar to the specific implementation of the multi-tenant-oriented multi-path reliable transmission control method described in the above embodiments of the present invention, and therefore has similar technical effects. For details, please refer to the description of the multi-tenant-oriented multi-path reliable transmission control method section. In order to reduce redundancy, it will not be repeated here.

[0102] Further embodiments of the present invention disclose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-tenant, multi-path reliable transmission control method as described in any of the above embodiments of the present invention.

[0103] It should be noted that the specific implementation of the computer-readable storage medium in the embodiments of the present invention is similar to the specific implementation described in the above embodiments of the multi-tenant multi-path reliable transmission control method of the present invention, and therefore has similar technical effects. For details, please refer to the description of the multi-tenant multi-path reliable transmission control method section. In order to reduce redundancy, it will not be repeated here.

[0104] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0105] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-tenant, multi-path reliable transmission control method, characterized in that, Includes the following steps: A tenant policy table is generated and distributed for each tenant. The tenant policy table includes a tenant identifier, a tenant level, and an authorization path pool. The authorization path pool contains multiple paths, and the authorization path pools associated with the tenant identifiers of different tenants are isolated from each other. The transmission status of each tenant on each path in the corresponding authorized path pool is initialized according to the tenant policy table; In response to a data packet to be sent, a preset tenant path mapping table is queried based on the tenant identifier and path selection identifier of the tenant to which the data packet belongs. The corresponding segmented route IPv6 path is determined from the authorized path pool associated with the tenant identifier as the target path. The tenant path mapping table is used to record the segmented route IPv6 paths corresponding to different combinations of tenant identifiers and different path selection identifiers. The same path selection identifier corresponds to different forwarding paths in different network planes under different tenants. The data packet is sent through the target path.

2. The multi-tenant-oriented multi-path reliable transmission control method according to claim 1, characterized in that, Before generating and distributing the tenant policy table for each tenant, the multi-tenant-oriented multi-path reliable transmission control method further includes: tenant resource slicing and path pool construction, including the following steps: Obtain the tenant level and / or tenant weight for each tenant; Based on the tenant level and / or the tenant weight, port resources, network plane resources, and path pool resources are allocated to each tenant. The allocation ratio of the port resources, the allocation ratio of the network plane resources, and the allocation ratio of the path pool resources are positively correlated with the tenant weight, and under the same tenant weight, the tenant with the higher tenant level is given priority in resource allocation. Based on the allocated port resources, network plane resources, and path pool resources, an authorized path pool is constructed for each tenant. A portion of the paths in the authorized path pool is divided into an active path pool, another portion is divided into a backup path pool, and unallocated reserved resources are used as a global backup resource pool.

3. The multi-tenant-oriented multi-path reliable transmission control method according to claim 2, characterized in that, The transmission status includes the number of bytes sent, the number of bytes acknowledged, the number of bytes unacknowledged, the basic round-trip time, the effective path rate, the path weight, and the path health score; During the transmission of the data packet through the target path, the multi-tenant-oriented multi-path reliable transmission control method further includes: path state maintenance and congestion awareness, comprising the following steps: The basic bandwidth-delay product is calculated based on the effective rate of the path and the basic round-trip time, and the amount of data in transit that can naturally exist on the path in the absence of queuing is determined based on the basic bandwidth-delay product. The in-transit excess is calculated based on the unconfirmed byte count and the product of the base bandwidth and delay. The congestion delay estimate is calculated based on the in-transit excess and the effective path rate, and the current congestion queuing delay level is determined based on the congestion delay estimate.

4. The multi-tenant-oriented multi-path reliable transmission control method according to claim 3, characterized in that, After calculating the congestion delay estimate based on the in-transit excess and the effective path rate, the multi-tenant-oriented multi-path reliable transmission control method further includes: queue-aware path weight adjustment, comprising the following steps: The congestion delay estimate, explicit congestion notification count, negative acknowledgment count, selective acknowledgment gap count, historical success rate, and path health score of the obtained path are also included. When the estimated congestion delay is lower than a preset congestion delay threshold, and the explicit congestion notification count, the negative acknowledgment count, and the selective acknowledgment gap count are all zero, the path weight of the path is increased so that new data packets preferentially select the path. When the congestion delay estimate increases or the explicit congestion notification count increases, the path weight of the path is reduced so that new packets use the path less, but the path is not removed from the active path pool. When the count of negative acknowledgments is greater than zero, selective retransmission is performed on the data packet corresponding to the negative acknowledgment, and the path weight of the path is reduced within a preset time window; When the selective confirmation response gap count on the same path reaches a preset gap count threshold, the path health score of the path is reduced. When the path health score is lower than a preset health threshold, the path is removed from the active path pool.

5. The multi-tenant-oriented multi-path reliable transmission control method according to claim 4, characterized in that, After calculating the congestion delay estimate based on the in-transit excess and the effective path rate, the multi-tenant-oriented multi-path reliable transmission control method further includes: queue-aware dynamic timeout threshold calculation and timeout processing, comprising the following steps: Calculate the dynamic timeout threshold based on the base round-trip timeout, the congestion delay estimate, the delay jitter, and the tenant level; When a data packet sent via the path fails to receive an acknowledgment after exceeding the dynamic timeout threshold, a timeout event is triggered, including: Obtain the congestion delay estimate for the path and the tenant class for the tenant; When the estimated congestion delay is higher than the congestion delay threshold, it is determined that the current timeout is caused by queuing delay, the waiting time of the data packet is extended, the path weight of the path is reduced, and the selection of the path by new data packets is reduced; When the estimated congestion delay is not higher than the congestion delay threshold, it is determined that the current timeout is caused by actual packet loss or path anomaly. Selective retransmission is performed on the data packet, and the path health score of the path is reduced.

6. The multi-tenant-oriented multi-path reliable transmission control method according to claim 5, characterized in that, After sending the data packet through the target path, the multi-tenant-oriented multi-path reliable transmission control method further includes: graded packet loss recovery, comprising the following steps: When the negative acknowledgment count is greater than zero, the data packet that needs to be recovered is determined based on the negative acknowledgment, selective retransmission is performed on the data packet, the path weight of the original transmission path of the data packet is reduced, and the negative acknowledgment count is cleared to zero after the retransmission is completed, so that the path is kept in the active path pool and the path is not directly removed due to a single negative acknowledgment. When the selective acknowledgment (SAC) gap count is greater than zero, a short out-of-order waiting window is initiated. After the short out-of-order waiting window ends, if the selective acknowledgment gap has been filled by a subsequent acknowledgment, the selective acknowledgment gap count is reset to zero. If the selective acknowledgment gap still exists, selective retransmission is performed on the missing data packets. When the cumulative selective acknowledgment gap count on the same path reaches a preset gap count threshold, the path health score of the path is reduced. When the path health score is lower than a preset health threshold, the path is removed from the active path pool, and a path is selected from the backup path pool to be added to the active path pool. When a data packet remains unacknowledged beyond the dynamic timeout threshold, and the negative acknowledgment count is zero, and the selective acknowledgment gap count is zero, the congestion delay estimate of the path and the tenant's tenant level are read. If the congestion delay estimate is higher than the congestion delay threshold, it is determined that queuing delay has slowed down acknowledgment, the waiting time is extended, and the path weight of the path is reduced. If the congestion delay estimate is not higher than the congestion delay threshold, it is determined that there is actual packet loss or path anomaly, selective retransmission is performed, and the path health score of the path is reduced.

7. The multi-tenant-oriented multi-path reliable transmission control method according to claim 6, characterized in that, After removing the path from the active path pool, the multi-tenant-oriented multi-path reliable transmission control method further includes: dynamically replacing the active path pool with the backup path pool, including the following steps: Select a backup path from the tenant's backup path pool and add it to the active path pool to maintain the number of paths in the active path pool; Perform background probing on abnormal paths that have been removed from the active path pool; When the background detection successfully reaches the preset recovery threshold consecutively, the abnormal path is added back to the backup path pool for subsequent scheduling. When the background detection does not reach the recovery threshold, the abnormal path remains in an inactive state.

8. The multi-tenant-oriented multi-path reliable transmission control method according to claim 6, characterized in that, During the transmission of the data packet through the target path, the multi-tenant-oriented multi-path reliable transmission control method further includes: tenant-level retransmission budget control, comprising the following steps: Set a retransmission budget for each tenant, wherein the retransmission budget represents the retransmission traffic quota allowed for the tenant within a preset rolling time window, and the retransmission traffic quota is expressed in any one of the following measurement methods: retransmission bytes, retransmission packets, retransmission requests, retransmission bandwidth percentage, or retransmission tokens. Both normal transmission traffic and retransmission traffic are included in the tenant's resource consumption, and the actual retransmission traffic count of the corresponding tenant is updated each time a retransmission occurs. When the actual retransmission traffic count of any tenant reaches the retransmission budget, a budget control operation is performed, including: Restrict new retransmissions triggered by ordinary timeout events for the aforementioned tenant; Retain necessary retransmissions triggered by negative acknowledgment events and persistent selective acknowledgment gap events for the tenant; Increase the dynamic timeout threshold for subsequent normal timeout events on each path of the tenant; Reduce the path weight of the tenant on the congested path to reduce the selection of the congested path for new data packets; When a tenant has an available alternative path, the abnormal path is removed from the active path pool, and a path is selected from the alternative path pool and added to the active path pool.

9. The multi-tenant-oriented multi-path reliable transmission control method according to claim 8, characterized in that, After sending the data packet through the target path, the multi-tenant-oriented multi-path reliable transmission control method further includes tenant connection release and resource reclamation, comprising the following steps: Stop sending new data packets for all paths in the authorized path pool corresponding to the tenant; For the tenant's currently unacknowledged data packets, perform cleanup and retransmission or discard processing according to the hierarchical retransmission strategy; After confirming that the tenant's transmission status has stabilized, the tenant's port resources, network plane resources, path pool resources, and retransmission budget are reclaimed, and the reclaimed resources are reintegrated into the allocatable resource set.

10. A multi-tenant, multi-path reliable transmission control system, characterized in that, include: The tenant control module is used to generate and distribute a tenant policy table for each tenant. The tenant policy table includes a tenant identifier, a tenant level, and an authorized path pool. The authorized path pool contains multiple network paths, and the authorized path pools associated with the tenant identifiers of different tenants are isolated from each other. The segmented routing IPv6 path pool module is used to store a preset tenant path mapping table. The tenant path mapping table is used to record the segmented routing IPv6 paths corresponding to combinations of different tenant identifiers and different path selection identifiers. The same path selection identifier corresponds to different forwarding paths in different network planes under different tenants. The queue-aware scheduling module is used to initialize the transmission status of each tenant on each network path in the corresponding authorized path pool according to the tenant policy table. In response to the data packet to be sent, the module queries the tenant path mapping table according to the tenant identifier and path selection identifier of the tenant to which the data packet belongs, determines the corresponding segmented route IPv6 path from the authorized path pool associated with the tenant identifier as the target path, and sends the data packet through the target path. The reliability recovery module is used to perform tiered packet loss recovery. Multi-plane physical network module, used to provide physical transmission resources with multiple ports and multiple network planes.

Images