Traffic control method, device, apparatus and storage medium

Abstract

The application discloses a traffic control method and device, equipment and storage medium, and relates to the technical field of network communication. The method comprises the following steps: obtaining a communication event generated in a training process of a current training task, and constructing a semantic transmission unit corresponding to the communication event; determining a safe sending rate of the semantic transmission unit according to network state data corresponding to a wide-area transmission network; determining an upper limit of a semantic rate of the semantic transmission unit based on semantic feature information of the semantic transmission unit, wherein the semantic feature information is used for representing an emergency degree of the communication event in the training process; determining a final sending rate of the semantic transmission unit according to the safe sending rate and the upper limit of the semantic rate, and performing traffic control on the semantic transmission unit according to the final sending rate. By using the technical scheme, the technical problem that the existing traffic control method cannot adapt to the dynamic change of the communication emergency degree with the training stage in large model training, resulting in insufficient precision of traffic control, is solved.

Description

Technical Field

[0001] This application relates to the field of network communication technology, and in particular to flow control methods, apparatus, devices and storage media. Background Technology

[0002] As the scale of deep learning models such as Large Language Models (LLMs), training data, and parallel training continues to expand, distributed training systems typically require multiple training nodes to collaborate on model training. Each node contains one or more Graphics Processing Units (GPUs), Central Processing Units (CPUs), memory, Remote Direct Memory Access (RDMA) network cards, and a high-speed interconnect network, exchanging data through the high-speed interconnect network. In training scenarios involving cross-regional, cross-data center, or long-distance links (such as wide-area RDMA scenarios), effectively controlling the massive RDMA traffic generated by training tasks is crucial to ensuring training efficiency and network stability.

[0003] Currently, existing flow control technologies for wide-area RDMA typically employ priority-based or static classification-based network control techniques. These techniques can differentiate between different types of traffic by configuring static priorities. However, during large model training, the urgency of different communication operations (such as gradient synchronization, data prefetching, and checkpoint writing) dynamically changes with the training phase and computation progress. Static priority configurations often fail to accurately reflect these real-time semantic requirements, resulting in insufficient flow control precision and consequently affecting the overall efficiency and stability of distributed training. Summary of the Invention

[0004] The main objective of this application is to provide a flow control method, apparatus, device, and storage medium, which aims to solve the technical problem that existing flow control methods cannot adapt to the dynamic changes in the urgency of communication during the training phase of large model training, resulting in insufficient accuracy of flow control.

[0005] To achieve the above objectives, this application proposes a flow control method, the method comprising: The communication events generated during the training process of the current training task are obtained, and the semantic transmission unit corresponding to the communication event is constructed. The semantic transmission unit is used to describe the runtime characteristics of the communication event in the current training task and the network characteristics during the data transmission process. The secure transmission rate of the semantic transmission unit is determined based on the network status data corresponding to the wide area transmission network. The semantic rate limit of the semantic transmission unit is determined based on the semantic feature information of the semantic transmission unit, and the semantic feature information is used to characterize the urgency of the communication event during the training process. The final transmission rate of the semantic transmission unit is determined based on the secure transmission rate and the upper limit of the semantic rate, and flow control of the semantic transmission unit is performed according to the final transmission rate.

[0006] In one embodiment, the step of constructing the semantic transmission unit corresponding to the communication event includes: Obtain the training semantic data and network transmission attributes of communication events generated during the training process of the current training task; The training phase, communication type, and service category of the communication event are determined based on the training semantic data and the network transmission attributes. Determine the estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal gain, and semantic confidence for the communication event; Based on the training phase, the communication type, the service category, the estimated remaining completion time, the semantic deadline, the slackness, the semantic loss, the semantic marginal gain, and the semantic confidence, a semantic transmission unit corresponding to the communication event is constructed.

[0007] In one embodiment, the step of determining the estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal gain, and semantic confidence corresponding to the communication event includes: The estimated remaining completion time of the communication event is determined based on the remaining data volume of the communication event and the historical completion rate estimate. The semantic deadline of the communication event is determined based on the dependencies between the training phase, the communication type, and the current training task. The slack of the communication event is determined based on the semantic deadline, the current time, and the estimated remaining completion time; The semantic loss of the communication event is determined based on the relaxation degree and the weight of the training progress of the communication event. The semantic marginal benefit of the communication event is determined based on the semantic loss and the amount of candidate service data for the communication event; The semantic confidence of the communication event is determined based on the first completeness of the training semantic data, the second completeness of the network transmission attributes, and the credibility of historical statistical information.

[0008] In one embodiment, the step of determining the secure transmission rate of the semantic transmission unit based on the network state data corresponding to the wide area transmission network includes: Based on the network status data corresponding to the wide area transmission network, determine the receiver pressure of the target receiver, the path pressure of the candidate transmission path, the baseline rate of the underlying congestion control feedback, and the upper limit of the resource group rate corresponding to the semantic transmission unit. The upper limit of the resource group rate is the hardware configurable rate upper limit of the queue group or the flow group. The receiver security rate corresponding to the semantic transmission unit is determined based on the receiver pressure, and the path security rate corresponding to the semantic transmission unit is determined based on the path pressure. The basic security rate of the semantic transmission unit is determined based on the reference rate, the receiver security rate, the path security rate, and the resource group rate limit, and the basic security rate is determined as the secure transmission rate of the semantic transmission unit.

[0009] In one embodiment, the step of determining the semantic rate limit of the semantic transmission unit based on the semantic feature information of the semantic transmission unit includes: Based on the semantic feature information of the semantic transmission unit, determine the semantic confidence, semantic marginal benefit, resource penalty value, risk penalty value, and historical service debt value corresponding to the service category of the semantic transmission unit. The admission score of the semantic transmission unit is determined based on the semantic confidence, the semantic marginal benefit, the resource penalty value, the risk penalty value, and the historical service arrears value. The admission score is used to characterize the priority of the communication event being served in the current control period. Each semantic transmission unit within the same resource contention group is assigned a semantic rate cap according to the admission score, wherein the sum of the semantic rate caps does not exceed the total security budget of the resource contention group.

[0010] In one embodiment, the step of performing flow control on the semantic transmission unit according to the final transmission rate includes: The credit quantity of the target receiver corresponding to the sending end is determined based on the final transmission rate and the control period length. The credit quantity is used to control the quota of data fragments released by the sending end to the target receiver. The release interval of adjacent data fragments is determined based on the final transmission rate and the data fragment size of the semantic transmission unit; The data fragments are released according to the credit amount, and each data fragment is sent sequentially according to the release interval to perform flow control on the semantic transmission unit.

[0011] In one embodiment, after the step of performing flow control on the semantic transmission unit according to the final transmission rate, the method further includes: The data integrity of the target semantic confidence scores of all currently active semantic transmission units in the current training task, the current network status data of the wide area transmission network, and the network telemetry data is obtained. The average semantic confidence score is determined based on the target semantic confidence scores. The current system state is determined based on the average semantic confidence, the current network state data, and the data integrity. The current system state includes semantic degradation state, network protection state, and complete rollback state. An abnormal rollback strategy is determined based on the current system state, and an abnormal adjustment is made based on the abnormal rollback strategy.

[0012] Furthermore, to achieve the above objectives, this application also proposes a flow control device, the device comprising: The semantic unit construction module is used to acquire communication events generated during the training process of the current training task and construct semantic transmission units corresponding to the communication events. The semantic transmission units are used to describe the runtime characteristics of the communication events in the current training task and the network characteristics during the data transmission process. The secure rate determination module is used to determine the secure transmission rate of the semantic transmission unit based on the network status data corresponding to the wide area transmission network. The rate limit determination module is used to determine the semantic rate limit of the semantic transmission unit based on the semantic feature information of the semantic transmission unit, wherein the semantic feature information is used to characterize the urgency of the communication event during the training process. The flow control module is used to determine the final transmission rate of the semantic transmission unit based on the secure transmission rate and the semantic rate limit, and to perform flow control on the semantic transmission unit according to the final transmission rate.

[0013] In addition, to achieve the above objectives, this application also proposes a flow control device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the flow control method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the flow control method described above.

[0015] This application provides a flow control method. It discloses acquiring communication events generated during the training process of the current training task and constructing semantic transmission units corresponding to these events. The semantic transmission units describe the runtime characteristics of the communication events in the current training task and their network characteristics during data transmission. The method determines the safe transmission rate of the semantic transmission units based on network state data corresponding to the wide-area transmission network. It determines the upper limit of the semantic rate of the semantic transmission units based on their semantic feature information, which characterizes the urgency of the communication events during training. Finally, it determines the final transmission rate of the semantic transmission units based on the safe transmission rate and the upper limit of the semantic rate, and performs flow control on the semantic transmission units according to the final transmission rate. Compared to… Existing flow control methods that rely on static priorities cannot adapt to the dynamic changes in communication urgency during large-scale model training. This application addresses this issue by acquiring real-time training semantic data and network transmission attributes of communication events, constructing semantic transmission units that describe the runtime characteristics of communication events in the current training task and the network characteristics during data transmission. This enables flow control to perceive dynamic semantic information such as training stage and communication type, and adaptively adjust the transmission rate of each communication event at different training stages based on the semantic rate upper limit determined by the semantic feature information. This solves the technical problem that existing flow control methods cannot adapt to the dynamic changes in communication urgency during large-scale model training, resulting in insufficient accuracy of flow control. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the flow control method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the flow control method of this application; Figure 3 This is a flowchart illustrating Embodiment 3 of the flow control method of this application; Figure 4 This is a flowchart of the anomaly detection and rollback process in the flow control method of this application; Figure 5 This is an overall flowchart of the flow control method of this application; Figure 6 A system block diagram provided for the flow control method of this application; Figure 7 This is a schematic diagram of the module structure of the flow control device according to an embodiment of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the flow control method in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is as follows: First, obtain the communication events generated during the training process of the current training task, and construct the semantic transmission unit corresponding to each communication event. The semantic transmission unit is used to describe the runtime characteristics of the communication event in the current training task and the network characteristics during data transmission. Second, determine the secure transmission rate of the semantic transmission unit based on the network status data corresponding to the wide area transmission network. Third, determine the upper limit of the semantic rate of the semantic transmission unit based on the semantic feature information of the semantic transmission unit. The semantic feature information is used to characterize the urgency of the communication event during the training process. Fourth, determine the final transmission rate of the semantic transmission unit based on the secure transmission rate and the upper limit of the semantic rate, and perform flow control on the semantic transmission unit according to the final transmission rate.

[0023] Existing flow control methods that rely on statically configured priorities cannot adapt to the real-time requirements of communication urgency dynamically changing during the training phase in large model training, resulting in insufficient accuracy of flow control.

[0024] This application provides a solution that, by acquiring training semantic data and network transmission attributes of communication events in real time, constructs a semantic transmission unit to describe the runtime characteristics of communication events in the current training task and the network characteristics during data transmission. This enables flow control to perceive dynamic semantic information such as the training stage and communication type, and adaptively adjusts the transmission rate of each communication event at different training stages based on the semantic rate upper limit determined by the semantic feature information. This solves the technical problem that existing flow control methods cannot adapt to the dynamic changes in the urgency of communication during large model training, resulting in insufficient accuracy of flow control.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or flow control device capable of performing the above functions. The following description uses a flow control device (hereinafter referred to as the device) as an example to illustrate this embodiment and the subsequent embodiments.

[0026] Based on this, embodiments of this application provide a flow control method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the flow control method of this application.

[0027] In this embodiment, the flow control method includes steps S10 to S40: Step S10: Obtain the communication events generated during the training process of the current training task, and construct the semantic transmission unit corresponding to the communication event. The semantic transmission unit is used to describe the runtime characteristics of the communication event in the current training task and the network characteristics during the data transmission process.

[0028] Understandably, the current training task can be a large-scale machine learning model training job being executed in a distributed training system. In a distributed training scenario, a training task is typically completed collaboratively by multiple training nodes, each of which can include one or more GPUs, CPUs, memory, RDMA network cards, and high-speed interconnects. For example, in a large language model training scenario based on the Transformer architecture, the current training task could be a language model training job with hundreds of billions of parameters, which can be distributed across multiple training nodes for parallel execution.

[0029] It is also understandable that a training iteration of the current training task typically includes multiple stages such as forward computation, backward computation, gradient synchronization, parameter update, data loading, checkpoint writing, logging, cache prefetching, and parameter or optimizer state unloading. Taking the training of a large language model based on the PyTorch and Megatron-LM framework as an example, the above-mentioned current training task can be a distributed training job of a Transformer model containing hundreds of billions of parameters, which is executed in parallel on multiple GPU nodes.

[0030] It should be understood that a communication event can be any data transfer operation performed between training nodes during the training process of the current training task to complete model training. In large-scale distributed training systems, nodes often need to frequently perform ensemble communication operations, including AllReduce, ReduceScatter, AllGather, and AllToAll. AllReduce is used to aggregate gradients and synchronize the results to each training node; ReduceScatter is used to distribute the results as fragments after reduction; AllGather is used to collect data fragments from each node; and AllToAll is commonly seen in expert parallel scenarios such as MoE, used to achieve data exchange between nodes. Furthermore, the communication events in this embodiment can also include point-to-point communication operations such as Send / Recv, RDMA Write, and RDMA Read, all of which can be considered as a communication event. For example, during a gradient synchronization process, each training node performs an AllReduce ensemble communication; this AllReduce operation is a communication event.

[0031] It should be noted that a semantic transmission unit can be a communication execution object obtained by semantically abstracting communication events, capable of independent admission control and rate adjustment. It can encapsulate the runtime characteristics of a communication event in the current training task and its network characteristics during data transmission into a unified control object. For example, in the gradient synchronization phase, an AllReduce aggregate communication operation is encapsulated as a semantic transmission unit. This unit includes both the runtime characteristic that the AllReduce operation belongs to the gradient synchronization phase and the network characteristics such as the target receiver and candidate paths. In this embodiment, a semantic transmission unit can correspond to an NCCL chunk, an aggregate communication substream, a queue group transmission, a tensor fragment, a checkpoint block, or an optimizer state unloading transmission block, etc. In practical applications, semantic transmission units are used to enable the device to simultaneously consider the impact of communication events on training progress and the network resource usage of communication events when making flow control decisions.

[0032] It should be understood that runtime features can be training progress-related attributes of communication events in the current training task, which can be used to describe the temporal position, dependencies, and urgency of communication events during the training process. Specifically, runtime features can include the training phase to which the communication event belongs (e.g., forward computation phase, backpropagation phase, gradient synchronization phase, parameter update phase, data prefetching phase, checkpoint writing phase, optimizer state loading / unloading phase, etc.), communication type (e.g., AllReduce, ReduceScatter, AllGather, AllToAll, Send / Recv, RDMA Write, RDMA Read, etc.), service category (e.g., critical class, elastic class, background class, etc.), estimated remaining completion time, semantic deadline, slackness, semantic loss, semantic marginal benefit, and semantic confidence, etc. In this embodiment, runtime features are used to enable the device to perceive the urgency of communication events during the training process when making flow control decisions.

[0033] It should also be understood that network characteristics can be the transmission path and resource-related attributes involved in the data transmission of a communication event. These characteristics can be used to describe the network resource usage and transmission constraints of the communication event. Specifically, network characteristics can include information such as the target receiver, queue group information, candidate wide area path or rail, total number of bytes transmitted, number of remaining bytes, and RDMA NIC identifier. Furthermore, network characteristics can also include the queue group or flow group information to which the communication event belongs, as well as the occupancy vectors of the receiver, link, path, queue group, and RNIC resources for that communication event. For example, the network characteristics of a communication event might indicate that its target receiver is Node B, the candidate transmission path is Path 1, and the remaining data volume is 256MB.

[0034] In its implementation, the device first obtains relevant information about communication events from the training framework, the ensemble communication library, and the runtime system. Specifically, the device can obtain the current training stage, iteration number, and dependencies of the current training task by registering runtime hooks with the training framework; and it can obtain the communication type, target receiver, and queue group information of the communication event by listening to communication events in the ensemble communication library. After obtaining the communication event, the device can extract training semantic data and network transmission attributes from the communication event and associate the extracted training semantic data and network transmission attributes with the communication event to generate a semantic transmission unit.

[0035] Step S20: Determine the secure transmission rate of the semantic transmission unit based on the network status data corresponding to the wide area transmission network.

[0036] It should be noted that a wide area transmission network can be a communication network that carries RDMA semantic transmission across regions, data centers, or long distances. For example, a wide area transmission network can be an RDMA network consisting of training nodes distributed in data centers in different cities connected by dedicated lines or the Internet. In wide area transmission network scenarios, RDMA transmission faces challenges such as increased round-trip latency due to long-distance transmission, slow path state changes, incomplete cross-domain network telemetry, and the potential for latency fluctuations in individual key set communications to be amplified into training iteration tail latency.

[0037] It should be understood that network status data can be parameters used to reflect the current operational status of a wide area transmission network, and can be used to assess the current level of network congestion, available bandwidth, and transmission risks. Specifically, network status data can include the baseline rate of the underlying RDMA congestion control feedback, the proportion of ECN (Explicit Congestion Notification) markers, the number of CNP (Congestion Notification Packets), RTT (Round-Trip Time), RTT growth trends, PFC (Priority-based Flow Control) pause duration, send and receive queue counters, receiver buffer occupancy, link utilization, available path bandwidth, path switching events, packet loss rate, retransmission count, timeout events, and whether telemetry data is missing or expired. In practical applications, network status data can be provided by the underlying RDMA congestion control mechanism, receiver queues, link telemetry systems, and wide area path status monitoring systems.

[0038] It should be noted that the safe transmission rate can be the maximum transmission rate allowed for the semantic transmission unit under the current operating conditions of the wide area transmission network, without causing network congestion, packet loss, or PFC pauses. In this embodiment, the safe transmission rate is the upper limit of the transmission rate imposed by the underlying network on the semantic transmission unit, used to ensure that semantic control does not exceed the safe transmission envelope allowed by the underlying RDMA congestion control mechanism. It can be determined by multiple constraints, including the baseline rate of the underlying RDMA congestion control feedback, the safe reception rate that the receiver can withstand, the safe rate of the candidate wide area path, and the hardware configurable rate limit of the queue group or flow group. In this embodiment, the minimum value among these four rates can be used as the basic safe rate of the semantic transmission unit. For example, if the baseline rate of the underlying congestion control feedback is 40Gbps, the safe rate of the receiver is 35Gbps, the safe rate of the path is 38Gbps, and the resource group rate limit is 50Gbps, then the safe transmission rate is 35Gbps.

[0039] In its implementation, after constructing the semantic transmission unit, the device collects network status data from underlying RDMA congestion control, RNIC counters, receiver queues, link telemetry, and wide-area path status. Based on this network status data, it calculates receiver pressure, path pressure, the baseline rate fed back by underlying congestion control, and the upper limit of the resource group rate. Receiver pressure is calculated based on the receiver queue length, completion queue backlog, and available receiver buffer; path pressure is calculated based on round-trip time, round-trip time growth trend, packet loss risk, and link utilization. Subsequently, the device determines the receiver safe rate based on receiver pressure and the path safe rate based on path pressure. The minimum value among the baseline rate fed back by underlying congestion control, the receiver safe rate, the path safe rate, and the upper limit of the resource group rate is then determined as the base safe rate for the semantic transmission unit, and this base safe rate is used as the safe transmission rate for the semantic transmission unit.

[0040] Step S30: Determine the upper limit of the semantic rate of the semantic transmission unit based on the semantic feature information of the semantic transmission unit, wherein the semantic feature information is used to characterize the urgency of the communication event during the training process.

[0041] It should be noted that semantic feature information can be feature data extracted from semantic transmission units to characterize the urgency of communication events during training. It can reflect the expected impact of delayed service on training progress. Specifically, semantic feature information can include semantic confidence, semantic marginal benefit, historical service deficit value corresponding to the service category, resource penalty value, and risk penalty value. Among these, semantic confidence measures the reliability of the semantic recognition result; semantic marginal benefit represents the amount of semantic loss that can be reduced per unit of service volume; historical service deficit value is used to avoid long-term unavailability of low-priority service categories; and resource penalty value and risk penalty value are used to incorporate network pressure into semantic decision-making.

[0042] It should also be noted that the semantic rate cap can be a rate limit value calculated based on the semantic feature information of the communication event, allowing the semantic layer to use the communication event. It is a rate constraint imposed on the communication event by the semantic execution layer based on the training semantics, reflecting the service priority of the communication event during the training process. In practical applications, within the same resource contention group, the semantic rate cap of each semantic transmission unit can be allocated according to the admission score. The semantic transmission unit with the higher admission score obtains a higher semantic rate cap, and the sum of the semantic rate caps of all semantic transmission units does not exceed the total safety budget of the resource contention group. In this embodiment, the semantic rate cap is used to enable the training semantics to directly participate in flow control decisions, allowing high-urgency communication events to obtain network resources preferentially. For example, for a communication event in the gradient synchronization phase with a relaxation close to zero, its semantic rate cap is set to a large proportion of the total safety budget of the resource contention group to ensure that the communication event can be completed in a timely manner and avoid GPU idling.

[0043] In its implementation, after determining the safe transmission rate, the device extracts semantic feature information from the semantic transmission unit and calculates the admission score of the semantic transmission unit based on the semantic confidence, semantic marginal benefit, resource penalty value, risk penalty value, and historical service arrears value in the semantic feature information. The admission score characterizes the priority of communication events being served within the current control period. Then, the device can allocate semantic rate caps to each semantic transmission unit within the same resource contention group according to their admission scores.

[0044] Step S40: Determine the final transmission rate of the semantic transmission unit based on the secure transmission rate and the semantic rate limit, and perform flow control on the semantic transmission unit according to the final transmission rate.

[0045] It should be understood that the final transmission rate can be the actual rate at which the semantic transmission unit transmits data. This rate is the final execution rate constrained by both the secure transmission rate and the semantic rate cap. In this embodiment, the final transmission rate can be the smaller value between the secure transmission rate and the semantic rate cap. The secure transmission rate represents the secure rate boundary allowed by the underlying RDMA network, and the semantic rate cap represents the priority rate allocation determined by the training semantics. The final transmission rate simultaneously satisfies both network security constraints and training semantic priority constraints. In practical applications, the final transmission rate ensures that the semantic execution layer can only perform admission, sorting, and rate limiting within the secure transmission envelope allowed by the underlying congestion control, and will not exceed the secure boundary given by the underlying congestion control due to high semantic priority.

[0046] In its implementation, after determining the secure transmission rate and the upper limit of the semantic rate, the device compares these two values ​​and determines the smaller value as the final transmission rate of the semantic transmission unit. Subsequently, the device determines the credit quantity for the target receiver corresponding to the sender based on the final transmission rate and the control period length. This credit quantity controls the quota of data fragments released by the sender to the target receiver. Furthermore, the device determines the release interval between adjacent data fragments based on the final transmission rate and the data fragment size of the semantic transmission unit. Finally, data fragments are released according to the credit quantity, and each data fragment is sent sequentially according to the release interval, thereby achieving flow control of the semantic transmission unit.

[0047] This embodiment provides a flow control method. The method discloses acquiring communication events generated during the training process of the current training task and constructing semantic transmission units corresponding to these communication events. The semantic transmission units describe the runtime characteristics of the communication events in the current training task and their network characteristics during data transmission. The method determines the safe transmission rate of the semantic transmission units based on network state data corresponding to the wide area transmission network. It determines the upper limit of the semantic rate of the semantic transmission units based on their semantic feature information, which characterizes the urgency of the communication events during training. Finally, it determines the final transmission rate of the semantic transmission units based on the safe transmission rate and the upper limit of the semantic rate, and performs flow control on the semantic transmission units according to the final transmission rate. Compared to… Existing flow control methods that rely on static priorities cannot adapt to the dynamic changes in communication urgency during large model training. This embodiment acquires training semantic data and network transmission attributes of communication events in real time, and constructs a semantic transmission unit to describe the runtime characteristics of communication events in the current training task and the network characteristics during data transmission. This enables flow control to perceive dynamic semantic information such as training stage and communication type, and adaptively adjust the sending rate of each communication event in different training stages based on the semantic rate upper limit determined by the semantic feature information. This solves the technical problem that existing flow control methods cannot adapt to the dynamic changes in communication urgency during large model training, resulting in insufficient accuracy of flow control.

[0048] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating Embodiment 2 of the flow control method of this application.

[0049] In this embodiment, the step of constructing the semantic transmission unit corresponding to the communication event includes steps S11 to S14: Step S11: Obtain the training semantic data and network transmission attributes of the communication events generated during the training process of the current training task.

[0050] It should be noted that training semantic data can be information describing the training progress-related attributes of communication events in the current training task. This information may include, but is not limited to, the current training task number, the current training iteration number, the current training stage (such as the forward computation stage, backpropagation stage, gradient synchronization stage, parameter update stage, data prefetching stage, checkpoint writing stage, optimizer state loading / unloading stage), training dependencies, input buffer level, prefetch queue length, checkpoint lag, optimizer state access time, GPU wait time, etc. In this embodiment, training semantic data can be obtained from the training framework (such as PyTorch, TensorFlow), the collection communication library (such as NCCL), and the runtime system.

[0051] It should also be noted that network transmission attributes can be information describing the transmission path and resource-related attributes involved in the data transmission process of a communication event. These attributes may include, but are not limited to, the communication type of the communication event, the set communication identifier, the tensor identifier, the data block identifier, the target receiver, queue group information, candidate wide area paths or rails, the total number of bytes transmitted, the number of remaining bytes, and the RDMA network card identifier. In this embodiment, network transmission attributes can be obtained from the communication event monitoring interface of the set communication library and the queue pair metadata during RDMA runtime.

[0052] In practical applications, devices can acquire training semantic data by registering runtime hooks with the training framework. Specifically, the device can register callback functions at key execution points of the training framework (such as the start of forward computation, the start of backward computation, the start of gradient synchronization, and the start of parameter updates). When the training framework reaches a key execution point, the callback function is triggered and reports the identifier of the current training phase, iteration number, and dependency information to the device. Simultaneously, the device can also acquire network transmission attributes by listening to communication events in the ensemble communication library. Specifically, the device can register a listening interface on the communication execution path of the ensemble communication library. When the ensemble communication library is about to execute a communication operation, the listening interface is triggered and reports information such as the communication type, ensemble communication identifier, tensor identifier, data block identifier, target receiver, queue group information, candidate wide-area path, total number of bytes transmitted, and number of bytes remaining. Furthermore, the device can also obtain information such as input buffer level, prefetch queue length, checkpoint lag, optimizer state access time, and GPU wait time from the runtime system. Finally, the device can integrate the acquired information to obtain the training semantic data and network transmission attributes of the communication events.

[0053] Step S12: Determine the training stage, communication type, and service category of the communication event based on the training semantic data and the network transmission attributes.

[0054] It should be understood that the training phase can be the specific execution stage of the current training task within a complete training iteration. In large-scale distributed training, a training iteration typically involves multiple different computation and communication phases sequentially, including the forward computation phase, backpropagation phase, gradient synchronization phase, parameter update phase, data prefetching phase, checkpoint writing phase, and optimizer state loading / unloading phase. Specifically, in the forward computation phase, each training node performs the forward propagation computation of the neural network; in the backpropagation phase, each training node performs the backpropagation computation and calculates the gradients; in the gradient synchronization phase, each training node synchronizes gradient data through ensemble communication; in the parameter update phase, each training node updates the model parameters based on the synchronized gradients; in the data prefetching phase, each training node preloads the data needed for the next round of training from the storage system; in the checkpoint writing phase, each training node saves the current model state to persistent storage; and in the optimizer state loading / unloading phase, each training node migrates the optimizer state between GPU memory and CPU memory or storage. In practical applications, the training phase can be provided by runtime hooks of the training framework; for example, a hook function of the training framework can indicate that the current stage is the gradient synchronization phase.

[0055] It should also be understood that the communication type can be the type of specific aggregate communication operation performed by the communication event or the type of point-to-point communication operation. It can be used to determine the data exchange mode and the scope of participating nodes in the communication event, including AllReduce, ReduceScatter, AllGather, AllToAll, Send / Recv, RDMA Write, and RDMA Read. Specifically, AllReduce aggregates data from all participating nodes and distributes the aggregation result to all nodes; ReduceScatter performs data reduction and distributes the fragmented result to nodes; AllGather collects data fragments from each node and broadcasts them to all nodes; AllToAll enables data exchange between all nodes; Send / Recv is used for point-to-point sending and receiving between two nodes; RDMA Write directly writes data to remote memory via the RDMA protocol; and RDMA Read directly reads data from remote memory via the RDMA protocol. In practical applications, the communication type can be provided by the aggregate communication library. For example, the communication event listening interface of the aggregate communication library can indicate that the current communication event is of type AllReduce.

[0056] It should also be understood that service categories can be classifications of communication events based on their urgency and latency tolerance during training. These categories can be used to differentiate the importance of different communication events in the training progress, enabling differentiated flow control strategies during resource contention. Specifically, service categories can include critical, resilient, and background categories. Specifically, if a communication event is located on the synchronization barrier of the current iteration, a pipelined parallel dependency chain, a tensor parallel critical dependency chain, a MoE AllToAll critical path, or is about to block parameter updates, then the service category of that communication event is critical. If the communication event corresponds to normal prefetching, normal unloading, normal checkpointing, or deferred MoE communication, then the service category of that communication event is resilient. If the communication event corresponds to data rebalancing, cache filling, background maintenance, or low-urgency checkpointing, then the service category of that communication event is background. In practical applications, the device can comprehensively infer the service category based on the training stage, communication type, and training dependencies. For example, an AllReduce communication located on the synchronization barrier of the gradient synchronization stage is determined to be a critical service category.

[0057] In its implementation, after acquiring training semantic data and network transmission attributes, the device can determine the training stage, communication type, and service category of a communication event based on these attributes. Specifically, the device can determine the training stage to which the communication event belongs based on the training framework hook information in the training semantic data, and determine the communication type based on the set communication library information in the network transmission attributes. Then, the device can determine the service category of the communication event based on the training stage, communication type, and dependencies of the current training task. If the communication event is located on the synchronization barrier of the current iteration, the pipelined parallel dependency chain, the tensor parallel critical dependency chain, the MoE AllToAll critical path, or is about to block parameter updates, the service category of the communication event can be determined as the critical category; if the communication event corresponds to normal prefetching, normal offloading, normal checkpointing, or deferred MoE communication, the service category of the communication event can be determined as the elastic category; if the communication event corresponds to data rebalancing, cache filling, background maintenance, or low-urgency checkpointing, the service category of the communication event can be determined as the background category.

[0058] Step S13: Determine the estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal gain, and semantic confidence corresponding to the communication event.

[0059] Understandably, the estimated remaining completion time can be the estimated time required to complete the communication event, which can be used to assess the urgency of completing the communication event at the current service rate. In this embodiment, the estimated remaining completion time can be calculated based on the current remaining data volume of the communication event and the historical completion rate estimate. Specifically, the estimated remaining completion time can be equal to the remaining data volume divided by the historical completion rate estimate, where the historical completion rate estimate is obtained by performing an exponential moving average of the actual completion rates observed over the most recent control cycles. Furthermore, the estimated remaining completion time can be dynamically updated as the execution progress and rate of the communication event change.

[0060] It is also understandable that the semantic deadline can be the latest time when a communication event should be completed, indicating that after this time, communication delays may negatively impact the training process, including barrier waiting, GPU idling, pipeline bubbles, prefetch starvation, optimizer blocking, or checkpoint lag. In practical applications, the semantic deadline can be calculated based on the training stage, communication type, and training dependencies. For example, for barrier-related ensemble communication, the semantic deadline can be determined based on the start time of subsequent computation stages or synchronization points; for pipelined parallel transmission, the semantic deadline can be determined based on the activation or gradient time required for downstream micro-batches; for prefetch transmission, the semantic deadline can be determined based on the expected exhaustion time of the input buffer; for optimizer state unloading or loading transmission, the semantic deadline can be determined based on the time required for the corresponding optimizer state during the parameter update stage; for checkpoint transmission, the semantic deadline can be determined based on the maximum allowable checkpoint lag; and for background maintenance transmission, the semantic deadline can be determined based on the minimum service cycle or background task completion window.

[0061] It's important to note that relaxation can be considered as the remaining time margin before the semantic deadline for a communication event, and it can be used to quantify the urgency of the communication event in terms of time. In practical applications, relaxation can be calculated based on the semantic deadline, the current time, and the estimated remaining completion time. For example, relaxation can be equal to the semantic deadline minus the current time minus the estimated remaining completion time. Specifically, if the relaxation is greater than zero, it means the communication event is expected to complete before the deadline, with a time leeway; if the relaxation is equal to zero, it means the communication event is expected to complete exactly at the deadline; if the relaxation is less than zero, it means the communication event is expected to exceed the deadline, potentially causing training blockage.

[0062] It should also be noted that semantic loss can be a quantification of the expected negative impact of delayed communication events on the training process. It can be used to assess the extent to which communication events, when not served in a timely manner, harm the training progress. In this embodiment, semantic loss can be obtained by a weighted sum of multiple loss terms, which may include barrier waiting loss, GPU idling loss, pipeline bubble loss, prefetch starvation loss, and checkpoint lag loss. In practical applications, to ensure that semantic loss changes continuously around the deadline, the device can use a relaxation-based smoothing penalty function for calculation. When the relaxation is large, the penalty approaches zero; when the relaxation is close to zero or negative, the penalty increases rapidly.

[0063] It should be noted that the semantic marginal benefit can be the amount of semantic loss that can be reduced by serving a unit of data for the communication event within the current control period. It can be used to quantify the marginal contribution of serving the communication event to the training progress. In this embodiment, the semantic marginal benefit can be calculated based on the rate of change of semantic loss. If the semantic marginal benefit is large, it means that the communication can significantly reduce training pauses after being served, and should be given priority admission; if the semantic marginal benefit is small, it means that the communication still has a large latency tolerance and can be appropriately postponed.

[0064] It should also be noted that semantic confidence can be a quantitative value of the credibility of the semantic recognition results of communication events. It can be used to measure the reliability of semantic understanding results such as training phase recognition, service category judgment, semantic deadline estimation, and dependency recognition of communication events. In this embodiment, semantic confidence can be calculated by weighting the completeness of training framework events, the completeness of set communication events, the clarity of dependencies, and the credibility of historical statistical information. Its value ranges from 0 to 1, with a higher value indicating a more credible semantic recognition result. In practical applications, when the semantic confidence is lower than a preset threshold, the device does not use fine-grained semantic control, but instead enters a semantic degradation mode or uses coarse-grained service category control.

[0065] Further, step S13 includes: Step S131: Determine the estimated remaining completion time of the communication event based on the remaining data volume of the communication event and the historical completion rate estimate.

[0066] Understandably, the remaining data volume can be the total amount of data in a communication event that has not yet been transmitted at the current moment, and it can be used to characterize the transmission progress of the communication event. During RDMA transmission, the total data volume of a communication event is usually divided into multiple data chunks (e.g., NCCL chunks) for transmission. The device can calculate the remaining data volume at the current moment in real time by monitoring the number of data chunks that have been confirmed to have been transmitted and combining it with the total data volume of the communication event.

[0067] It is also understood that the historical completion rate estimate can be the data transmission rate actually observed in the communication event within one or more recent historical control cycles, which can be used to estimate the transmission capacity of the communication event under current network conditions. In this embodiment, the historical completion rate estimate can be obtained by performing an exponential moving average on the actual completion rate observed in the most recent control cycle to eliminate the noise effect caused by instantaneous fluctuations. The corresponding calculation formula can be:

[0068] In the formula, This represents the smoothed completion rate at the current moment, which is also the estimated historical completion rate. This represents the actual completion rate observed in the most recent control cycle; This indicates the smooth completion rate of the previous control cycle; Indicates the length of the control cycle; This represents the rate smoothing coefficient, with a value range of (0,1).

[0069] Among them, the actual completion rate in the most recent control cycle The corresponding calculation formula can be:

[0070] In the formula, This refers to a communication unit (i.e., a semantic transmission unit). In the interval The actual number of bytes completed.

[0071] The device can then calculate the estimated remaining completion time based on the number of remaining bytes and historical completion rate estimates. The corresponding calculation formula can be:

[0072] in, Indicates communication unit The estimated remaining completion time; Indicates the number of bytes remaining. This represents an estimate of the historical completion rate; It is a very small positive number used to avoid division by zero when the historical rate is 0.

[0073] Step S132: Determine the semantic deadline of the communication event based on the dependency relationship between the training phase, the communication type, and the current training task.

[0074] In this embodiment, different methods can be used to calculate the semantic deadline for different types of communication events. Specifically, for barrier-related aggregated communication, the semantic deadline... It can be determined based on the start time of subsequent calculation stages or synchronization points, and the corresponding calculation formula is as follows:

[0075] In the formula, This indicates the estimated start time for the subsequent calculation phase. This indicates that a safety time has been reserved.

[0076] For pipeline parallel transmission This can be derived from the activation or gradient time required by the downstream micro-batch, and the corresponding calculation formula can be:

[0077] In the formula, This indicates the time it takes for the downstream micro-batch to need this data. This indicates the safety redundancy time of the production line.

[0078] For prefetch transmission This can be derived from the estimated exhaustion time of the input buffer, and the corresponding calculation formula is as follows:

[0079] In the formula, This indicates the amount of data currently available in the input buffer. This indicates the rate at which the training process consumes input data. This indicates the safe reserve time for the buffer zone.

[0080] For optimizer state unloading or loading transport. This can be derived from the time required for the corresponding optimizer state during the parameter update phase, and the corresponding calculation formula can be:

[0081] In the formula, This indicates the time when the state is accessed during the parameter update phase. This indicates the time allotted for optimizer access to safety.

[0082] For checkpoint transmission It can be derived from the maximum permissible checkpoint lag, and the corresponding calculation formula is:

[0083] In the formula, This indicates the time when the last checkpoint was successfully completed. This indicates the maximum permissible checkpoint delay.

[0084] For background maintenance and transmission It can come from the minimum service cycle or the background task completion window, and the corresponding calculation formula is:

[0085] In the formula, Indicates the start time of the background task. This indicates that the window is allowed to complete.

[0086] Step S133: Determine the slack of the communication event based on the semantic deadline, the current time, and the estimated remaining completion time.

[0087] Understandably, the current time can be the system time corresponding to the time when the device performs various calculation operations within the current control cycle, which can be provided by the device's system clock.

[0088] In this embodiment, the relaxation degree can be calculated based on the semantic deadline, the current time, and the estimated remaining completion time. The corresponding calculation formula is as follows:

[0089] In the formula, Indicates communication unit At any moment The degree of relaxation; Indicates a semantic deadline; Indicates the current time; This indicates the estimated remaining completion time.

[0090] In practical applications, if If so, it means that the communication event is expected to be completed before the deadline; if If , it means that the communication event is expected to complete exactly at the deadline; if If the communication event is expected to exceed the deadline, it may cause training to be blocked.

[0091] Step S134: Determine the semantic loss of the communication event based on the relaxation and the training progress influence weight of the communication event.

[0092] It should be understood that the training progress impact weight can be a coefficient used to characterize the contribution of different types of training progress loss to the total semantic loss. It can be used to reflect the differences in the impact of different training stages or different communication event types on training progress, thereby making the calculation of semantic loss more consistent with the actual training scenario. In this embodiment, the semantic loss can be obtained by weighted summation of multiple loss terms (including barrier waiting loss, GPU idling loss, pipeline bubble loss, prefetch starvation loss, and checkpoint lag loss). Each loss term corresponds to a training progress impact weight, which can be determined by the communication category, training stage, and historical analysis results. For example, for critical AllReduce communication located on the synchronization barrier in the gradient synchronization stage, its barrier waiting loss weight can be set to a higher value (e.g., 0.4), because this communication delay will directly cause the GPU to idle at the barrier; while for background communication, its checkpoint lag loss weight can be set to a lower value (e.g., 0.1), because the latency tolerance of checkpoint transmission is relatively high.

[0093] In this embodiment, the formula for calculating semantic loss can be:

[0094] In the formula, Indicates communication unit At any moment Total semantic loss; This indicates a barrier awaiting loss; This indicates GPU idle time loss; This indicates bubble loss in the production line; This indicates the anticipated loss due to hunger; This indicates the loss due to checkpoint delays; , , , and These represent the weights of the corresponding loss terms, that is, the weights that affect the training progress.

[0095] In practical applications, to ensure that the semantic loss varies continuously around the deadline, the device can employ a relaxation-based smoothing penalty function, the corresponding calculation formula of which is:

[0096] In the formula, This represents the smoothing penalty value based on relaxation; when When it is large, A value close to 0 indicates that there is still ample time for the communication; when When it is close to 0 or negative, A rapid increase indicates that communication delays pose a risk to training.

[0097] In addition, typical loss terms can be calculated as follows:

[0098]

[0099]

[0100]

[0101]

[0102] In the formula, This indicates the probability that the GPU will idle if the communication fails to complete in time. This indicates the probability of bubbles forming in the production line; This indicates the probability of causing prefetch starvation; Indicates the target checkpoint interval.

[0103] Step S135: Determine the semantic marginal benefit of the communication event based on the semantic loss and the amount of candidate service data for the communication event.

[0104] Understandably, the candidate service data volume can be the amount of data to be transmitted that the device assumes to allocate to the communication event within a control period. This volume can be used to calculate the degree to which semantic loss can be reduced after serving this portion of data. In this embodiment, the value of the candidate service data volume can be fixed (e.g., the size of a data fragment) or dynamically determined based on the current remaining data volume of the communication event and the length of the control period.

[0105] In this embodiment, the formula for calculating the semantic marginal benefit can be:

[0106] In the formula, Indicates communication unit At any moment The semantic marginal benefit, that is, the decrease in semantic loss caused by the amount of service per unit byte; This represents the semantic loss when no additional services are currently provided; This indicates that the current cycle is the transmission period for this communication unit. Semantic loss after bytes; This indicates the number of candidate service bytes available to be allocated to this communication unit in the current cycle. In practical applications, if... A larger value indicates that the communication service can significantly reduce training pauses and should be given priority access; if The smaller value indicates that the communication still has a large tolerance for delay and can be appropriately postponed.

[0107] Step S136: Determine the semantic confidence of the communication event based on the first completeness of the training semantic data, the second completeness of the network transmission attributes, and the credibility of historical statistical information.

[0108] It should be noted that the first completeness can be an indicator used to characterize whether the training semantic data is complete and without omissions during the acquisition and transmission process, and it can be used to measure whether the acquired training semantic data is sufficient to support accurate semantic recognition. Correspondingly, the second completeness can be an indicator used to characterize whether the network transmission attributes are complete and without omissions during the acquisition and transmission process, and it can be used to measure whether the acquired network transmission attributes are sufficient to support accurate communication event recognition.

[0109] It should be understood that historical statistical information can be historical data records accumulated over multiple control periods to assist in semantic recognition and confidence assessment. This may include, but is not limited to, statistics on the time spent during historical training phases, statistics on the completion rate of historical communication events, statistics on the prediction error of historical semantic deadlines, and statistics on the accuracy of historical service category judgments. In practical applications, historical statistical information can help the device determine whether the current semantic recognition result is consistent with historical patterns, thereby improving the reliability of semantic confidence assessment.

[0110] It should also be understood that credibility can be an indicator used to characterize the reliability of historical statistical information for the reference value of current semantic recognition results. It can be used to measure whether historical statistical information is sufficient to support the verification of the semantic recognition results of current communication events.

[0111] In this embodiment, the formula for calculating semantic confidence can be:

[0112] In the formula, Indicates communication unit At any moment semantic confidence, This indicates the completeness of the training framework events, which is also the first completeness of the training semantic data; This represents the integrity of the collection of communication events, which is also the second integrity of network transmission attributes; Indicates the reliability of dependency identification; Indicates the reliability of historical statistical information; , , and It is a weight, and satisfies:

[0113] Step S14: Construct the semantic transmission unit corresponding to the communication event based on the training phase, the communication type, the service category, the estimated remaining completion time, the semantic deadline, the slackness, the semantic loss, the semantic marginal gain, and the semantic confidence.

[0114] In this embodiment, the semantic transmission unit can be defined as:

[0115] in, This represents the i-th semantic transmission unit; This indicates the communication unit number, which is assigned at runtime; This indicates the training task number, provided by the training framework. This indicates the current training iteration number, provided by the training framework; This indicates the training phase, provided by the training framework hook or training scheduler. The communication type is indicated by the collection communication library; The service category is inferred by this module based on the training phase, communication type, and dependencies. The target receiver is indicated by the collection communication library or QP metadata; qpi indicates the QP or QP group to which it belongs, provided by the RDMA runtime; pathi indicates the candidate wide area path or rail, provided by the routing, network interface card, or runtime path table. The remaining number of bytes at time t is obtained by subtracting the number of bytes completed from the total number of bytes. This represents the historical completion rate estimate, calculated from the actual throughput over the most recent control cycles. The semantic deadline is estimated from the training dependencies, buffer states, checkpoint states, and optimizer state access times. This represents semantic confidence, with a value range of [0,1], and is calculated from semantic source integrity and consistency. This represents a resource occupancy vector, used to describe the unit's occupancy of resources at the receiver, link, path, QP group, and RNIC.

[0116] Each communication unit Resources Occupation (That is, the resource occupancy vector) can be represented as:

[0117] In the formula, Indicates communication unit Use resources , This indicates that it is not used. Additionally, if it is necessary to describe the resource usage percentage, this can also be set to... For example, if a communication unit only occupies a portion of the bandwidth of a certain path, then It can be set to the corresponding usage ratio.

[0118] Furthermore, to avoid frequent oscillations in admission decisions caused by transient noise, this embodiment can adjust any network observation index. Smoothing using exponential moving average:

[0119] in, Represents the resources observed in the current period. Network metrics such as RTT, CNP count, PFC duration, queue length, or retransmission count; Indicates the smoothed index; This represents the smoothing coefficient, and its value range is... ; Indicates the control cycle.

[0120] In this embodiment, the estimated remaining completion time is determined based on the remaining data volume of the communication event and the historical completion rate estimate. The semantic deadline is determined based on the training stage, communication type, and training dependencies. The slack is then determined based on the semantic deadline, the current time, and the estimated remaining completion time. The semantic loss is determined based on the slack and the weight of the training progress impact. The semantic marginal benefit is determined based on the semantic loss and the candidate service data volume. Finally, the semantic confidence is determined based on the first integrity of the training semantic data, the second integrity of the network transmission attributes, and the credibility of historical statistical information. This constructs a multi-dimensional semantic feature set for the communication event, including time constraints, progress impact, and confidence assessment. Since multi-dimensional semantic features can accurately quantify the time urgency and delayed service cost of each communication event during the training process, and semantic confidence reflects the reliability of the semantic recognition results, flow control decisions can be finely scheduled based on accurate semantic information under normal conditions. When semantic confidence is insufficient, it automatically switches to a conservative control mode, thereby improving training communication efficiency while ensuring the robustness and security of control decisions.

[0121] In practical implementation, after determining the training phase, communication type, service category, estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal gain, and semantic confidence corresponding to the communication event, the device can encapsulate the training phase, communication type, service category, estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal gain, semantic confidence, and information such as the target receiver, queue group information, candidate wide area path, and remaining bytes in the network transmission attributes into a unified semantic transmission unit. This semantic transmission unit is then added to the semantic transmission unit set of the current control cycle for use in subsequent processing steps.

[0122] In this embodiment, by acquiring the training semantic data and network transmission attributes of communication events generated during the training process of the current training task, and determining the training stage, communication type, and service category of the communication events based on the training semantic data and network transmission attributes, the estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal benefit, and semantic confidence of the communication events are then determined. Finally, based on this information, a semantic transmission unit corresponding to the communication events is constructed, enabling the original communication events and training semantic information to be integrated into a unified control object, achieving precise quantification and differentiated expression of the urgency of communication events. Simultaneously, because the semantic transmission unit contains multi-dimensional semantic feature information such as estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal benefit, and semantic confidence, subsequent traffic control can adopt differentiated rate control strategies for communication events of different service categories and urgency levels. This ensures that critical path communication is prioritized while avoiding long-term starvation of low-priority traffic, thereby improving the accuracy of traffic control and training efficiency in distributed training scenarios.

[0123] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating Embodiment 3 of the flow control method of this application.

[0124] In this embodiment, step S20 further includes steps S21 to S23: Step S21: Determine the receiver pressure of the target receiver, the path pressure of the candidate transmission path, the baseline rate of the underlying congestion control feedback, and the upper limit of the resource group rate based on the network status data corresponding to the wide area transmission network. The upper limit of the resource group rate is the hardware configurable rate upper limit of the queue group or the flow group.

[0125] It should be understood that the target receiver can be the remote computing node or RDMA network card endpoint to which the communication data corresponding to the semantic transmission unit is to be sent. This can be provided by the ensemble communication library or QP metadata, typically represented as a node identifier, IP address, RDMA network card identifier, or queue pair number. In distributed training scenarios, the target receiver can be other GPU nodes participating in the same training task. For example, in an AllReduce ensemble communication, the target receiver can be one of all other nodes participating in that ensemble communication.

[0126] It should also be understood that receiver pressure can be seen as an inverse measure of the target receiver's capacity margin to process incoming data from various senders at the current moment. It can be used to reflect the degree of risk of continuing to inject data into the target receiver. Specifically, a higher receiver pressure indicates that continuing to send data to that receiver is more likely to cause buffer overflow or processing congestion, thereby triggering PFC pause or packet loss.

[0127] In this embodiment, the receiving end pressure can be calculated based on the receiving queue length, the completion queue backlog, and the available receiving buffer. The corresponding calculation formula is as follows:

[0128] In the formula, Indicates the receiving end At any moment The receiving end pressure; Indicates the length of the receiver's receive queue; Indicates the maximum safe length of the receive queue; This indicates that the queue has been completed or that the queue backlog has been cleared. This indicates the maximum safe backlog in the queue. This indicates that the receiving end has a receive buffer available; This indicates the maximum receive buffer size at the receiving end; , and Represent the weights and satisfy:

[0129] It should be noted that a candidate transmission path can be one or more wide area network transmission paths that the communication data corresponding to the semantic transmission unit can take from the sending end to the target receiving end. In wide area RDMA transmission scenarios that span regions or data centers, there are usually multiple physical or logical paths between the sending end and the receiving end, such as paths formed by different optical cables, different wavelength division multiplexing channels, or different routing strategies. The candidate transmission path is one of these paths that can be used for transmission.

[0130] It should also be noted that path pressure can be an indicator used to characterize the congestion level and transmission quality of a candidate transmission path at the current moment. It can be used to describe the risk level of continuing to send data through that path. Specifically, higher path pressure indicates that continuing to use that path is more likely to cause increased congestion, increased retransmissions, or timeouts. In this embodiment, path pressure can be calculated based on the current round-trip time, the round-trip time growth trend, the risk of packet loss or retransmission, and link utilization. The corresponding calculation formula is as follows:

[0131] In the formula, Representing a path At any moment Path pressure; Representing a path The current round-trip time; Representing a path Historical benchmark round-trip time; This indicates an increasing trend in round-trip time delay; This indicates a risk of packet loss, retransmission, or timeout. Indicates path utilization; , , and This represents the weights, and their sum is 1. In practical applications, if the path round-trip delay increases rapidly, retransmissions increase, or utilization is too high, then... Increasing the size of the semantic admissions can reduce the sudden injection of semantics into the path.

[0132] It should be noted that the underlying congestion control can be an existing congestion control mechanism in the RDMA transport layer, such as DCQCN (Data Center Quantized Congestion Notification), HPCC (High Precision Congestion Control), TIMELY, or Swift congestion control protocols. In practical applications, the underlying congestion control mechanism can adjust the transmission rate based on network layer or transport layer signals, which may include ECN tags, CNP messages, RTT, queue length, PFC pause frames, and other information.

[0133] It should also be noted that the reference rate can be the allowed transmission rate fed back to the sender by the underlying RDMA congestion control mechanism at the current moment. It can reflect the safe rate boundary that will not cause network congestion deterioration, calculated by the underlying congestion control mechanism based on network layer and transport layer signals (such as ECN tags, CNP messages, RTT, queue length, etc.). In this embodiment, the reference rate can serve as one of the constraints on the basic safe rate to ensure that the semantic execution layer can only allocate rates within the safe rate envelope allowed by the underlying congestion control.

[0134] It should be noted that the resource group rate limit can be a hardware-configurable rate limit for queue groups or flow groups. This can be provided by the network interface card (NIC) driver or switch configuration and can be statically or dynamically configured based on hardware capabilities. A queue group can refer to a set of queue pairs (QPs) with the same communication semantics, target receiver, or resource contention relationship; a flow group can refer to a set of flows with the same transmission characteristics. In practical applications, the resource group rate limit can reflect rate constraints at the hardware or driver level, used to limit the maximum transmission rate of a specific communication flow at a very fine granular level, preventing excessive consumption of hardware resources by traffic within a particular queue group.

[0135] In its implementation, the device can collect network status data from the underlying RDMA congestion control mechanism, RNIC counters, receiver queues, link telemetry systems, and wide-area path status monitoring systems. This data includes information such as receiver queue length, completion queue backlog, and available receiver buffers, and the device calculates receiver pressure based on this information. Simultaneously, the device can collect current round-trip time (RTT), historical baseline RTT, RTT growth trends, packet loss or retransmission risks, and link utilization for candidate transmission paths from the wide-area path status monitoring system, and calculate path pressure based on this information. Furthermore, the device can obtain the baseline rate fed back to the transmitter at the current moment from the underlying RDMA congestion control mechanism, and obtain the resource group rate limit of the queue group or flow group to which the semantic transmission unit belongs from the network interface card driver or hardware configuration.

[0136] Step S22: Determine the receiver security rate corresponding to the semantic transmission unit based on the receiver pressure, and determine the path security rate corresponding to the semantic transmission unit based on the path pressure.

[0137] It should be understood that the receiver safe rate can be the maximum receiving rate that the target receiver can safely withstand under the current receiver pressure. It can be derived from the receiver pressure and is used to limit the rate at which the sender sends data to the receiver in order to prevent the receiver buffer from overflowing or to handle congestion.

[0138] In this embodiment, the receiver's safe rate can be obtained by multiplying the receiver's maximum safe reception rate by the receiver's pressure margin. The corresponding calculation formula is as follows:

[0139] In the formula, Indicates the receiving end At any moment Safe receiving rate; Indicates the maximum safe reception rate at the receiving end; This indicates the receiver pressure. In practical applications, when the receiver pressure increases, the receiver's safe rate decreases accordingly; when the receiver pressure decreases, the receiver's safe rate increases accordingly.

[0140] It should also be understood that the path safety rate can be the maximum transmission rate that a candidate transmission path can safely carry under the current path pressure. It can be derived from the path pressure and is used to limit the rate at which data is sent through the path in order to prevent path congestion from worsening or packet loss.

[0141] In this embodiment, the path safety rate can be obtained by multiplying the maximum path safety bandwidth by the path pressure margin, and the corresponding calculation formula is as follows:

[0142] In the formula, Representing a path Safe speed; Indicates the maximum safe bandwidth of the path; This represents path pressure. In practical applications, when path pressure approaches 1, the path safe rate approaches 0; when path pressure approaches 0, the path safe rate approaches the maximum safe bandwidth.

[0143] Step S23: Determine the basic security rate of the semantic transmission unit based on the reference rate, the receiver security rate, the path security rate, and the resource group rate limit, and determine the basic security rate as the secure transmission rate of the semantic transmission unit.

[0144] It should be noted that the basic safe rate can be the maximum safe transmission rate allowed by the semantic transmission unit at the current moment, obtained by comprehensively considering underlying congestion control, receiver state, path state, and hardware resource constraints. This rate ensures that the semantic execution layer can only allocate rates within the safe transmission envelope allowed by the underlying congestion control. In this embodiment, the basic safe rate can be determined by the minimum value among the base rate, receiver safe rate, path safe rate, and resource group rate upper limit, specifically expressed as:

[0145] In the formula, Represents communication events At any moment The basic security rate; This represents the baseline rate given by the underlying RDMA congestion control. Indicates the target receiver Affordable receiver safety rate; Indicates candidate wide-area paths Path safety rate; This indicates the maximum speed limit of the resource group.

[0146] Further, the step of determining the semantic rate upper limit of the semantic transmission unit based on the semantic feature information of the semantic transmission unit includes: Step S31: Determine the semantic confidence, semantic marginal benefit, resource penalty value, risk penalty value, and historical service debt value corresponding to the service category of the semantic transmission unit based on the semantic feature information of the semantic transmission unit.

[0147] It should be noted that the resource penalty value can be a quantified penalty that a semantic transmission unit needs to bear for occupying specific network resources (such as receivers, transmission paths, queue groups, etc.). It can be used to reflect the resource contention cost incurred by the semantic transmission unit when using specific network resources. In this embodiment, the resource penalty value can be determined by a weighted sum of the semantic transmission unit's occupation of each network resource and the shadow price of each network resource, specifically expressed as: ,in, Represents a collection of resources. Semantic transmission unit Use resources (A value of 1 indicates use, and a value of 0 indicates no use). Representing resources At any moment The shadow price.

[0148] Among them, resource collection It can be defined as:

[0149] In the formula, Represents the set of resources at the receiving end; Represents a link, queue, or collection of rail resources; Represents a wide-area path resource set; This represents a QP group or stream group resource set.

[0150] In addition, resources Shadow price The corresponding calculation formula can be:

[0151] In the formula, Representing resources At any moment The shadow price; Indicates current period resources The actual load; Representing resources Safe capacity; Indicates the update step size; .

[0152] It should also be noted that when a receiver, path, or queue group is continuously overloaded, its shadow price increases, the resource penalty value of the communication unit using the resource increases accordingly, and the admission score decreases; when the resource becomes idle, the shadow price gradually decreases, and the resource penalty value decreases accordingly.

[0153] It should be noted that the risk penalty value can be a quantified value of the penalty that a semantic transmission unit needs to bear due to network transmission risks (such as receiver congestion, path congestion, PFC pause risk, retransmission risk, etc.). In this embodiment, the risk penalty value can be determined by the weighted sum of receiver pressure, path pressure, PFC risk, and retransmission risk, and can be specifically expressed as follows: ,in, Indicates the risk penalty coefficient. Indicates communication unit At any moment The network risk value.

[0154] Among them, network risk value It can be obtained by weighted summation of target receiver pressure, candidate path pressure, PFC risk, and retransmission risk. The corresponding calculation formula is as follows:

[0155] In the formula, Indicates communication unit At any moment Network risk value; Indicates the pressure at the target receiver; This indicates the pressure on candidate paths; Indicates PFC risk; This indicates a risk of retransmission or timeout. , , and This represents the risk weight.

[0156] It should be noted that the historical service deficit value corresponding to a service category can be the accumulated difference between the actual service volume obtained by a certain service category (such as critical, elastic, or back-end) over a period of time and the minimum service volume it should receive. It can be used to avoid service starvation caused by the long-term suppression of elastic and back-end communication by critical communication.

[0157] In this embodiment, the device can maintain a service debt value for each service category, wherein the formula for calculating the service debt value can be:

[0158] In the formula, Indicates service category At any moment The outstanding service debt; Indicates service category The minimum service rate; Indicates the control cycle; Indicates service category The actual volume of services received in the current period; service categories These can be critical, resilient, or backend classes. Among them, communication for critical classes is primarily driven by semantic marginal benefits. You can set it to a smaller value or 0; for elastic and backend classes, set it to a non-zero value. In order to avoid prolonged hunger.

[0159] Step S32: Determine the admission score of the semantic transmission unit based on the semantic confidence, the semantic marginal benefit, the resource penalty value, the risk penalty value, and the historical service arrears value. The admission score is used to characterize the priority of the communication event being served in the current control period.

[0160] It should be noted that the admission score can be a comprehensive score used to characterize the priority of a communication event in the current control cycle. It is the core basis for the semantic execution layer to allocate rates. The higher the admission score, the more worthy the communication event is of being prioritized for service in the current control cycle.

[0161] In this embodiment, the admission score can be determined jointly by semantic confidence, semantic marginal benefit, service debt compensation, resource penalty value, and risk penalty value, and the corresponding calculation formula can be:

[0162] In the formula, Indicates communication unit At any moment The admission score; Indicates semantic confidence; Indicates semantic marginal benefit; Indicate category Service debt weighting; This indicates that the service in this category is currently in arrears; Indicates communication unit Use resources ; Representing resources The shadow price; Indicates the network risk value; This represents the risk penalty coefficient.

[0163] The first part of the admission score Indicates semantic revenue and service debt compensation, Part Two Indicates resource congestion penalty, Part 3 This indicates penalties for cyber risks.

[0164] Step S33: Allocate semantic rate limits to each semantic transmission unit within the same resource contention group according to the admission score, wherein the sum of the semantic rate limits does not exceed the total security budget of the resource contention group.

[0165] Understandably, a resource contention group can be a competitive set of semantic transmission units (STUs) sharing the same network resources (e.g., the same receiver, the same transmission path, the same queue group, or the same resource group). Within this resource contention group, each STU needs to compete for a limited resource budget, so the device can allocate rates within the group based on admission scores. In practical applications, resource contention groups can be divided according to different resource dimensions. For example, based on the target receiver dimension, all STUs sending to the same receiver constitute a resource contention group; based on the candidate transmission path dimension, all STUs using the same path constitute a resource contention group; based on the queue group dimension, all STUs belonging to the same queue group constitute a resource contention group.

[0166] It is also understood that the total security budget can be the maximum total bandwidth that a resource contention group is allowed to use under current network conditions, and this total security budget is equal to the security capacity of the resource at the current moment.

[0167] In practical implementation, after determining the admission score of each semantic transmission unit, the device can allocate semantic rate caps to each semantic transmission unit within the same resource contention group according to the admission score. Specifically, the device can first determine the resource contention group. The scope, and to obtain the total security budget for the resource competition group. The total security budget equals the security capacity of the resource at the current moment. Next, the device sorts the semantic transmission units within the resource contention group according to their admission scores, with higher admission scores granting them a higher rate allocation ratio. Then, the softmax function is used to proportionally allocate the total security budget of the resource contention group according to the admission scores of each semantic transmission unit, determining the upper limit of the semantic rate for each semantic transmission unit. The corresponding calculation formula can be:

[0168] In the formula, This indicates that the semantic layer wants to provide communication units The upper limit of the rate; Represents the set of competing resources within the same resource group; This represents a temperature parameter used to control the smoothness of rate distribution; Indicates the competitive group The total security budget. When Smaller devices tend to favor high-scoring communication units, meaning the communication unit with the highest admission score receives the majority of the budget; when The larger capacity allows for smoother rate allocation, ensuring that each communication unit receives a certain proportion of the budget.

[0169] In this embodiment, the device can aggregate QPs with the same receiving end, the same service category, the same path, or the same resource contention relationship into a resource group, and set a group-level rate limit:

[0170] In the formula, Indicates resource group The upper limit of the rate; Indicates communication unit Belongs to QP group If the hardware or driver does not support QP group rate limiting, equivalent rate limiting can be achieved through the user-space communication library shim or software pacing.

[0171] In addition, the device can also handle wide-area paths. Set path budget:

[0172] In the formula, Representing a path The current available budget; Representing a path Maximum secure bandwidth; This represents path pressure. In practical applications, the higher the path pressure, the lower the path budget; after the path stabilizes, the budget gradually recovers, rather than immediately returning to its maximum value.

[0173] In practical applications, after determining the semantic rate upper limit, the device can determine the final transmission rate of the semantic transmission unit based on the secure transmission rate and the semantic rate upper limit. The corresponding formula can be expressed as:

[0174] In the formula, Indicates the final transmission rate; Indicates the safe transmission rate; Indicates the upper limit of semantic rate.

[0175] In this embodiment, semantic confidence, semantic marginal benefit, resource penalty value, risk penalty value, and historical service deficit value corresponding to the service category are determined based on the semantic feature information of the semantic transmission unit. Then, an admission score is determined based on this information. Finally, semantic rate caps are allocated to each semantic transmission unit within the same resource contention group according to the admission score. This ensures that the allocation of semantic rate caps simultaneously considers the marginal contribution of communication events to training progress, resource congestion, network transmission risks, and long-term fairness among service categories. This guarantees that critical communications with high semantic marginal benefits receive higher priority, while preventing low-priority service categories from becoming starved due to long-term bandwidth relinquishment. Furthermore, by limiting the sum of semantic rate caps to within the safe total budget of the resource contention group, semantic allocation is guaranteed not to exceed the safe capacity of the receiver and path. This allows for fine-grained rate allocation driven by training semantics while ensuring load balancing and long-term fairness within the resource contention group.

[0176] Further, the step of performing flow control on the semantic transmission unit according to the final transmission rate includes: Step S41: Determine the credit quantity of the target receiver corresponding to the sending end based on the final transmission rate and control period length. The credit quantity is used to control the quota of data fragments released by the sending end to the target receiver.

[0177] It should be understood that the control cycle length can be the time interval at which the device repeatedly executes flow control decisions, and it can be used to determine the update frequency of flow control decisions. In practical applications, within each control cycle, the device recalculates parameters such as the final transmission rate, credit count, and data fragment release interval for each semantic transmission unit, and performs flow control according to these parameters within the current cycle. In this embodiment, the control cycle length can be configured according to system requirements and network conditions, for example, it can be set to 1 millisecond, 5 milliseconds, or 10 milliseconds.

[0178] It should be noted that the credit limit can be the quota of data fragments that the sender is allowed to send to the target receiver within a control period. It can be used to control the number of data fragments that the sender releases to the target receiver, thereby limiting the rate at which the sender injects data into a specific receiver.

[0179] In this embodiment, the credit quantity can be calculated from the final transmission rate and the control period length, and the corresponding formula is:

[0180] In the formula, This indicates that the current period's distribution to the sender is... Receiver Queue Group The amount of credit; This indicates the final transmission rate corresponding to the sender, receiver, and queue group; Indicates the length of the control cycle; This represents the number of bytes corresponding to a credit (e.g., the size of a data slice or the number of bytes corresponding to a credit unit). This indicates rounding down. The sender can only release the corresponding number of data fragments after obtaining credit.

[0181] Step S42: Determine the release interval of adjacent data fragments based on the final transmission rate and the data fragment size of the semantic transmission unit.

[0182] Understandably, the data fragment size can be the data volume of each smallest transmission unit after the transmitted data corresponding to the communication event has been divided. This size can be determined by the aggregated communication library or RDMA transport layer based on hardware capabilities and transmission efficiency. Correspondingly, adjacent data fragments can be any two consecutive data fragments arranged in the transmission sequence within the same semantic transmission unit, according to their sending order. In practical applications, after dividing the total data volume of the communication event into multiple data fragments, the device can send the data fragments sequentially. The sending interval between adjacent data fragments is controlled by a release interval, which is calculated based on the final sending rate and the data fragment size.

[0183] It is also understood that the release interval can be the transmission time interval between adjacent data fragments, which can be used to control the transmission rhythm of data fragments so that the actual transmission rate matches the final transmission rate. In this embodiment, the release interval can be equal to the transmission time of each data fragment, that is, the data fragment size divided by the final transmission rate, and the corresponding formula can be:

[0184] in, Indicates communication unit At any moment The release interval of adjacent data fragments; Indicates the size of the data fragment; Indicates the final transmission rate; This represents a small constant to prevent division by zero. In practical applications, when the final transmission rate is high, the release interval is short, and data fragmentation is more concentrated; when the final transmission rate is low, the release interval is long, and data fragmentation is more sparse.

[0185] Step S43: Release the data fragments according to the credit quantity, and send each data fragment sequentially according to the release interval to perform flow control on the semantic transmission unit.

[0186] In its implementation, after determining the credit quantity and release interval, the device can release data fragments according to the credit quantity and send each data fragment sequentially according to the release interval to perform flow control on the semantic transmission unit. Specifically, the device can first grant the sender an upper limit on the number of data fragments that can be released within the current control period according to the determined credit quantity. After obtaining the credits, the sender gains the right to release the corresponding number of data fragments. Next, the device can control the sender to release and send each data fragment sequentially according to the determined release interval. For multiple data fragments of the same semantic transmission unit, after sending one data fragment, the sender can wait for one release interval before sending the next data fragment. Subsequently, the device can repeat the above process until all data fragments of the semantic transmission unit have been sent or the current control period ends. In this embodiment, the credit quantity and release interval jointly realize the flow control of the semantic transmission unit. The credit quantity limits the upper limit of the number of data fragments that the sender can send to the target receiver within the current control period, while the release interval controls the sending rhythm of data fragments. By combining the amount of credits and the release interval, the actual sending rate is matched with the final sending rate.

[0187] In this embodiment, the credit quantity is determined based on the final transmission rate and control period length, and the release interval of adjacent data fragments is determined based on the final transmission rate and data fragment size. Then, data fragments are released according to the credit quantity and sent sequentially according to the release interval. This ensures that the actual transmission behavior of the sender is subject to the dual constraints of credit quota and release interval. This not only ensures that the actual transmission rate matches the final transmission rate, but also effectively suppresses multiple senders from simultaneously injecting data into the same target receiver. This avoids problems such as receiver buffer overflow, PFC pause, or packet loss, and achieves accurate conversion and stability protection from semantic rate decision to RDMA runtime actions.

[0188] Furthermore, after the step of performing flow control on the semantic transmission unit according to the final transmission rate, the method further includes: Step S51: Obtain the data integrity of the target semantic confidence of all currently active semantic transmission units in the current training task, the current network status data of the wide area transmission network, and the network telemetry data.

[0189] It should be understood that the currently active semantic transmission unit can be a semantic transmission unit that has not yet completed all data transmission within the current control cycle and is still in an active state; correspondingly, the target semantic confidence score can be the semantic confidence score corresponding to the currently active semantic transmission unit. In practical applications, when performing anomaly detection, the device can obtain relevant information about all currently active semantic transmission units, including the semantic confidence score, service category, remaining data volume, and estimated remaining completion time of each semantic transmission unit. Furthermore, by statistically analyzing the semantic confidence scores of all currently active semantic transmission units, the reliability of the overall semantic recognition quality can be evaluated. The target semantic confidence score ranges from 0 to 1; a higher value indicates a more reliable semantic recognition result for the communication event.

[0190] It should also be understood that the current network status data can be parameters used to reflect the operating status of the wide area transmission network at the current moment. This can include information such as the current round-trip time (RTT) of candidate transmission paths, historical baseline RTT, RTT growth trend, number of congestion notification messages, duration or number of priority flow control pauses, number of retransmissions or timeouts, packet loss rate, ECN marking ratio, link utilization, and path switching events. In this embodiment, the device can detect whether network anomalies have occurred based on the current network status data and trigger corresponding fallback strategies according to the degree of anomaly.

[0191] Understandably, network telemetry data can be various types of telemetry information collected by devices from the wide area network, including but not limited to receiver queue length, completion queue backlog, receiver available buffer, path bandwidth, path utilization, PFC pause information, CNP count, ECN-tagged message count, RTT sample value, repeat count, timeout statistics, etc. In practical applications, network telemetry data can be reported by network devices (such as switches and routers) through telemetry protocols, provided by RNIC hardware counters, or collected by active probing mechanisms.

[0192] It is also understandable that data integrity can be an indicator used to characterize whether network telemetry data is complete and without missing parts during acquisition and transmission. It can be used to measure whether the acquired network telemetry data is sufficient to support accurate network status assessment and anomaly detection. The value of data integrity ranges from 0 to 1, with a higher value indicating more complete network telemetry data. For example, if all telemetry data in the current control cycle is successfully acquired, the data integrity is 1.0; if some telemetry data is lost, the data integrity value decreases to 0.7.

[0193] Step S52: Determine the average semantic confidence based on the target semantic confidence of each target.

[0194] It should be understood that the average semantic confidence score can be the arithmetic mean of the target semantic confidence scores of all currently active semantic transmission units, which can be used to evaluate the reliability of the current overall semantic recognition quality. In this embodiment, the calculation formula for the average semantic confidence score can be:

[0195] In the formula, Indicates time The average semantic confidence; Represents the set of currently active semantic transmission units; This indicates the number of active semantic transmission units in the set; Indicates communication unit The semantic confidence level. In practical applications, when the average semantic confidence level is lower than a preset threshold, it indicates that the overall semantic recognition quality is low and needs to enter a semantic degradation state.

[0196] Step S53: Determine the current system state based on the average semantic confidence, the current network state data, and the data integrity. The current system state includes semantic degradation state, network protection state, and complete rollback state.

[0197] It should be noted that the current system state can be the operating state of the device at the current moment, which can be used to determine what control strategy the device is currently employing. In this embodiment, the current system state can include a normal state, a semantic degradation state, a network protection state, and a complete rollback state, which can be represented as follows:

[0198] in, Indicates the current system status. This indicates the normal semantic access and rate-limiting state, i.e., the normal state; This indicates a semantic degradation state, in which only three types of coarse-grained control are used: critical classes, elastic classes, and backend classes. This indicates a network protection state, at which point the device tightens receiver credit and path budget; This indicates a complete rollback state, where the device suspends semantic control and resumes basic RDMA behavior.

[0199] It should also be noted that the device can comprehensively determine its current operating state based on factors such as average semantic confidence, current network status data, and data integrity, and execute corresponding control strategies. Furthermore, the device can switch between different states to adapt to dynamic changes in the training quality of semantic recognition and network conditions.

[0200] It should be noted that semantic degradation state refers to a working mode where, when semantic recognition quality is insufficient, the device stops fine-grained rate control based on individual semantic transmission units and switches to coarse-grained rate control based on service categories (such as critical, resilient, and backend categories). In semantic degradation state, the device no longer relies on detailed semantic parameters of individual communication events (such as semantic marginal benefits, slackness, semantic loss, etc.) for rate allocation, but only performs category-level rate control based on the service category to which the communication event belongs, in order to ensure the stability and security of the flow control method when semantic confidence is insufficient.

[0201] It should also be noted that network protection mode refers to the operating mode in which the device tightens credit issuance at the receiver, reduces path budget, and compresses rates for resilient and background classes when network congestion or anomalies occur. In network protection mode, the device can prioritize protecting critical path communications while reducing network pressure, thereby quickly reducing network load during network anomalies and preventing congestion propagation and network performance degradation. Specifically, the device can reduce the credit issuance rate, decreasing the amount of data injected into the receiver; compress the available budget for transmission paths, reducing the transmission rate through congested paths; and prioritize compressing rates for resilient and background class communications to protect critical class communications from impact.

[0202] It should be understood that a full rollback state can refer to a situation where, when network anomalies persist for an extended period or telemetry data is missing for an extended period, the device suspends all semantic control logic and reverts to a basic RDMA transmission behavior mode. In a full rollback state, the device no longer performs any semantic-based admission control, rate allocation, or priority ordering. Instead, it completely relinquishes control to the underlying RDMA congestion control mechanism, which independently determines the transmission rate. This ensures that communication within the distributed training system can continue without complete interruption when reliable semantic or network status information is unavailable.

[0203] In practical implementation, after determining the average semantic confidence score, the device can determine the current system state based on the average semantic confidence score, current network state data, and data integrity. Specifically, the device can first detect semantic anomalies and determine whether the average semantic confidence score is lower than the minimum semantic confidence threshold, i.e.

[0204] In the formula, This represents the average semantic confidence level. This represents the minimum semantic confidence threshold.

[0205] If the average semantic confidence is lower than the minimum semantic confidence threshold, and the low confidence state continues for M control cycles, then it can be determined that a semantic degradation state is required.

[0206] Simultaneously, the device can detect abnormal network conditions and determine whether various indicators in the current network status data exceed preset thresholds. Specifically, the device can determine whether the current round-trip time of the path exceeds the baseline round-trip time. times, that is:

[0207] And determine whether the number of congestion notification messages exceeds the threshold. ,Right now:

[0208] At the same time, the device can determine whether the duration or number of priority flow control pauses exceeds the threshold. ,Right now:

[0209] In addition, the device can also determine whether the number of retransmissions or timeouts exceeds a threshold. ,Right now:

[0210] In the formula, Representing a path Current RTT; Representing a path Historical benchmark RTT; Indicates the RTT amplification threshold; Representing a path The number of CNPs on; Indicates the CNP threshold; Indicates the duration or number of PFC pauses; Indicates the PFC threshold; Indicates the number of retransmissions or timeouts; This indicates the retransmission threshold.

[0211] When any of the above conditions are met, the device can determine that there is a network anomaly and needs to enter network protection mode.

[0212] In addition, the device can detect telemetry anomalies and determine the integrity of network telemetry data. Does the number of missing records consistently exceed a set threshold? When critical telemetry data is missing or expires for more than [time period], If this occurs, a telemetry anomaly can be identified. The device can then determine the current system status based on the anomaly detection results. If the telemetry anomaly conditions are met or the network anomaly persists for an extended period... Then, the current system state can be determined to be a complete rollback state, and the corresponding judgment condition can be expressed as:

[0213] or:

[0214] in, This indicates the duration for which the system remains in network protection mode. Indicates the maximum network protection duration for entering full rollback; Indicates the duration of critical telemetry loss; This indicates the maximum permissible telemetry missing time.

[0215] When any of the above conditions are met, the device can determine that the current system state is a full rollback state. Otherwise, if the network anomaly condition is met, the current system state can be determined to be a network protection state; if the semantic anomaly condition is met, the current system state can be determined to be a semantic degradation state. In addition, if the device is currently in a rollback state and the anomaly has disappeared beyond the cooldown recovery time, it can gradually restore the system to a normal state.

[0216] Step S54: Determine the abnormal rollback strategy based on the current system state, and make abnormal adjustments based on the abnormal rollback strategy.

[0217] Understandably, an anomaly fallback strategy can be a set of specific control measures determined based on the current system state to address anomalies, specifying the specific operations to be performed under different system states. In this embodiment, the anomaly fallback strategy may include: in a semantic degradation state, stopping fine-grained semantic control and switching to coarse-grained control based on service category; in a network protection state, reducing the credit issuance rate, compressing the available transmission path budget, and prioritizing the compression of rates for elastic and backend communication; in a full fallback state, suspending all semantic control logic and resuming basic RDMA transmission behavior. In practical applications, the device can select a matching anomaly fallback strategy based on the current system state and execute the control measures specified by that strategy. For example, if the current system state is network protection state, the anomaly fallback strategy is to "reduce the credit issuance rate to 50% of the normal value and compress the rate cap for elastic and backend communication to 30% of the normal value."

[0218] Subsequently, when the anomaly disappears, the device does not immediately restore full semantic control, but instead uses a recovery coefficient. Gradual recovery is used to avoid rate spikes and oscillations caused by switching directly back to semantic control from the fallback state. This can be represented as follows:

[0219] In the formula, This indicates the actual rate during the cooling recovery phase; This represents the coefficient of restitution, with a value range of 100%. ; This represents the final rate obtained from semantic admission computation. Indicates the base security rate. During initial recovery. , and then gradually increased to 1.

[0220] In this embodiment, by acquiring the target semantic confidence scores of all currently active semantic transmission units, the current network status data of the wide area transmission network, and the data integrity of network telemetry data, the average semantic confidence score is determined based on each target semantic confidence score. The current system state is then determined based on the average semantic confidence score, the current network status data, and the data integrity. Finally, an anomaly fallback strategy is determined based on the current system state, and anomaly adjustments are made based on this strategy. This allows for automatic switching to a semantic degradation state, a network protection state, or a complete fallback state based on different degrees of anomalies in semantic confidence scores and network states. After the anomaly disappears, normal semantic control is gradually switched back through cooling recovery. This enables the automatic reduction of dependence on semantic information and tightening of the transmission rate to protect network stability in abnormal scenarios such as unreliable semantic recognition, network congestion, or missing telemetry. At the same time, semantic control is smoothly restored after the anomaly disappears to avoid rate spikes and oscillations. Thus, communication efficiency can be improved by utilizing training semantic information while ensuring system security.

[0221] In the specific implementation, refer to Figure 4 , Figure 4 This is a flowchart of the anomaly detection and rollback process in the flow control method of this application. Figure 4As shown, the anomaly detection and rollback process in this embodiment can include three stages: anomaly detection, degradation mode, and rollback and recovery. In the anomaly detection stage, the device can determine whether the network telemetry data is complete and whether data loss has timed out. When critical telemetry data is detected to be missing or expired beyond a preset time threshold, anomalies in the telemetry data are determined. Simultaneously, the device can acquire the current round-trip time (RTT) and historical baseline RTT for each candidate transmission path to calculate the amplification ratio of the current RTT relative to the historical baseline RTT. If the amplification ratio exceeds the RTT amplification threshold, an RTT anomaly is determined. Furthermore, the device can acquire the receiver queue length and completion queue backlog to determine if the receiver is experiencing a continuous increase in queue depth due to congestion or frequent PFC pause frame transmission. When the receiver queue depth continuously increases and approaches the maximum safe queue length, or the PFC pause duration exceeds a threshold, queue pressure anomalies are determined. The device can continuously execute the above anomaly detection operations, triggering the corresponding degradation mode when any anomaly condition is met. In the degradation mode stage, the device can determine the current system state based on the anomaly detection results and enter the corresponding degradation mode. Specifically, when a device detects a semantic anomaly, it can enter a semantic degradation state. In this state, the device no longer uses fine-grained semantic information corresponding to a single semantic transmission unit for rate allocation. Instead, it switches to a coarse-grained control mode based on service category, prioritizing and allocating rates only according to the service category (critical, resilient, or backend) to which the communication event belongs. When a device detects a network anomaly, it enters a network protection state. In this state, the device can tighten credit issuance at the receiver, reduce the quota of data fragments injected by the sender to the target receiver, and slow down the release rate of data fragments to reduce network pressure. When the device detects a timeout for missing telemetry data or a network anomaly that persists for more than the maximum network protection duration, it enters a full rollback state. In this state, the device can suspend all semantic control logic, no longer executing any semantic-based admission control, rate allocation, or priority ranking. Instead, it reverts to basic RDMA transmission behavior, with the underlying RDMA congestion control mechanism determining the transmission rate independently. Subsequently, if the anomaly disappears, the device does not immediately resume full semantic control but enters a cooling-off recovery period. During this period, the device can use a recovery coefficient to control the transition from the fallback state to the normal state. Initially, during recovery, the device transmits entirely at the basic safe rate, i.e., basic RDMA behavior. As time progresses, the device can gradually increase the value of the recovery coefficient until it fully restores to the complete semantic admission and rate limiting control of the normal state. This cooling-off recovery mechanism avoids the rate surge and control oscillations caused by directly switching back to semantic control from the fallback state, ensuring smooth state transitions and system stability.

[0222] It should be noted that, referring to Figure 5 , Figure 5 This is a flowchart illustrating the overall flow control method of this application. Figure 5 As shown, the flow control method in this application can be divided into three stages: a semantic modeling stage, a service computation stage, and a congestion control stage. These three stages are executed sequentially, forming a complete closed-loop control process from semantic perception to admission decision and then to secure execution. In the semantic modeling stage, the device can perform semantic information collection and modeling operations. Specifically, the device can first obtain training semantic data such as the training stage, iteration number, and dependencies of the current training task from the runtime hooks of the training framework. Simultaneously, it can obtain network transmission attributes such as communication type, target receiver, candidate path, and remaining bytes from the communication event monitoring interface of the collection communication library. Furthermore, it can obtain network status data such as receiver pressure, path pressure, baseline rate from the underlying congestion control feedback, and resource group rate limit from the wide area network status perception module. Subsequently, the device can fuse the training semantic data and network transmission attributes to generate semantic transmission units corresponding to communication events. Based on the training stage, communication type, and dependencies in the semantic transmission units, the communication events are divided into different service categories, including critical, resilient, and background categories, and the deadlines for different categories are calculated. During the service calculation phase, the device can perform admission score calculation and service debt update operations. Specifically, the device first obtains the minimum service rate set for critical, resilient, and backend classes, as well as the actual service volume obtained by each class in the current control period. Based on the difference between the minimum service rate and the actual service volume obtained for each class, the device updates the historical service debt value for each service class. Then, it calculates the admission score for each semantic transmission unit based on semantic marginal benefits, service debt compensation, resource penalty value, and risk penalty value. Finally, the admission score of each semantic transmission unit is used as the quantitative basis for the priority of the communication event being served in the current control period. During the congestion control phase, the device can perform rate constraint, admission enforcement, and anomaly fallback operations. Specifically, the device first detects whether the semantic confidence is lower than the threshold, whether there are anomalies in the network status (e.g., abnormal increase in RTT, surge in CNP, PFC pause duration exceeding the threshold), and whether network telemetry data is missing. When an anomaly is detected, the device can execute the corresponding fallback strategy according to the anomaly type, including entering semantic degradation state, network protection state, or complete fallback state.

[0223] It should also be noted that, referring to Figure 6 , Figure 6 A system block diagram provided for the flow control method of this application. (e.g.) Figure 6As shown, the flow control system in this application may include a training-side semantic modeling module, a network state awareness module, a semantic admission decision module, a security execution module, and an anomaly detection and rollback module. These modules cooperate to form a complete flow control link from semantic awareness to security execution. In the semantic abstraction and modeling stage, the training-side semantic modeling module can obtain information such as the training stage, iteration number, and dependencies of the current training task from the training framework. Simultaneously, it can obtain network transmission attributes such as the communication type, target receiver, queue group information, candidate path, and remaining bytes of communication events from the communication library. The module then fuses the obtained training semantic data and network transmission attributes to generate a semantic RDMA unit stream. This semantic RDMA unit stream can serve as input data for subsequent modules, carrying all semantic and network information used for flow control decisions. In the network state awareness stage, the network state awareness module can obtain network state information from the wide-area RDMA network, smooth the obtained network state information to remove instantaneous noise, calculate the pressure indicators of each resource, and then pass the calculated basic security rate and various network pressure indicators to the semantic admission decision module and the security execution module. During the semantic admission decision phase, the semantic admission decision module receives semantic RDMA unit streams from the semantic modeling module on the training side and network state information from the network state awareness module to perform admission priority calculation. Specifically, the semantic admission decision module first calculates the admission priority (i.e., admission score) for each semantic transmission unit, sorts the semantic transmission units according to the admission score, determines the semantic rate cap for each semantic transmission unit based on the admission score, and passes the semantic rate cap to the security execution module. During the security execution phase, the security execution module receives the semantic rate cap from the semantic admission decision module and the basic security rate from the network state awareness module, and performs security trade-offs and alignment operations. Specifically, the security execution module determines the number of credits for the sender corresponding to the target receiver based on the final transmission rate and control period length, and allocates the credits to the sender. Simultaneously, the security execution module determines the release interval of adjacent data fragments based on the final transmission rate and data fragment size, and releases each data fragment sequentially according to the release interval. Furthermore, the security execution module can accurately translate the rate decisions received from the semantic admission decision module into executable actions during RDMA runtime through multi-level control methods such as group token bucket rate limiting, queue group rate limiting, receiver credit control, and release intervals. During the anomaly detection and rollback phase, the anomaly detection and rollback module continuously monitors the system's operating status and triggers rollback control when an anomaly is detected. Subsequently, if the anomaly disappears, the anomaly detection and rollback module can perform a cooling recovery operation, gradually increasing the recovery coefficient to smoothly transition the device from the rollback state back to the normal semantic control state.

[0224] In this embodiment, the receiver pressure of the target receiver, the path pressure of the candidate transmission path, the reference rate of the underlying congestion control feedback, and the upper limit of the resource group rate are determined based on the network state data corresponding to the wide area transmission network. Then, the receiver safety rate is determined based on the receiver pressure, and the path safety rate is determined based on the path pressure. Finally, the minimum value among the reference rate, receiver safety rate, path safety rate, and upper limit of the resource group rate is determined as the base safety rate as the safe transmission rate of the semantic transmission unit. This allows the final transmission rate to be limited to the minimum value of the reference rate, receiver safety rate, path safety rate, and upper limit of the resource group rate. As a result, the semantic execution layer can only allocate rates within the safe transmission envelope allowed by the underlying congestion control mechanism, receiver state, path state, and hardware resources. This can improve communication efficiency while ensuring the stability and security of the wide area RDMA network.

[0225] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the flow control method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0226] This application also provides a flow control device, please refer to... Figure 7 The flow control device includes: The semantic unit construction module 10 is used to acquire communication events generated during the training process of the current training task and construct semantic transmission units corresponding to the communication events. The semantic transmission units are used to describe the runtime characteristics of the communication events in the current training task and the network characteristics during the data transmission process. The secure rate determination module 20 is used to determine the secure transmission rate of the semantic transmission unit based on the network status data corresponding to the wide area transmission network. The rate limit determination module 30 is used to determine the semantic rate limit of the semantic transmission unit based on the semantic feature information of the semantic transmission unit, wherein the semantic feature information is used to characterize the urgency of the communication event during the training process. The flow control module 40 is used to determine the final transmission rate of the semantic transmission unit based on the secure transmission rate and the semantic rate limit, and to perform flow control on the semantic transmission unit according to the final transmission rate.

[0227] The flow control device provided in this application, employing the flow control method described in the above embodiments, can solve the technical problem that existing flow control methods cannot adapt to the dynamic changes in communication urgency during large model training, leading to insufficient accuracy in flow control. Compared with the prior art, the beneficial effects of the flow control device provided in this application are the same as those of the flow control method provided in the above embodiments, and other technical features of the flow control device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0228] This application provides a flow control device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the flow control method in Embodiment 1 above.

[0229] The following is for reference. Figure 8 The diagram illustrates a structural schematic of a flow control device suitable for implementing embodiments of this application. The flow control device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The flow control device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0230] like Figure 8As shown, the flow control device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the flow control device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the flow control device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows flow control devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0231] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0232] The flow control device provided in this application, employing the flow control method in the above embodiments, can solve the technical problem of flow control. Compared with the prior art, the beneficial effects of the flow control device provided in this application are the same as those of the flow control method provided in the above embodiments, and other technical features of the flow control device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0233] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0234] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0235] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the flow control method in the above embodiments.

[0236] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0237] The aforementioned computer-readable storage medium may be included in the flow control device; or it may exist independently and not be assembled into the flow control device.

[0238] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the flow control device, the flow control device: acquires communication events generated during the training process of the current training task, and constructs semantic transmission units corresponding to the communication events. The semantic transmission units are used to describe the runtime characteristics of the communication events in the current training task and the network characteristics during data transmission. The flow control device determines the safe transmission rate of the semantic transmission units based on the network state data corresponding to the wide area transmission network. The flow control device determines the upper limit of the semantic rate of the semantic transmission units based on the semantic feature information of the semantic transmission units. The semantic feature information is used to characterize the urgency of the communication events during the training process. The flow control device determines the final transmission rate of the semantic transmission units based on the safe transmission rate and the upper limit of the semantic rate, and performs flow control on the semantic transmission units according to the final transmission rate.

[0239] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0240] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0241] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0242] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described flow control method. This solves the technical problem that existing flow control methods cannot adapt to the dynamic changes in communication urgency during large model training, leading to insufficient accuracy in flow control. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the flow control method provided in the above embodiments, and will not be repeated here.

[0243] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A flow control method, characterized in that, The method includes: The communication events generated during the training process of the current training task are obtained, and the semantic transmission unit corresponding to the communication event is constructed. The semantic transmission unit is used to describe the runtime characteristics of the communication event in the current training task and the network characteristics during the data transmission process. The semantic transmission unit is constructed based on the training stage, communication type, service category, estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal benefit and semantic confidence of the communication event. The secure transmission rate of the semantic transmission unit is determined based on the network status data corresponding to the wide area transmission network. The semantic rate limit of the semantic transmission unit is determined based on the semantic feature information of the semantic transmission unit, and the semantic feature information is used to characterize the urgency of the communication event during the training process. The final transmission rate of the semantic transmission unit is determined based on the secure transmission rate and the upper limit of the semantic rate, and flow control of the semantic transmission unit is performed according to the final transmission rate.

2. The method as described in claim 1, characterized in that, The step of constructing the semantic transmission unit corresponding to the communication event includes: Obtain the training semantic data and network transmission attributes of communication events generated during the training process of the current training task; The training phase, communication type, and service category of the communication event are determined based on the training semantic data and the network transmission attributes. Determine the estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal gain, and semantic confidence for the communication event; Based on the training phase, the communication type, the service category, the estimated remaining completion time, the semantic deadline, the slackness, the semantic loss, the semantic marginal gain, and the semantic confidence, a semantic transmission unit corresponding to the communication event is constructed.

3. The method as described in claim 2, characterized in that, The step of determining the estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal gain, and semantic confidence corresponding to the communication event includes: The estimated remaining completion time of the communication event is determined based on the remaining data volume of the communication event and the historical completion rate estimate. The semantic deadline of the communication event is determined based on the dependencies between the training phase, the communication type, and the current training task. The slack of the communication event is determined based on the semantic deadline, the current time, and the estimated remaining completion time; The semantic loss of the communication event is determined based on the relaxation degree and the weight of the training progress of the communication event. The semantic marginal benefit of the communication event is determined based on the semantic loss and the amount of candidate service data for the communication event; The semantic confidence of the communication event is determined based on the first completeness of the training semantic data, the second completeness of the network transmission attributes, and the credibility of historical statistical information.

4. The method as described in claim 1, characterized in that, The step of determining the secure transmission rate of the semantic transmission unit based on the network status data corresponding to the wide area transmission network includes: Based on the network status data corresponding to the wide area transmission network, determine the receiver pressure of the target receiver, the path pressure of the candidate transmission path, the baseline rate of the underlying congestion control feedback, and the upper limit of the resource group rate corresponding to the semantic transmission unit. The upper limit of the resource group rate is the hardware configurable rate upper limit of the queue group or the flow group. The receiver security rate corresponding to the semantic transmission unit is determined based on the receiver pressure, and the path security rate corresponding to the semantic transmission unit is determined based on the path pressure. The basic security rate of the semantic transmission unit is determined based on the reference rate, the receiver security rate, the path security rate, and the resource group rate limit, and the basic security rate is determined as the secure transmission rate of the semantic transmission unit.

5. The method as described in claim 1, characterized in that, The step of determining the semantic rate upper limit of the semantic transmission unit based on the semantic feature information of the semantic transmission unit includes: Based on the semantic feature information of the semantic transmission unit, determine the semantic confidence, semantic marginal benefit, resource penalty value, risk penalty value, and historical service debt value corresponding to the service category of the semantic transmission unit. The admission score of the semantic transmission unit is determined based on the semantic confidence, the semantic marginal benefit, the resource penalty value, the risk penalty value, and the historical service arrears value. The admission score is used to characterize the priority of the communication event being served in the current control period. Each semantic transmission unit within the same resource contention group is assigned a semantic rate cap according to the admission score, wherein the sum of the semantic rate caps does not exceed the total security budget of the resource contention group.

6. The method as described in claim 1, characterized in that, The step of performing flow control on the semantic transmission unit according to the final transmission rate includes: The credit quantity of the target receiver corresponding to the sending end is determined based on the final transmission rate and the control period length. The credit quantity is used to control the quota of data fragments released by the sending end to the target receiver. The release interval of adjacent data fragments is determined based on the final transmission rate and the data fragment size of the semantic transmission unit; The data fragments are released according to the credit amount, and each data fragment is sent sequentially according to the release interval to perform flow control on the semantic transmission unit.

7. The method according to any one of claims 1 to 6, characterized in that, After the step of performing flow control on the semantic transmission unit according to the final transmission rate, the method further includes: The data integrity of the target semantic confidence scores of all currently active semantic transmission units in the current training task, the current network status data of the wide area transmission network, and the network telemetry data is obtained. The average semantic confidence score is determined based on the target semantic confidence scores. The current system state is determined based on the average semantic confidence, the current network state data, and the data integrity. The current system state includes semantic degradation state, network protection state, and complete rollback state. Determine an abnormal rollback strategy based on the current system state, and make abnormal adjustments based on the abnormal rollback strategy.

8. A flow control device, characterized in that, The device includes: The semantic unit construction module is used to acquire communication events generated during the training process of the current training task and construct semantic transmission units corresponding to the communication events. The semantic transmission units are used to describe the runtime characteristics of the communication events in the current training task and the network characteristics during the data transmission process. The semantic transmission units are constructed based on the training stage, communication type, service category, estimated remaining completion time, semantic deadline, slack, semantic loss, semantic marginal benefit and semantic confidence of the communication events. The secure rate determination module is used to determine the secure transmission rate of the semantic transmission unit based on the network status data corresponding to the wide area transmission network. The rate limit determination module is used to determine the semantic rate limit of the semantic transmission unit based on the semantic feature information of the semantic transmission unit, wherein the semantic feature information is used to characterize the urgency of the communication event during the training process. The flow control module is used to determine the final transmission rate of the semantic transmission unit based on the secure transmission rate and the semantic rate limit, and to perform flow control on the semantic transmission unit according to the final transmission rate.

9. A flow control device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the flow control method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the flow control method as described in any one of claims 1 to 7.

Images