Intelligent computing cloud platform based on set communication mode computing power network traffic dynamic scheduling method and device

Abstract

The application provides a computing power network traffic dynamic scheduling method and device based on a set communication mode of an intelligent computing cloud platform, relates to the fields of intelligent computing centers, intelligent computing centers, computing power infrastructure and intelligent computing cloud technology, and comprises the following steps: S1, performing semantic identification on traffic generated by computing power running tasks of the intelligent computing cloud platform; S2, mapping the traffic to corresponding quality of service queues according to the semantic identification result and maintaining a traffic occupation list between each physical path and different traffic types; S3, determining a communication ring according to parallel topology information of a to-be-scheduled computing power running task; S4, selecting a physical path for communication transmission for the communication ring according to communication ring participant nodes and the traffic occupation list; and S5, monitoring communication completion times of each transmission section on the communication ring and adjusting the arrangement order of nodes in the communication ring to bypass an abnormal transmission section when the section is abnormal. The application can effectively alleviate congestion problems caused by unordered competition of traffic.

Description

Technical Field

[0001] This invention relates to the fields of intelligent computing centers, smart computing centers, computing infrastructure, and smart cloud technologies, specifically to a method and apparatus for dynamic scheduling of computing network traffic based on a collective communication mode in an intelligent computing cloud platform. Background Technology

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "smart computing centers" have emerged.

[0003] An "intelligent computing center" refers to a facility that provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models) by utilizing large-scale heterogeneous computing resources, including general-purpose and intelligent computing power. Intelligent computing centers encompass facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0004] "Intelligent computing center" includes, but is not limited to, "intelligent computing center".

[0005] "Intelligent computing center" or artificial intelligence computing center is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting artificial intelligence computing architecture.

[0006] "Computing power" is the core of "intelligent computing center" and "smart computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to execute a certain computing requirement. It is the computing power to achieve the target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] With the widespread adoption of large-scale models, the computing power demand of intelligent computing cloud platforms is growing exponentially. In large-scale distributed computing tasks, hundreds or thousands of computing accelerator cards need to frequently perform aggregate communication operations such as All Reduce to synchronize computation results. To support massive computing power interconnection, intelligent computing cloud platforms generally adopt lossless networks such as RDMA over Converged Ethernet (RoCE) or InfiniBand to support zero-packet-loss transmission of underlying computing power data. For multi-tenant shared network environments, existing resource isolation mostly adopts static rate limiting strategies based on tenant dimensions. This mechanism only performs coarse-grained management based on tenant identity and fails to identify the semantic nature of traffic. However, the traffic generated by computing power tasks of the same tenant has significantly different sensitivities to latency. Aggregate communication operations are usually completed collaboratively by a communication ring composed of all computing power nodes and are synchronization points on the critical path of computing power iteration. Delay in any network segment will cause overall computing power blockage. During computing power iteration computation, the checkpoint upload task is executed synchronously, which will cause computing power iteration computation to pause. After the checkpoint upload is completed, the blockage will end and the operation will continue, thus reducing training efficiency. Current technologies cannot distinguish between the aforementioned traffic semantics. When both types of traffic occur concurrently, elastic traffic may crowd out the bandwidth required for aggregated communication, causing "slow flow" with abnormal latency in the intelligent computing cloud platform, resulting in a significant decrease in overall computing power throughput. In addition, current technologies lack the ability to detect the superimposed congestion on the physical path of communication rings for multiple computing power running tasks; and when abnormal latency occurs in the underlying physical link, the system cannot proactively adjust the computing power communication ring topology to bypass the damaged link.

[0008] It is evident that since the emergence of intelligent computing centers, there has been a lack of intelligent scheduling schemes for computing network traffic that can jointly reconstruct the data plane perception and control plane. This has resulted in the inability to solve the congestion problem caused by disorderly traffic competition, which has become a key bottleneck restricting the high availability of intelligent computing cloud platform networks and the efficiency of aggregated communication transmission. This problem is an urgent issue to be solved in this field. Summary of the Invention

[0009] This invention provides a method and apparatus for dynamic scheduling of computing network traffic based on a collective communication mode in an intelligent computing cloud platform, in order to solve the congestion problem caused by disorderly competition of traffic in the prior art.

[0010] To solve the above problems, the present invention is implemented as follows: In a first aspect, the present invention provides a method for dynamic scheduling of computing network traffic based on a collective communication mode in an intelligent computing cloud platform, comprising: Step S1: Perform semantic recognition on the traffic generated by the intelligent computing cloud platform during the execution of computing power tasks; Step S2: Based on the semantic recognition results, map the traffic to the service quality queue corresponding to the semantic category, and based on the service quality queue, maintain a traffic occupancy list between each physical path and different traffic types; Step S3: Obtain the parallel topology information of the task to be scheduled by computing power, and determine the communication ring of the task to be scheduled by computing power when performing aggregate communication operation based on the parallel topology information, wherein the communication ring includes multiple participating nodes; Step S4: Based on the multiple participating nodes and the traffic occupancy list, select a physical path that meets the preset occupancy conditions for communication transmission for the communication ring corresponding to the computing power running task to be scheduled; Step S5: Continuously monitor the communication completion time of each transmission segment on the communication ring, and when it is determined that there is an abnormal transmission segment in the communication ring based on the communication completion time, adjust the node arrangement order in the communication ring to bypass the abnormal transmission segment, wherein the transmission segment consists of at least two participating nodes.

[0011] In one embodiment, step S1 includes at least one of the following: Step S1.1: Intercept the calling behavior of the computing power running task in the collection communication operation, and identify the traffic generated by the calling behavior as synchronization traffic as the first identification result; Step S1.2: Perform semantic classification and identification on the message arrival time interval and autocorrelation characteristics of the traffic of the computing power running task within the sliding time window to determine the second identification result.

[0012] In one embodiment, step S1.2 includes: Step S1.2.1: Aggregate the traffic into independent flows according to the packet 5-tuple, and count the packet arrival time interval sequence of the flow within the sliding time window; Step S1.2.2: Calculate the coefficient of variation of the message arrival time interval sequence and detect the autocorrelation peak of the message arrival time interval sequence at a preset lag. Step S1.2.3: If the coefficient of variation is less than a first preset threshold and the autocorrelation peak value is greater than a second preset threshold, then the semantic category of the flow corresponding to the traffic is determined to be synchronization traffic; Step S1.2.4: If the coefficient of variation is greater than or equal to the first preset threshold, and / or the autocorrelation peak value is less than or equal to the second preset threshold, then the traffic corresponding to the flow is determined to be elastic traffic or management traffic by combining the port range and historical tags in the packet quintuple.

[0013] In one embodiment, step S1 further includes: Step S1.3: When the traffic flow simultaneously contains both the first identification result and the second identification result, and the first identification result and the second identification result conflict, the first identification result shall prevail.

[0014] In one embodiment, step S2 includes at least one of the following: Step S2.1: Map the traffic whose semantic recognition result is synchronization traffic to a lossless high-priority transmission queue; Step S2.2: Map the traffic whose semantic recognition result is elastic traffic to a fallback transmission queue, and actively reduce the transmission rate of the fallback transmission queue according to the preset rate reduction control logic when an explicit congestion notification flag is detected. Step S2.3: Map the traffic whose semantic recognition result is management traffic to an independent management queue.

[0015] In one embodiment, step S2.2 includes: Step S2.2.1: Calculate the proportion of messages carrying explicit congestion notification flags received within the sliding time window, and reduce the transmission rate in an exponential backoff manner when the proportion of the messages exceeds a preset proportion threshold. Step S2.2.2: After the message ratio falls back below the preset ratio threshold, gradually restore the transmission rate.

[0016] In one embodiment, step S4 includes: Step S4.1: Obtain the synchronization traffic usage status and bandwidth usage of each physical path from the traffic usage list, wherein the traffic usage list is dynamically updated as the communication ring is established and released; Step S4.2: Based on the occupancy status and bandwidth of the synchronization traffic already carried by the multiple participating nodes and each physical path, select the available physical path with the lowest occupancy status from each physical path and allocate it to the communication ring as the physical path that meets the preset occupancy conditions.

[0017] In one embodiment, step S5 includes: Step S5.1: Calculate the communication completion time of each transmission segment in a single set communication operation of the communication ring, and calculate the mean and standard deviation of the completion time of all transmission segments in the communication ring in this communication. Step S5.2: In response to the fact that the difference between the communication completion time and the mean completion time of any transmission segment is greater than a preset multiple of the standard deviation, the corresponding transmission segment is determined as the target transmission segment; Step S5.3: If the target transmission segment continues to exist in a preset number of consecutive communication cycles, then the target transmission segment is determined to be the abnormal transmission segment.

[0018] In one embodiment, step S5 further includes: Step S5.4: Adjust the node arrangement order within the communication ring so that the participating nodes at both ends of the abnormal transmission segment are in a non-adjacent state; Step S5.5: The adjusted node arrangement order is issued through the topology configuration interface based on the rank mapping relationship provided by the aggregate communication library, so that subsequent aggregate communication operations are executed according to the adjusted node arrangement order.

[0019] Secondly, the present invention also provides a dynamic scheduling device for computing power network traffic based on a collective communication mode for an intelligent computing cloud platform, comprising: The identification module is used to semantically identify the traffic generated by the intelligent computing cloud platform during the execution of computing power tasks; The mapping module is used to map the traffic to the service quality queue corresponding to the semantic category based on the semantic recognition result, and maintain a traffic occupancy list between each physical path and different traffic types based on the service quality queue. The acquisition module is used to acquire the parallel topology information of the computing power task to be scheduled, and to determine the communication ring of the computing power task to be scheduled when performing a collection communication operation based on the parallel topology information, wherein the communication ring includes multiple participating nodes; The allocation module is used to select a physical path that meets preset occupancy conditions for communication transmission for the communication ring corresponding to the computing power running task to be scheduled, based on the multiple participating nodes and the traffic occupancy list. The monitoring module is used to continuously monitor the communication completion time of each transmission segment on the communication ring, and when it is determined that there is an abnormal transmission segment in the communication ring based on the communication completion time, the node arrangement order in the communication ring is adjusted to bypass the abnormal transmission segment, wherein the transmission segment consists of at least two participating nodes.

[0020] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the method for dynamic scheduling of computing power network traffic based on a collective communication mode of the intelligent computing cloud platform described in the first aspect above.

[0021] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for dynamic scheduling of computing power network traffic based on a collective communication mode in the intelligent computing cloud platform described in the first aspect above.

[0022] Fifthly, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the method for dynamic scheduling of computing power network traffic based on a collective communication mode in the intelligent computing cloud platform described in the first aspect above.

[0023] In this invention, step S1 involves semantic recognition of the traffic generated by the intelligent computing cloud platform during the execution of computing power tasks; step S2 involves mapping the traffic to the service quality queue corresponding to the semantic category based on the semantic recognition results, and maintaining a traffic occupancy list between each physical path and different traffic types based on the service quality queue; step S3 involves obtaining the parallel topology information of the computing power tasks to be scheduled, and determining the communication ring when the computing power tasks to be scheduled perform aggregated communication operations based on the parallel topology information, wherein the communication ring includes multiple participating nodes; step S4 involves selecting a physical path that meets the preset occupancy conditions for communication transmission for the communication ring corresponding to the computing power tasks to be scheduled based on the multiple participating nodes and the traffic occupancy list; and step S5 involves continuously monitoring the communication completion time of each transmission segment on the communication ring, and when it is determined that there is an abnormal transmission segment in the communication ring based on the communication completion time, adjusting the node arrangement order in the communication ring to bypass the abnormal transmission segment, wherein the transmission segment consists of at least two participating nodes. This can effectively alleviate congestion problems caused by disordered traffic competition in multi-tenant mixed deployment scenarios of computing power clusters, and when there are abnormal transmission segments in the communication ring, it can proactively adjust the node arrangement order in the communication ring to bypass the abnormal transmission segments, thereby improving the overall transmission efficiency and network robustness of the aggregated communication. Attached Figure Description

[0024] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a method for dynamic scheduling of computing power network traffic based on a collective communication mode in an intelligent computing cloud platform, provided by the present invention. Figure 2 This is a schematic diagram of traffic semantic recognition features and differentiated QoS mapping proposed in this disclosure; Figure 3 This is a schematic diagram of the joint load balancing of communication ring participating nodes and physical paths based on the parallel topology-aware communication method proposed in this disclosure. Figure 4 This is a schematic diagram comparing the slow flow detection and dynamic reconstruction of the communication loop before and after, based on the present disclosure. Figure 5 This is a general framework diagram of a collective communication semantic scheduling system proposed in this disclosure; Figure 6 This is a schematic diagram of the overall process of the dynamic scheduling method for computing power network traffic based on the aggregated communication mode of the intelligent computing cloud platform proposed in this disclosure; Figure 7 This is a structural diagram of a dynamic scheduling device for computing power network traffic based on a collective communication mode for an intelligent computing cloud platform provided by the present invention. Figure 8 This is a structural diagram of an electronic device provided by the present invention. Detailed Implementation

[0026] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The “computing power” mentioned in this invention refers to the ability of computer equipment or computing / data centers to process information. It is the ability of computer hardware and software to work together to perform a certain computing requirement. It is the computing power to achieve the target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0028] The "computational power" (CP) described in this invention refers to the ability of a data center server to process data and output results. It is a comprehensive indicator of a data center's computing power, encompassing general computing power, supercomputing power, and intelligent computing power. The commonly used unit of measurement is floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), with higher values ​​indicating stronger overall computing power. It is estimated that 1 EFLOPS is approximately the computing power output of 5 Tianhe-2A supercomputers, 500,000 mainstream server CPUs, or 2 million mainstream laptops. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0029] The "Network Power" (NP) mentioned in this invention refers to the performance of data transmission capabilities of computing facilities, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, and involves network transmission within and between data centers. It is a comprehensive indicator for measuring network transmission scheduling capabilities.

[0030] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon operation. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and internal storage devices within servers. The commonly used unit of measurement for storage capacity is exabytes (EB, 1EB = 2^60 bytes), the commonly used unit of measurement for performance is the number of read / write operations per second (IOPS / TB), and the disaster recovery ratio is an important indicator of security and reliability.

[0031] The "computing infrastructure" mentioned in this invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, enabling centralized computing, storage, transmission, and application of information.

[0032] The "new information infrastructure" mentioned in this invention refers to network infrastructure such as 5G networks, fiber optic broadband networks, backbone networks, international communication networks, and satellite internet; computing infrastructure such as data centers, general computing centers, intelligent computing centers, and supercomputing centers; and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0033] The “computing power” mentioned in this invention includes: general computing power, intelligent computing power, and supercomputing power.

[0034] The "general computing power" mentioned in this invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0035] The "intelligent computing power" mentioned in this invention refers to: a computing platform deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various artificial intelligence innovative applications, such as natural language processing and machine vision.

[0036] The “supercomputing power” mentioned in this invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0037] The "intelligent computing center" described in this invention refers to a facility that, through the use of large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), primarily provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models). The intelligent computing center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0038] The "intelligent computing cloud platform" mentioned in this invention, abbreviated as "intelligent computing cloud", refers to a cloud computing platform that integrates hardware and software resources based on an intelligent computing center.

[0039] The "intelligent computing center" mentioned in this invention includes, but is not limited to, "smart computing center".

[0040] The "intelligent computing center" mentioned in this invention, also known as an artificial intelligence computing center, is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0041] The "computing center" mentioned in this invention refers to a facility that is mainly composed of infrastructure such as wind, thermal, hydro, and electricity, and IT hardware and software equipment, and has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0042] The "supercomputing center" mentioned in this invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters. It can provide large-scale computing, storage and network services and is widely used in aerospace, defense, oil exploration, climate modeling and genome sequencing and other application scenarios.

[0043] The “computing resources” mentioned in this invention refer to the technologies and facilities required for the development of the digital society that have the ability to compute, transmit, store and apply information, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guaranteeing resources such as wind, fire, water and electricity.

[0044] The "model" mentioned in this invention includes, but is not limited to, "large language model" and "multimodal large model".

[0045] The "large language model" mentioned in this invention refers to a large-scale language model (LLM), which is a language model with a large number of parameters. It is designed to understand and generate human language. It is trained with a large amount of text data and can perform a wide range of tasks, including text summarization, translation, and sentiment analysis.

[0046] The “Multimodal Large Models” mentioned in this invention refer to models that combine multimodal information such as text, images, videos, and audio for training, including but not limited to multimodal large language models.

[0047] The “collective communication mode” described in this invention refers to a standardized communication mechanism for data synchronization or reduction among multiple computing nodes in distributed parallel computing, typically such as the All Reduce or All Gather operation initiated by the NCCL communication library.

[0048] The “computing power running task” mentioned in this invention refers to a specific workload or job executed on computing power resources that requires a certain amount of computing power support, usually involving complex data processing, numerical calculation, model training or simulation scenarios.

[0049] The "semantic recognition" described in this invention refers to the process of classifying the service attributes and behavioral characteristics (i.e., "semantics") of traffic transmitted in a network, such as classifying it into synchronous, elastic, or management categories.

[0050] The "Quality of Service queue" mentioned in this invention refers to a forwarding buffer queue set up in a network device (such as a switch) for traffic of different priorities. For example, it may include a lossless high-priority queue, a rollback queue, and a separate management queue.

[0051] The "physical path" mentioned in this invention refers to the end-to-end network forwarding path that carries communication between adjacent participating nodes in a communication ring. It is usually identified by the network device ports and links it passes through (e.g., Spine switches and their corresponding ports).

[0052] The "traffic occupancy list" mentioned in this invention refers to a record table or data structure maintained by the scheduling layer, used to record the number of synchronous communication rings currently carried on each physical path and their occupied bandwidth, providing a basis for ring construction and routing for subsequent new tasks.

[0053] The "parallel topology information" mentioned in this invention refers to the node organization and partitioning method of the task to be scheduled for computing power during distributed training, such as the node partitioning of the tensor parallel group (TP) or the data parallel group (DP).

[0054] The "communication ring" mentioned in this invention refers to a logical communication topology set composed of specific computing nodes (i.e., participating nodes) that are interconnected and transmit data in a set communication operation.

[0055] The “transmission segment” mentioned in this invention refers to an end-to-end data transmission segment consisting of two adjacent participating node pairs in a communication ring.

[0056] The “abnormal transmission segment” mentioned in this invention refers to a transmission segment whose communication completion time continuously deviates from the normal baseline value. For example, it may be a physical link with abnormal conditions such as physical congestion hotspots or hardware aging.

[0057] The “calling behavior” described in this invention refers to the action of an application or computing task calling the API exposed by the underlying collection communication library (such as NCCL) (e.g., triggering an All Reduce operation).

[0058] The “synchronization traffic” described in this invention refers to network traffic generated by aggregated communication operations that is highly sensitive to latency and has periodic burst characteristics.

[0059] The “message arrival time interval” mentioned in this invention refers to the time difference between the arrival of two adjacent network messages at the detection node in the same data stream, which can be used to indirectly reflect the relevant characteristics of traffic from a statistical perspective.

[0060] The "autocorrelation characteristic" mentioned in this invention refers to the periodic recurrence pattern of the message arrival time interval sequence on the time series, such as whether there is a significant autocorrelation peak at a training step period.

[0061] The “message quintuple” mentioned in this invention refers to the set of basic attributes used to uniquely identify a data stream in network communication, which typically includes the source IP address, destination IP address, source port, destination port, and protocol number.

[0062] The "coefficient of variation" mentioned in this invention refers to a mathematical parameter used to measure the dispersion of message arrival time intervals, such as the ratio of the standard deviation of the time interval to its mean.

[0063] The "elastic traffic" mentioned in this invention refers to business traffic that has a demand for network bandwidth but a high tolerance for instantaneous latency, such as traffic generated by checkpoint uploads or data prefetching.

[0064] The "management traffic" mentioned in this invention refers to traffic used to maintain system operation and ensure basic reachability between nodes but not to participate in business bandwidth competition, such as heartbeat messages or control signaling.

[0065] The "lossless high-priority transmission queue" described in this invention refers to a queue that reuses the inherent PFC (priority-based flow control) packet loss-free mechanism of lossless networks and is assigned the highest forwarding priority to ensure that synchronous traffic has priority in obtaining forwarding resources.

[0066] The "reversible transmission queue" described in this invention refers to a queue configured with active congestion control logic, characterized in that it can actively reduce the transmission rate and give up network bandwidth to high-priority traffic when network congestion occurs.

[0067] The "Explicit Congestion Notification Mark" mentioned in this invention refers to a congestion mark (ECN-CE) added to the header of a data packet when a network forwarding device detects link congestion, which is used to report the network congestion status to the sending end.

[0068] The "exponential backoff method" described in this invention refers to the following: when the sending end receives congestion markers exceeding a threshold proportion, it rapidly reduces the sending window or decreases the sending rate by an exponential magnitude to quickly alleviate network congestion.

[0069] The "synchronous traffic occupancy status and bandwidth occupancy" mentioned in this invention refers to a comprehensive indicator used to evaluate the current load level of a physical path, which combines the number of synchronous communication rings carried by the path with the actual bandwidth resources consumed.

[0070] The “single ensemble communication operation” described in this invention refers to a complete ensemble communication process executed by the computing power running task during the training iteration process (such as one All Reduce data synchronization).

[0071] The "mean completion time and standard deviation of completion time" mentioned in this invention refer to statistical parameters used to establish the baseline and allowable fluctuation range of the time consumption of each transmission segment within the communication loop.

[0072] The "communication cycle" mentioned in this invention refers to the iterative cycle in which the same set of communication operations is repeatedly executed in a computing power operation task. Confirmation through M consecutive communication cycles aims to eliminate misjudgments caused by instantaneous network jitter.

[0073] The "node arrangement order" mentioned in this invention refers to the logical sequential connection relationship of each participating node in the communication ring.

[0074] The “collective communication library” mentioned in this invention refers to a software middleware (such as NCCL) responsible for scheduling and managing the underlying communication of multiple nodes in a distributed cluster.

[0075] It is important to emphasize that implementing computing network traffic scheduling based on aggregated communication in intelligent computing cloud platforms faces extremely complex challenges, which will be elaborated in detail below: First, the difficulty in perceiving the semantics of traffic leads to disorderly competition and congestion. Traditional network isolation typically only extends to the tenant or IP 5-tuple level. The network layer cannot perceive the true business semantics of computing traffic (such as synchronous traffic like All Reduce / All Gather on the critical path, and elastic traffic like checkpoint upload with higher tolerance). If indiscriminate unified forwarding is used, the large volume of concurrent elastic flows often crowds out the bandwidth required for synchronous flows, causing latency in the aggregated communication loop, congesting the entire computing cluster, and wasting GPU computing resources.

[0076] Second, static routing mechanisms have vague perception capabilities and can easily lead to congestion caused by overlapping physical paths. Current computing networks, in the absence of intervention, generally rely on the Equal Cost Multipath (ECMP) mechanism for hash distribution. This static hashing mechanism cannot perceive the parallel topology information of the application layer tasks to be scheduled (such as the node division of tensor parallel groups and data parallel groups), nor can it know the current actual carrying status of each physical link. In scenarios with multiple concurrent tasks, it is very easy for the communication ring of multiple tasks to be blindly hashed, resulting in the superposition of synchronous traffic on the same physical path (such as the same Spine port), forming congestion hotspots.

[0077] Third, the lack of dynamic reconstruction and self-healing capabilities for "slow flow". Computing networks have extremely high requirements for the forwarding latency of physical hardware. If the forwarding latency of a physical link gradually increases due to reasons such as aging switch port hardware (i.e., a "slow flow" anomaly), traditional static communication ring topologies cannot proactively avoid this link. Due to the lack of runtime dynamic awareness and reconstruction capabilities, the completion time of the entire communication ring will be continuously slowed down by individual abnormal transmission segments. At this time, it is often necessary to rely on the manual intervention of maintenance personnel for troubleshooting and hardware replacement, resulting in extremely high troubleshooting costs and seriously hindering the continuity of computing tasks.

[0078] In summary, the industry urgently needs a computing network scheduling scheme that can both finely identify traffic semantics and actively select routes based on parallel topology and achieve slow flow self-healing. This invention solves this pressing problem in the field of computing power.

[0079] Please see Figure 1 , Figure 1 This is a flowchart of a method for dynamic scheduling of computing power network traffic based on a collective communication mode in an intelligent computing cloud platform, as provided by the present invention. Figure 1 As shown, the method includes: Step S1: Perform semantic recognition on the traffic generated by the intelligent computing cloud platform during the execution of computing power tasks.

[0080] For example, in this embodiment, when performing semantic recognition on the traffic generated by the intelligent computing cloud platform during the execution of computing power tasks, it can be based on a pre-trained machine learning model, or it can be based on a third-party semantic recognition device, without any limitation.

[0081] It is understood that this embodiment does not limit the type and number of computing power running tasks. In this embodiment, by performing semantic recognition on the traffic flowing through the nodes of computing power running tasks in the running state in the intelligent computing cloud platform, real-time perception of the node load status can be achieved.

[0082] It's important to note that in the distributed training scenario of an intelligent computing cloud platform, the network traffic generated by different tasks within the same tenant (such as aggregate communication as a key synchronization point and checkpoint upload as a background task) exhibits different sensitivities to latency. Traditional network isolation typically employs static rate limiting strategies at the tenant level, lacking awareness of the true business semantic characteristics of the traffic. This coarse-grained management approach prevents the system from providing fine-grained protection for critical business operations, often resulting in large, elastic concurrent traffic volumes crowding out the bandwidth required for synchronization streams, causing overall computational congestion.

[0083] Therefore, it is urgent to accurately semantically identify the traffic passing through network nodes during the operation of computing tasks on the intelligent computing cloud platform before the data is actually queued and forwarded, so as to provide reference information for the subsequent implementation of service quality queue allocation.

[0084] The beneficial technical effect achieved by step S1 is that it enables online differentiation of different characteristic traffic generated during the operation of computing power tasks, thus providing a prerequisite for subsequent implementation of differentiated network resource scheduling.

[0085] Step S2: Based on the semantic recognition results, map the traffic to the service quality queue corresponding to the semantic category, and maintain a traffic usage list between each physical path and different traffic types based on the service quality queue.

[0086] In this embodiment, the traffic usage list can be used to record the number of synchronous communication rings currently carried on each physical path (such as each Spine switch port) and their occupied bandwidth.

[0087] It should be noted that even after obtaining the semantic features of network traffic, if a unified forwarding and caching mechanism is still used, the underlying network devices still cannot prioritize high-priority traffic when congestion occurs. Traditional data center networks lack end-to-end quality of service isolation and do not record the real-time distribution of specific service traffic on physical links. In this case, not only will critical synchronous flows be damaged while queuing, but the upper-layer scheduler will also be unable to grasp the true load information of the physical path.

[0088] Therefore, it is urgent to map traffic to the corresponding quality of service queues for resource isolation based on semantic recognition results, and to simultaneously maintain traffic occupancy lists for each physical path and different traffic types, thereby laying the foundation for avoiding physical path congestion and achieving joint load balancing in the future.

[0089] The beneficial technical effects achieved by step S2 are: it can establish a mapping relationship between traffic semantics and the underlying network service quality queue, thereby ensuring forwarding resources for high-priority services; at the same time, the dynamically maintained traffic occupancy list can provide data support for global joint load balancing of the control plane.

[0090] Step S3: Obtain the parallel topology information of the task to be scheduled by the computing power, and determine the communication ring of the task to be scheduled by the computing power when performing the collective communication operation based on the parallel topology information. The communication ring includes multiple participating nodes.

[0091] It is understandable that when a task to be scheduled to run performs a set of communication operations, a specific set of nodes is usually required to form a communication ring to work together. In this embodiment, by obtaining the parallel topology information of the task to be scheduled to run (such as tensor parallel group and data parallel group), the logical collaborative relationship between computing nodes can be accurately perceived.

[0092] The beneficial technical effect achieved by step S3 is that it enables the underlying network scheduling system to proactively perceive the distributed training parallel topology of the application layer, thereby providing reference information for subsequent physical path allocation.

[0093] Step S4: Based on the list of multiple participating nodes and traffic usage, select a physical path that meets the preset usage conditions for communication transmission for the communication ring corresponding to the task to be scheduled.

[0094] It should be noted that in lossless network architectures with multipath forwarding capabilities, traditional Equal Cost Multipath (ECMP) load balancing mechanisms typically employ static hash algorithms, lacking a dynamic intervention mechanism for the real-time carrying status of physical paths. Under this approach, the system cannot avoid hotspot links, easily leading to superimposed congestion of multiple concurrent task communication rings on the same physical device (such as the same Spine port), resulting in a sharp drop in network performance.

[0095] Therefore, it is urgent to proactively select idle or low-load physical paths that meet preset occupancy conditions for the communication ring based on multiple participating nodes and the maintained traffic occupancy list, thereby providing support for resolving the superimposed congestion caused by static routing in the future.

[0096] The beneficial technical effect achieved by step S4 is that by performing routing selection during the establishment phase of the communication ring, physical paths that meet the conditions of idle or low load can be allocated to the communication ring, thereby effectively avoiding the problem of overlapping congestion of synchronous traffic of multiple concurrent tasks on the same physical link.

[0097] Step S5: Continuously monitor the communication completion time of each transmission segment on the communication ring, and when it is determined that there is an abnormal transmission segment in the communication ring based on the communication completion time, adjust the node arrangement order in the communication ring to bypass the abnormal transmission segment. The transmission segment consists of at least two participating nodes.

[0098] It should be noted that during long-term computing tasks, the underlying network hardware may age or experience localized congestion, leading to abnormal latency (i.e., "slow flow"). Traditional aggregated communication libraries typically statically fix the order of participating nodes, making it difficult for the computing platform to handle physical-level slow flow anomalies, thus affecting the overall training progress.

[0099] Therefore, there is an urgent need for a method to continuously monitor the communication completion time of each transmission segment on the communication ring, and to dynamically adjust the node arrangement order within the communication ring to bypass abnormal transmission segments when abnormal transmission segments are identified.

[0100] The beneficial technical effect achieved by step S5 is that it enables real-time dynamic perception of abnormal latency in the network (such as slow flow caused by hardware aging), and automatically bypasses abnormal physical links by adjusting the node order, thereby effectively avoiding the impact of abnormal links on the overall computing efficiency.

[0101] In this invention, step S1 involves semantic recognition of the traffic generated by the intelligent computing cloud platform during the execution of computing power tasks; step S2 involves mapping the traffic to the service quality queue corresponding to the semantic category based on the semantic recognition results, and maintaining a traffic occupancy list between each physical path and different traffic types based on the service quality queue; step S3 involves obtaining the parallel topology information of the computing power tasks to be scheduled, and determining the communication ring when the computing power tasks to be scheduled perform aggregated communication operations based on the parallel topology information, wherein the communication ring includes multiple participating nodes; step S4 involves selecting a physical path that meets the preset occupancy conditions for communication transmission for the communication ring corresponding to the computing power tasks to be scheduled based on the multiple participating nodes and the traffic occupancy list; and step S5 involves continuously monitoring the communication completion time of each transmission segment on the communication ring, and when it is determined that there is an abnormal transmission segment in the communication ring based on the communication completion time, adjusting the node arrangement order in the communication ring to bypass the abnormal transmission segment, wherein the transmission segment consists of at least two participating nodes. This can effectively alleviate congestion problems caused by disordered traffic competition in multi-tenant mixed deployment scenarios of computing power clusters, and when there are abnormal transmission segments in the communication ring, it can proactively adjust the node arrangement order in the communication ring to bypass the abnormal transmission segments, thereby improving the overall transmission efficiency and network robustness of the aggregated communication.

[0102] In one embodiment, step S1 includes at least one of the following: Step S1.1: Intercept the calling behavior of the computing power running task in the collection communication operation, and identify the traffic generated by the calling behavior as synchronization traffic as the first identification result.

[0103] In other words, this embodiment can use a direct identification channel when performing traffic semantic recognition: by intercepting API calls exposed by the aggregate communication library (such as NCCL) (such as the call entry points of operations like All Reduce and All Gather), the five-tuple information of the network traffic generated by this call is directly bound to the "synchronization class" tag. Since this identification method combines deterministic behavioral information from the application layer, its identification result can be used as the judgment basis with the highest confidence.

[0104] Step S1.2: Semantic classification and identification of the message arrival time interval and autocorrelation characteristics of the traffic of the computing power running task within the sliding time window are performed to determine the second identification result.

[0105] In other words, in this embodiment, an indirect identification channel can be used for traffic semantic recognition, which is a fallback identification when the communication library call behavior cannot be intercepted.

[0106] The beneficial technical effects achieved by steps S1.1 and S1.2 are: they can effectively combine the two channels of direct interception of application layer behavior and indirect identification of network layer packet statistical features, taking into account both the accuracy of semantic recognition and the fallback identification capability when the calling interface is not visible, thereby effectively improving the reliability of traffic semantic recognition.

[0107] In one embodiment, step S1.2 includes: Step S1.2.1: Aggregate the traffic into independent flows according to the packet 5-tuple, and count the packet arrival time interval sequence of the flow within the sliding time window; Step S1.2.2: Calculate the coefficient of variation of the message arrival time interval sequence and detect the autocorrelation peak of the message arrival time interval sequence at the preset lag. Step S1.2.3: If the coefficient of variation is less than the first preset threshold and the autocorrelation peak value is greater than the second preset threshold, then the semantic category of the flow corresponding to the flow is determined to be the synchronous flow. Step S1.2.4: If the coefficient of variation is greater than or equal to the first preset threshold, and / or the autocorrelation peak is less than or equal to the second preset threshold, then the traffic corresponding to the flow is determined to be elastic traffic or management traffic by combining the port range in the packet quintuple and the historical label.

[0108] In other words, in this embodiment, when performing traffic semantic recognition, the generated network traffic can be aggregated into independent flows according to the five-tuple (source / destination IP, source / destination port, protocol number). The packet arrival time interval sequence {Δτ_j} of the flow within the sliding time window is statistically analyzed, and its coefficient of variation CV = std(Δτ) / mean(Δτ) is calculated. Autocorrelation analysis is performed on the interval sequence to detect whether there is a significant peak at a lag of one training step period. If the coefficient of variation is lower than a preset threshold and the autocorrelation peak is higher than a preset threshold, the flow is determined as a "synchronization candidate". Otherwise, it is determined as "elastic" or "management" based on the five-tuple port range and historical labels.

[0109] The beneficial technical effects achieved by steps S1.2.1 to S1.2.4 are: the ability to effectively combine the variation coefficient of message arrival time interval and autocorrelation peak value for quantitative analysis, thereby accurately identifying synchronous traffic with periodic burst characteristics and elastic traffic with continuous transmission.

[0110] In one embodiment, step S1 further includes: Step S1.3: When both the first identification result and the second identification result exist for the traffic, and the first identification result and the second identification result conflict, the first identification result shall prevail.

[0111] The beneficial technical effect achieved by step S1.3 is that by taking the identification result determined by the intercepted call behavior as the standard when conflicts occur in different identification channels, the reliability of the semantic identification result can be effectively guaranteed.

[0112] Based on the above embodiments, after traffic semantic recognition is completed, differentiated processing paths can be imported respectively, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of traffic semantic recognition features and differentiated QoS mapping proposed in this disclosure.

[0113] In one embodiment, step S2 includes at least one of the following: Step S2.1: Map traffic whose semantic recognition result is synchronization traffic to a lossless high-priority transmission queue.

[0114] In this embodiment, synchronization traffic is mapped to a lossless high-priority transmission queue. This queue reuses the PFC no-drop mechanism inherent in RoCE networks and is given a higher scheduling priority than other queues, which can ensure that synchronization traffic gets priority in forwarding resources when queuing for forwarding at the switch.

[0115] Step S2.2: Map traffic with semantic recognition results of elastic traffic to a fallback transmission queue, and actively reduce the transmission rate of the fallback transmission queue according to the preset rate reduction control logic when an explicit congestion notification flag is detected.

[0116] In this embodiment, by actively reducing the transmission rate of the fallback transmission queue according to a preset rate-reduction control logic when an explicit congestion notification flag is detected, bandwidth can be actively transferred to synchronous traffic when network congestion occurs, thereby ensuring the transmission stability of synchronous traffic.

[0117] Step S2.3: Map traffic whose semantic recognition result is management traffic to an independent management queue.

[0118] In this embodiment, management traffic (such as heartbeats and control signaling) is mapped to a separate management queue, which guarantees basic reachability but does not participate in the adjustment of bandwidth contention priority.

[0119] The beneficial technical effects achieved by steps S2.1 to S2.3 are: while ensuring that synchronous traffic is queued with the highest priority under the lossless network architecture, elastic traffic can actively slow down and relinquish bandwidth when network congestion occurs, thereby avoiding mutual interference between different semantic services.

[0120] In one embodiment, step S2.2 includes: Step S2.2.1: Calculate the proportion of messages carrying explicit congestion notification flags received within the sliding time window, and reduce the sending rate in an exponential backoff manner when the proportion of messages exceeds a preset proportion threshold. Step S2.2.2: After the message ratio falls back below the preset ratio threshold, gradually restore the sending rate.

[0121] In this embodiment, elastic traffic is mapped to a fallback transmission queue. When a network device detects congestion and returns an explicit congestion notification (ECN-CE) flag, the sender corresponding to the queue actively reduces the transmission rate according to a preset rate reduction control logic (e.g., the proportion of packets receiving the ECN-CE flag within a sliding window is counted, and when the proportion exceeds a preset threshold, the transmission window is reduced exponentially until the proportion falls below the threshold and then gradually restored). This allows bandwidth to be actively transferred to synchronous traffic when network congestion occurs.

[0122] The beneficial technical effects achieved by steps S2.2.1 to S2.2.2 are as follows: By combining explicit congestion notification with the exponential backoff algorithm, network traffic can respond quickly and smoothly to the congestion state, thereby effectively suppressing the continuous deterioration of the congestion situation.

[0123] In one embodiment, step S4 includes: Step S4.1: Obtain the synchronization traffic usage status and bandwidth usage of each physical path from the traffic usage list. The traffic usage list is dynamically updated as the communication ring is established and released. Step S4.2: Based on the occupancy status and bandwidth of the synchronization traffic already carried by multiple participating nodes and each physical path, select the available physical path with the lowest occupancy status from each physical path and allocate it to the communication ring as a physical path that meets the preset occupancy conditions.

[0124] It should be noted that in network structures with multi-path forwarding capabilities, such as Leaf-Spine, multiple equivalent forwarding paths with comparable latency and bandwidth typically exist between any pair of computing nodes. These paths may be forwarded via different Spine switches, and traffic is distributed through equal-cost multipath (ECMP) mechanisms in the absence of intervention. Therefore, after the set of participating nodes in the communication ring is determined based on the task parallel topology information, the specific physical path through which communication between adjacent pairs of participating nodes in the ring is transmitted remains a degree of freedom that can be actively intervened by the scheduling layer. The path selection described in this invention operates on this degree of freedom.

[0125] The beneficial technical effects achieved by steps S4.1 to S4.2 are: by continuously monitoring the occupancy status of the synchronous flow of each physical path, the optimal physical path allocation for the computing power network can be achieved.

[0126] In one possible implementation, in a two-layer Leaf-Spine network, a physical path can be uniquely identified by the Spine switches it traverses and their corresponding ports, with each Spine switch port serving as the statistical granularity. In other network structures such as three-layer Fat-Tree, Dragonfly, or rail-optimized networks, a physical path can be identified by the combination of ports of the various levels of switching equipment it traverses, and the statistical granularity of the occupancy list is adjusted accordingly. The differences in path identification methods and statistical granularity under the above-mentioned various network structures do not affect the implementation of the path selection mechanism described in this invention.

[0127] like Figure 3 As shown, Figure 3 This is a schematic diagram of the joint load balancing of communication ring participating nodes and physical paths based on the parallel topology-aware communication ring participating node determination proposed in this disclosure. The scheduling layer maintains a "physical path and synchronization traffic occupancy list" (i.e., the aforementioned traffic occupancy list), recording the number of synchronization communication rings currently carried on each physical path (such as each Spine switch port) and their occupied bandwidth. When a new training task is submitted and its parallel topology (such as the node division of tensor parallel groups and data parallel groups) is determined, the path load balancer queries the occupancy list based on the set of participating nodes involved in the communication operations of the task set. It then selects the available physical path with the lowest current occupancy for the task's communication ring to complete the inter-node routing allocation, thereby proactively avoiding superimposed congestion with communication rings of other tasks on the same physical path or its constituent links (such as the same Spine switch port) during the communication ring establishment phase.

[0128] In one embodiment, step S5 includes: Step S5.1: Calculate the communication completion time of each transmission segment in a single set communication operation of the communication ring, and calculate the mean and standard deviation of the completion time of all transmission segments in the communication ring in this communication. Step S5.2: In response to the fact that the difference between the communication completion time and the mean completion time of any transmission segment is greater than a preset multiple of the standard deviation, the corresponding transmission segment is determined as the target transmission segment. Step S5.3: If the target transmission segment continues to exist for a preset number of consecutive communication cycles, the target transmission segment is determined to be an abnormal transmission segment.

[0129] For example, suppose a synchronous communication ring consists of N adjacent participating node pairs forming a transmission segment in a single aggregate communication operation, and the completion time of the i-th segment in this communication is denoted as t. i (i = 1, 2, …, N). Define the mean and standard deviation of the completion time for this communication in the loop: mean Standard deviation .

[0130] When there exists a segment k satisfying (t) k When the value of θ > σ (θ is a preset multiple, typically 3, i.e., the 3σ principle is adopted), it is determined that there is an abnormal delay in the current communication of this segment. In order to avoid misjudgment caused by instantaneous network jitter, the physical link corresponding to segment k is finally determined to be an abnormal link (because it usually manifests as a local congestion hotspot, it is also referred to as a hotspot link in this paper) only when the abnormal phenomenon continues to occur in M ​​consecutive communication cycles (i.e., M consecutive collection communication operations of the same ring).

[0131] The beneficial technical effects achieved by steps S5.1 to S5.3 are as follows: by introducing a dynamic statistical threshold composed of the mean and standard deviation and a multi-cycle continuous confirmation condition, false alarms caused by occasional network jitter can be effectively filtered out, thereby improving the accuracy and reliability of slow flow anomaly judgment.

[0132] In one embodiment, step S5 further includes: Step S5.4: Adjust the node arrangement order within the communication ring so that the participating nodes at both ends of the abnormal transmission segment are in a non-adjacent state; Step S5.5: The adjusted node arrangement order is issued through the topology configuration interface based on the rank mapping relationship provided by the set communication library, so that subsequent set communication operations are executed according to the adjusted node arrangement order.

[0133] In other words, in this embodiment, after identifying the abnormal link, the ring reconstruction proposer can generate an adjusted arrangement order of participating nodes in the communication ring, so that the participating nodes at both ends of the original segment k in the adjusted ring no longer maintain an adjacent relationship (i.e., no longer communicate directly through the identified abnormal physical link). This adjusted arrangement order is then distributed through a custom topology configuration interface provided by the aggregated communication library (such as a re-initialization interface based on rank mapping), so that subsequent aggregated communication operations are automatically executed according to the new ring topology, bypassing the abnormal link. Figure 4 As shown, Figure 4 This is a comparative diagram of slow flow detection and dynamic reconstruction of the communication loop before and after, based on the present disclosure. Figure 4 The left side shows the case where the completion time of a certain segment deviates significantly from the mean before reconstruction (in the example, the completion time of this segment is about 5.8 times the mean of the other segments, exceeding the 3σ threshold); the right side shows the effect of the completion time of each segment returning to uniformity after reconstruction and the original abnormal physical links no longer being used by the ring topology.

[0134] The beneficial technical effects achieved by steps S5.4 to S5.5 are as follows: By changing the logical arrangement order of participating nodes within the communication ring topology, network fault self-healing can be achieved without manual intervention. By directly issuing a new topology through the aggregated communication library to physically bypass abnormal links, the operation and maintenance intervention cost of the intelligent computing cloud platform can be significantly reduced.

[0135] In summary, the intelligent computing cloud platform dynamic scheduling method for computing power network traffic based on the collective communication mode proposed in this application can be implemented based on a pre-configured collective communication semantic scheduling system, such as... Figure 5 As shown, Figure 5 This is a general framework diagram of a semantic scheduling system for aggregated communication proposed in this disclosure. The system proposed in this application includes a task layer, an identification layer, a scheduling layer, and a network layer. The task layer carries aggregated communication traffic for training tasks (such as AllReduce / All Gather initiated by NCCL) and elastic traffic such as checkpoint upload and data prefetching; the identification layer consists of traffic identification agents deployed on the node side, which are responsible for semantic classification of traffic passing through the node; the scheduling layer includes four functional components: a lossless high-priority queue, a rollback queue, a path load balancer, and a ring reconstruction proposer; the network layer is the RoCE / InfiniBand lossless network infrastructure that carries the actual data transmission.

[0136] based on Figure 5 The overall flowchart of the system and the method proposed in this application is shown below. Figure 6 As shown, Figure 6This is a schematic diagram of the overall process of the dynamic scheduling method for computing power network traffic based on the aggregated communication mode of the intelligent computing cloud platform proposed in this disclosure. The method can be implemented on two independent execution planes: the data plane, which is executed continuously and sequentially for traffic passing through network nodes, i.e., online semantic identification of traffic is completed in step a, and then mapped to differentiated QoS queues and forwarded according to queue policies in step b; the control plane, which is executed once during ring establishment for newly submitted training tasks with determined parallel topologies, i.e., joint load balancing of the communication ring and physical paths is completed in step c. There is no sequential execution relationship between steps a–b and step c; they only form a data dependency through a pre-maintained physical path and synchronous traffic occupancy list: the semantic identification result of step a determines the traffic categories counted in the list, and step c queries the list when selecting routes for the communication ring and updates the list when the communication ring is established and released. For a single task, the actual timing is as follows: step c first completes the establishment and path allocation of the communication ring; the communication traffic subsequently generated by this communication ring then enters the data plane for identification by step a and mapping by step b. The order in which the steps are written in the claims is a parallel enumeration and does not constitute a limitation on the execution order. After the communication ring is established and iterative communication begins, the system continuously monitors the communication completion time of each segment of the communication ring in step d. Once an abnormal link that meets preset conditions is determined, the system proceeds to step e to complete the dynamic reconstruction of the communication ring, and then returns to step d to enter the next monitoring cycle; if no abnormality is determined, the system also directly returns to step d to enter the next monitoring cycle.

[0137] Based on the above embodiments, this disclosure can achieve at least the following technical effects: 1. The variance of training iteration time is significantly reduced: Because synchronous traffic is exclusively assigned a lossless high-priority queue after being identified, and its communication ring path is actively balanced with other tasks to avoid the long-tail effect of delay in a single segment within the communication ring slowing down all participating nodes, the dispersion of training iteration completion time (such as the P99 / P50 ratio) is expected to be reduced from about 1.3 to 1.4 under the existing static rate limiting scheme to about 1.05 to 1.1 (estimated value, the specific value varies with the cluster size and the number of concurrent tenants).

[0138] 2. Elastic traffic peaks no longer impact training throughput: Because elastic traffic such as checkpoint uploads and data prefetching is identified and isolated into a fallback queue, and receives active speed reduction from ECN when the network is congested, the collective communication of training tasks no longer directly competes for bandwidth with this type of traffic. The jitter of training iteration time caused by concurrent checkpoint uploads and other operations is expected to be reduced from about ±10% to 15% under the existing scheme to about ±3% (estimated value).

[0139] 3. Reduced probability of congestion in multi-task communication rings: Because the path load balancer actively selects low-occupancy physical paths based on the real-time occupancy list during the communication ring establishment phase, replacing the static or path-coordination-free allocation method in existing technologies that lack runtime occupancy awareness, the probability of congestion occurring in synchronous communication rings of multiple tasks on the same physical network path or its constituent links (such as the same Spine switch port) is significantly reduced.

[0140] 4. Significantly accelerated detection and avoidance response of network link anomalies: Because the slow flow detection and ring reconstruction mechanism automatically determines abnormal links and automatically issues new ring topology configurations based on the statistical dispersion of communication completion time, it replaces the existing technology's method of relying on manual packet capture analysis and locating abnormal links. The detection and avoidance response of abnormal links is expected to be reduced from the hours required for the current manual investigation to the minutes or even less required for automated detection and reconstruction.

[0141] Please see Figure 7 , Figure 7 This is a structural diagram of a dynamic scheduling device for computing power network traffic based on a collective communication mode for an intelligent computing cloud platform provided by the present invention, as shown below. Figure 7 As shown, the intelligent computing cloud platform's computing power network traffic dynamic scheduling device 700 based on a collective communication mode includes: The identification module 701 is used to perform semantic recognition on the traffic generated by the intelligent computing cloud platform during the execution of computing power tasks. The mapping module 702 is used to map traffic to the service quality queue corresponding to the semantic category based on the semantic recognition result, and maintain a traffic usage list between each physical path and different traffic types based on the service quality queue. The acquisition module 703 is used to acquire the parallel topology information of the task to be scheduled by the computing power, and to determine the communication ring of the task to be scheduled by the computing power when performing the collective communication operation based on the parallel topology information. The communication ring includes multiple participating nodes. The allocation module 704 is used to select a physical path that meets the preset occupancy conditions for communication transmission for the communication ring corresponding to the computing power to be scheduled to run the task, based on multiple participating nodes and a traffic occupancy list; The monitoring module 705 is used to continuously monitor the communication completion time of each transmission segment on the communication ring, and when it is determined that there is an abnormal transmission segment in the communication ring based on the communication completion time, the node arrangement order in the communication ring is adjusted to bypass the abnormal transmission segment. The transmission segment consists of at least two participating nodes.

[0142] In one embodiment, the identification module 701 is configured to perform at least one of the following: Intercept the calling behavior of computing power running tasks in the collection communication operation, and identify the traffic generated by the calling behavior as synchronization traffic as the first identification result; The traffic of the task running the system computing power is semantically classified and identified based on the message arrival time interval and autocorrelation characteristics within the sliding time window to determine the second identification result.

[0143] In one embodiment, the identification module 701 is specifically used for: Traffic is aggregated into independent flows based on packet 5-tuples, and the packet arrival time interval sequence of the flow within the sliding time window is statistically analyzed. Calculate the coefficient of variation of the message arrival time interval sequence and detect the autocorrelation peak of the message arrival time interval sequence at a preset lag. If the coefficient of variation is less than the first preset threshold and the peak value of autocorrelation is greater than the second preset threshold, then the semantic category of the traffic corresponding to the flow is determined to be synchronous traffic. If the coefficient of variation is greater than or equal to the first preset threshold, and / or the autocorrelation peak is less than or equal to the second preset threshold, then the traffic corresponding to the flow will be determined as elastic traffic or management traffic by combining the port range in the packet quintuple with the historical tags.

[0144] In one embodiment, the identification module 701 is further configured to: When traffic has both a first identification result and a second identification result, and the first identification result and the second identification result conflict, the first identification result shall prevail.

[0145] In one embodiment, the mapping module 702 is used to implement at least one of the following: Map traffic whose semantic recognition result is synchronization traffic to a lossless high-priority transmission queue; Traffic that is semantically identified as elastic traffic is mapped to a fallback transmission queue, and the transmission rate of the fallback transmission queue is actively reduced according to the preset rate reduction control logic when an explicit congestion notification flag is detected. Traffic whose semantic recognition result is management traffic is mapped to a separate management queue.

[0146] In one embodiment, the mapping module 702 is specifically used for: The proportion of packets carrying explicit congestion notification flags received within a sliding time window is statistically analyzed, and the transmission rate is reduced exponentially when the proportion of packets exceeds a preset proportion threshold. The sending rate will be gradually restored after the message ratio falls below the preset ratio threshold.

[0147] In one embodiment, the allocation module 704 is specifically used for: The traffic usage status and bandwidth of each physical path are obtained from the traffic usage list. The traffic usage list is dynamically updated as the communication ring is established and released. Based on the synchronous traffic occupancy and bandwidth occupancy of multiple participating nodes and each physical path, the available physical path with the lowest occupancy is selected from all physical paths and allocated to the communication ring as a physical path that meets the preset occupancy conditions.

[0148] In one embodiment, the monitoring module 705 is specifically used for: The communication completion time of each transmission segment in a single aggregate communication operation is statistically analyzed, and the mean and standard deviation of the completion time of all transmission segments in the communication ring in this communication are calculated. If the difference between the communication completion time and the mean completion time of any transmission segment is greater than a preset multiple of the standard deviation, the corresponding transmission segment is determined as the target transmission segment. If the target transmission segment persists for a preset number of consecutive communication cycles, the target transmission segment is determined to be an abnormal transmission segment.

[0149] In one embodiment, the monitoring module 705 is further configured to: The node arrangement order within the communication ring is adjusted so that the participating nodes at both ends of the abnormal transmission segment are in a non-adjacent state; The adjusted node arrangement order is issued through the topology configuration interface based on rank mapping provided by the collective communication library, so that subsequent collective communication operations are executed according to the adjusted node arrangement order.

[0150] This can effectively alleviate congestion problems caused by disordered traffic competition in multi-tenant mixed deployment scenarios of computing power clusters, and when there are abnormal transmission segments in the communication ring, it can actively adjust the arrangement order of nodes in the communication ring to bypass the abnormal transmission segments, thereby improving the overall transmission efficiency and network robustness of the aggregated communication.

[0151] It should be noted that the intelligent computing cloud platform based on the aggregated communication mode computing network traffic dynamic scheduling device in this invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0152] The present invention also provides an electronic device, see below. Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 801, a processor 802, and a program or instructions stored in the memory 801 that run on the processor 802. When the program or instructions are executed by the processor 802, they can achieve the following: Figure 1The corresponding steps in the embodiment of the dynamic scheduling method for computing power network traffic based on the aggregated communication mode of the intelligent computing cloud platform, and the achievement of the same beneficial effect, will not be elaborated here.

[0153] The processor 802 can be a CPU, ASIC, FPGA, or GPU.

[0154] Those skilled in the art will understand that all or part of the steps of the above-described embodiment of the method for dynamic scheduling of computing power network traffic based on the collective communication mode of the intelligent computing cloud platform can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0155] The present invention also provides a readable storage medium on which a computer program is stored, and which, when executed by a processor, can perform the above-described functions. Figure 1 The corresponding steps in the embodiment of the dynamic scheduling method for computing power network traffic based on the aggregated communication mode of the intelligent computing cloud platform can achieve the same technical effect, and will not be described again here to avoid repetition. The storage medium mentioned includes, for example, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0156] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The various processes of the corresponding intelligent computing cloud platform's dynamic scheduling method for computing power network traffic based on the aggregated communication mode are described in detail here to avoid repetition, as they can achieve the same technical effect.

[0157] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this invention indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0158] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0159] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of the present invention.

[0160] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these modifications are within the protection scope of the present invention.

Claims

1. A method for dynamic scheduling of computing network traffic based on a collective communication mode in an intelligent computing cloud platform, characterized in that, include: Step S1: Perform semantic recognition on the traffic generated by the intelligent computing cloud platform during the execution of computing power tasks; Step S2: Based on the semantic recognition results, map the traffic to the service quality queue corresponding to the semantic category, and based on the service quality queue, maintain a traffic occupancy list between each physical path and different traffic types; Step S3: Obtain the parallel topology information of the task to be scheduled by computing power, and determine the communication ring of the task to be scheduled by computing power when performing aggregate communication operation based on the parallel topology information, wherein the communication ring includes multiple participating nodes; Step S4: Based on the multiple participating nodes and the traffic occupancy list, select a physical path that meets the preset occupancy conditions for communication transmission for the communication ring corresponding to the computing power running task to be scheduled; Step S5: Continuously monitor the communication completion time of each transmission segment on the communication ring, and when it is determined that there is an abnormal transmission segment in the communication ring based on the communication completion time, adjust the node arrangement order in the communication ring to bypass the abnormal transmission segment, wherein the transmission segment consists of at least two participating nodes.

2. The method as described in claim 1, characterized in that, Step S1 includes at least one of the following: Step S1.1: Intercept the calling behavior of the computing power running task in the collection communication operation, and identify the traffic generated by the calling behavior as synchronization traffic as the first identification result; Step S1.2: Perform semantic classification and identification on the message arrival time interval and autocorrelation characteristics of the traffic of the computing power running task within the sliding time window to determine the second identification result.

3. The method as described in claim 2, characterized in that, Step S1.2 includes: Step S1.2.1: Aggregate the traffic into independent flows according to the packet 5-tuple, and count the packet arrival time interval sequence of the flow within the sliding time window; Step S1.2.2: Calculate the coefficient of variation of the message arrival time interval sequence and detect the autocorrelation peak of the message arrival time interval sequence at a preset lag. Step S1.2.3: If the coefficient of variation is less than a first preset threshold and the autocorrelation peak value is greater than a second preset threshold, then the semantic category of the flow corresponding to the traffic is determined to be synchronization traffic; Step S1.2.4: If the coefficient of variation is greater than or equal to the first preset threshold, and / or the autocorrelation peak value is less than or equal to the second preset threshold, then the traffic corresponding to the flow is determined to be elastic traffic or management traffic by combining the port range and historical tags in the packet quintuple.

4. The method as described in claim 2, characterized in that, Step S1 further includes: Step S1.3: When the traffic flow simultaneously contains both the first identification result and the second identification result, and the first identification result and the second identification result conflict, the first identification result shall prevail.

5. The method as described in claim 1, characterized in that, Step S2 includes at least one of the following: Step S2.1: Map the traffic whose semantic recognition result is synchronization traffic to a lossless high-priority transmission queue; Step S2.2: Map the traffic whose semantic recognition result is elastic traffic to a fallback transmission queue, and actively reduce the transmission rate of the fallback transmission queue according to the preset rate reduction control logic when an explicit congestion notification flag is detected. Step S2.3: Map the traffic whose semantic recognition result is management traffic to an independent management queue.

6. The method as described in claim 5, characterized in that, Step S2.2 includes: Step S2.2.1: Calculate the proportion of messages carrying explicit congestion notification flags received within the sliding time window, and reduce the transmission rate in an exponential backoff manner when the proportion of the messages exceeds a preset proportion threshold. Step S2.2.2: After the message ratio falls back below the preset ratio threshold, gradually restore the transmission rate.

7. The method as described in claim 1, characterized in that, Step S4 includes: Step S4.1: Obtain the synchronization traffic usage status and bandwidth usage of each physical path from the traffic usage list, wherein the traffic usage list is dynamically updated as the communication ring is established and released; Step S4.2: Based on the occupancy status and bandwidth of the synchronization traffic already carried by the multiple participating nodes and each physical path, select the available physical path with the lowest occupancy status from each physical path and allocate it to the communication ring as the physical path that meets the preset occupancy conditions.

8. The method as described in claim 1, characterized in that, Step S5 includes: Step S5.1: Calculate the communication completion time of each transmission segment in a single set communication operation of the communication ring, and calculate the mean and standard deviation of the completion time of all transmission segments in the communication ring in this communication. Step S5.2: In response to the fact that the difference between the communication completion time and the mean completion time of any transmission segment is greater than a preset multiple of the standard deviation, the corresponding transmission segment is determined as the target transmission segment; Step S5.3: If the target transmission segment continues to exist in a preset number of consecutive communication cycles, then the target transmission segment is determined to be the abnormal transmission segment.

9. The method as described in claim 1, characterized in that, Step S5 further includes: Step S5.4: Adjust the node arrangement order within the communication ring so that the participating nodes at both ends of the abnormal transmission segment are in a non-adjacent state; Step S5.5: The adjusted node arrangement order is issued through the topology configuration interface based on the rank mapping relationship provided by the aggregate communication library, so that subsequent aggregate communication operations are executed according to the adjusted node arrangement order.

10. A dynamic scheduling device for computing power network traffic based on a collective communication mode in an intelligent computing cloud platform, characterized in that, include: The identification module is used to semantically identify the traffic generated by the intelligent computing cloud platform during the execution of computing power tasks; The mapping module is used to map the traffic to the service quality queue corresponding to the semantic category based on the semantic recognition result, and maintain a traffic occupancy list between each physical path and different traffic types based on the service quality queue. The acquisition module is used to acquire the parallel topology information of the computing power task to be scheduled, and to determine the communication ring of the computing power task to be scheduled when performing a collection communication operation based on the parallel topology information, wherein the communication ring includes multiple participating nodes; The allocation module is used to select a physical path that meets preset occupancy conditions for communication transmission for the communication ring corresponding to the computing power running task to be scheduled, based on the multiple participating nodes and the traffic occupancy list. The monitoring module is used to continuously monitor the communication completion time of each transmission segment on the communication ring, and when it is determined that there is an abnormal transmission segment in the communication ring based on the communication completion time, the node arrangement order in the communication ring is adjusted to bypass the abnormal transmission segment, wherein the transmission segment consists of at least two participating nodes.

11. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the program implements the steps of the method for dynamic scheduling of computing power network traffic based on a collective communication mode in an intelligent computing cloud platform as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for dynamic scheduling of computing power network traffic based on a collective communication mode in an intelligent computing cloud platform as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The method includes computer instructions that, when executed by a processor, implement the steps of the method for dynamic scheduling of computing power network traffic based on a collective communication mode in an intelligent computing cloud platform as described in any one of claims 1 to 9.

Images