Intelligent Selection Method for RDMA Transfer Mode for Distributed AI Training

Abstract

This invention discloses an intelligent RDMA transmission mode selection method for distributed AI training. The method includes: acquiring multi-dimensional data features during deep learning training, including tensor structure features, data access pattern features, data timeliness features, data dependency features, and data change rate features; identifying the current training stage, including a forward propagation stage, a backward propagation stage, and a parameter update stage; selecting the optimal RDMA transmission mode matching the current training scenario based on the multi-dimensional data features and the training stage through an adaptive decision matrix, wherein the optimal RDMA transmission mode includes one or more combinations of RDMA Write, RDMA Read, Send / Receive, and Atomic operations; switching the RDMA transmission mode in real time according to dynamic changes in the network environment, employing a gradual traffic migration strategy to ensure uninterrupted training data flow during switching, and optimizing subsequent mode selection decisions through a historical performance learning mechanism.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to an intelligent selection method for RDMA transmission mode for distributed AI training. Background Technology

[0002] Remote Direct Memory Access (RDMA) technology serves as the core communication support for distributed artificial intelligence training systems and is widely used for parameter synchronization and data transmission in large-scale deep learning models.

[0003] Specifically, this technology system covers the entire process from data feature perception to transmission mode decision-making, including key aspects such as tensor structure analysis, access pattern recognition, timeliness detection, and dependency modeling. As the scale of deep learning models grows exponentially, single-machine training can no longer meet computational demands, making distributed training the industry standard. Communication efficiency directly impacts training performance. In existing technologies, the selection of RDMA transmission modes mainly relies on static configuration, empirical rule-based methods, and general optimization methods. Static configuration, with its preset operating modes, cannot adapt to dynamically changing training environments; empirical rule-based methods, based on simple features such as data size, do not consider the specific needs of deep learning training; and general optimization methods employ traditional network optimization strategies without specific adaptation for AI workloads. These methods suffer from systemic flaws in deep learning training scenarios, specifically: they fail to consider the differences in data characteristics during the training phase (forward propagation, back propagation, parameter updates), leading to a mismatch between mode selection and computational requirements; they ignore the impact of tensor structure (dense / sparse) and access patterns (sequential / random) on transmission performance; they lack dynamic awareness of the training state, making it impossible to adjust transmission strategies in real time; and they fail to achieve collaborative optimization between hardware accelerators and RDMA transmission, resulting in low resource utilization. Based on this, existing technologies exhibit significant performance bottlenecks in communication latency, bandwidth utilization, and CPU overhead. Particularly in large-scale sparse model training scenarios, redundant transmission and suboptimal mode selection result in communication overhead accounting for as much as 30-50%, severely restricting the efficiency improvement of distributed AI training. Summary of the Invention

[0004] The main objective of this invention is to provide an intelligent selection method for RDMA transmission mode for distributed AI training.

[0005] Another objective of this invention is to propose an intelligent selection device for RDMA transmission mode for distributed AI training.

[0006] The third objective of this invention is to provide a computer device.

[0007] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above objectives, a first aspect of the present invention proposes an intelligent selection method for RDMA transmission mode for distributed AI training, comprising:

[0009] S1, Obtain multi-dimensional data features during deep learning training, including tensor structure features, data access pattern features, data timeliness features, data dependency features, and data change rate features;

[0010] S2, identify the current training phase, which includes a forward propagation phase, a backward propagation phase, and a parameter update phase;

[0011] S3, based on the multi-dimensional data features and the training phase, select the optimal RDMA transmission mode that matches the current training scenario through an adaptive decision matrix. The optimal RDMA transmission mode includes one or more combinations of RDMA Write, RDMA Read, Send / Receive, and Atomic operations.

[0012] S4. The RDMA transmission mode is switched in real time according to the dynamic changes in the network environment. A gradual traffic migration strategy is adopted to ensure that the training data stream is not interrupted during the switching process, and the subsequent mode selection decision is optimized through a historical performance learning mechanism.

[0013] In one embodiment of the present invention, the step of obtaining multi-dimensional data features during deep learning training further includes:

[0014] S11 captures tensor creation and operation events through API interceptors, parses the tensor's dimension, sparsity, and memory continuity features, and constructs feature vectors containing dense and sparse tensors.

[0015] S12 uses a lightweight access tracking mechanism to record the access sequence of tensor elements, identify sequential access, random access and aggregate access modes, and map them to the corresponding RDMA operation type according to the access mode.

[0016] In one embodiment of the present invention, identifying the current training phase further includes:

[0017] S21, by intercepting the forward, backward, and optimizer.step API calls of the deep learning framework, and combining memory access patterns and computational resource usage characteristics, distinguishes between read-intensive operations in the forward propagation phase, computation-intensive operations in the backward propagation phase, and write-intensive operations in the parameter update phase.

[0018] S22, select the transmission mode according to the characteristics of the identified stage, wherein RDMA Read is preferred in the forward propagation stage, a hybrid mode is used in the backward propagation stage, and RDMA Write or Atomic operation is preferred in the parameter update stage.

[0019] In one embodiment of the present invention, the step of selecting the optimal RDMA transmission mode based on multi-dimensional data features and the training phase further includes:

[0020] S31, maintain a dynamically updated mode-stage mapping matrix, divide the transmission task into small data blocks, medium data blocks and large data blocks according to the data block size, and correspond to RDMA Send / Recv, RDMA Read and RDMA Write operations respectively;

[0021] S32 quantifies the communication density of the task based on the communication-computation ratio (CCR). When CCR < 0.1, coarse-grained fragmentation is used to reduce communication overhead, and when CCR > 10, fine-grained fragmentation is used to improve parallelism.

[0022] In one embodiment of the present invention, the step of dynamically switching the RDMA transmission mode according to changes in the network environment further includes:

[0023] S41, save the current transmission state and pre-establish RDMA connection resources for the new mode, and guide new transmission requests to the new connection through a gradual traffic migration strategy, while ensuring that the old connection tasks are completed;

[0024] S42 introduces a switching hysteresis mechanism and a cooldown period mechanism, which triggers mode switching only when the performance indicators exceed the threshold multiple times in a row, and sets a cooldown period after switching to avoid frequent oscillations.

[0025] In one embodiment of the present invention, it further includes:

[0026] S5 constructs a weighted graph model based on the network topology, where nodes represent computing devices, edges represent network connections, and weights represent bandwidth, latency, and congestion status. Based on the topology graph, it calculates the optimal communication path and allocates data fragments, so that communication-intensive fragments are preferentially transmitted through high-bandwidth, low-latency paths.

[0027] To achieve the above objectives, a second aspect of the present invention provides an intelligent selection device for RDMA transmission mode for distributed AI training, comprising:

[0028] The multi-dimensional data feature acquisition module is used to acquire multi-dimensional data features during the deep learning training process. The multi-dimensional data features include tensor structure features, data access pattern features, data timeliness features, data dependency relationship features, and data change rate features.

[0029] The training phase identification module is used to identify the current training phase, which includes the forward propagation phase, the backward propagation phase, and the parameter update phase.

[0030] An adaptive decision matrix module is used to select the optimal RDMA transmission mode that matches the current training scenario based on the multi-dimensional data features and the training phase. The optimal RDMA transmission mode includes one or more combinations of RDMA Write, RDMA Read, Send / Receive, and Atomic operations.

[0031] The dynamic transmission mode switching module is used to switch the RDMA transmission mode in real time according to the dynamic changes in the network environment. It adopts a gradual traffic migration strategy to ensure that the training data stream is not interrupted during the switching process, and optimizes the subsequent mode selection decision through a historical performance learning mechanism.

[0032] To achieve the above objectives, a third aspect of this application provides a computer device comprising a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory, for implementing an RDMA transfer mode intelligent selection method for distributed AI training as described in the first aspect embodiment.

[0033] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements an intelligent selection method for RDMA transmission mode for distributed AI training as described in the first aspect embodiment.

[0034] The embodiments of the present invention have the following beneficial effects:

[0035] The methods, apparatus, electronic devices, and computer-readable storage media of the present invention can dynamically select the optimal RDMA transmission mode based on the multi-dimensional data characteristics of deep learning training and the training stage, significantly improving the communication efficiency and overall performance of distributed AI training. Attached Figure Description

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 A flowchart of an intelligent RDMA transmission mode selection method for distributed AI training provided in an embodiment of the present invention;

[0038] Figure 2 This is an overall architecture diagram of the RDMA transmission mode intelligent selection system provided in an embodiment of the present invention;

[0039] Figure 3 This is a flowchart of tensor structure analysis technology provided in an embodiment of the present invention;

[0040] Figure 4 A flowchart of the data access pattern recognition technology provided in this embodiment of the invention;

[0041] Figure 5 A flowchart of the data timeliness detection technology provided in this embodiment of the invention;

[0042] Figure 6 A flowchart illustrating the data dependency graph construction technique provided in this embodiment of the invention;

[0043] Figure 7 A flowchart of the data change rate evaluation technology provided in this embodiment of the invention;

[0044] Figure 8 A flowchart of the training phase recognition technology provided in this embodiment of the invention;

[0045] Figure 9 A flowchart of the pattern-stage mapping matrix technology provided in this embodiment of the invention;

[0046] Figure 10 This is a flowchart of the data size hierarchical decision-making technology provided in an embodiment of the present invention;

[0047] Figure 11 This is a flowchart of batch processing awareness scheduling technology provided in an embodiment of the present invention;

[0048] Figure 12 A flowchart of priority-aware mapping technology provided in an embodiment of the present invention;

[0049] Figure 13 This is a flowchart of gradient sparsity-aware piecewise partitioning provided in an embodiment of the present invention;

[0050] Figure 14 This is a flowchart of the hierarchical perception segmentation process provided in an embodiment of the present invention;

[0051] Figure 15 This is a flowchart of the communication-computation ratio optimization fragmentation process provided in an embodiment of the present invention;

[0052] Figure 16 This is a flowchart of topology-aware fragmented routing provided in an embodiment of the present invention;

[0053] Figure 17 This is a flowchart of mixed-precision training slices provided in an embodiment of the present invention;

[0054] Figure 18 This is a flowchart of tensor compressed sensing transmission provided in an embodiment of the present invention;

[0055] Figure 19 A flowchart of large tensor layered transport provided in an embodiment of the present invention;

[0056] Figure 20 A flowchart of tensor semantic understanding provided for embodiments of the present invention;

[0057] Figure 21 This is a flowchart of tensor heat prediction provided in an embodiment of the present invention;

[0058] Figure 22 This is a flowchart of the operator computation mode perception provided in an embodiment of the present invention;

[0059] Figure 23 This is a flowchart of the zero-interruption mode switching provided in an embodiment of the present invention;

[0060] Figure 24 This is a flowchart of the mode switching trigger condition evaluation provided in an embodiment of the present invention;

[0061] Figure 25 This is a structural diagram of an RDMA transmission mode intelligent selection device for distributed AI training provided in an embodiment of the present invention. Detailed Implementation

[0062] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0063] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0064] The following description, with reference to the accompanying drawings, describes an intelligent selection method and apparatus for RDMA transmission mode for distributed AI training according to an embodiment of the present invention.

[0065] Example 1

[0066] This embodiment provides an intelligent selection method for RDMA transmission mode for distributed AI training. For example... Figure 1 As shown, the method includes the following steps:

[0067] S1, acquire multi-dimensional data features during deep learning training, including tensor structure features, data access pattern features, data timeliness features, data dependency features, and data change rate features.

[0068] Specifically, acquiring multi-dimensional data features during deep learning training is one of the core functions of the multi-dimensional data feature perception module in this invention. Its technical implementation is based on real-time acquisition and analysis of features such as tensor structure, data access mode, timeliness, dependency relationship and rate of change during training, providing key decision-making basis for subsequent intelligent selection of RDMA transmission mode.

[0069] In this embodiment of the invention, this step captures tensor operation events during training in real time through the API interceptor or computation graph parser of the deep learning framework. The system first parses the structural features of the tensor, including dimensions (e.g., ...). The tensor is evaluated based on several key characteristics, including shape regularity (e.g., whether it is a regular tensor) and storage format (e.g., dense or sparse storage). Secondly, an access tracking mechanism records the access sequence of tensor elements to identify their access patterns, such as sequential, random, or aggregate access. Timeliness features are determined by analyzing the time window in which data is used during the training phase to classify it as high-timeliness (e.g., activation values ​​in forward propagation), medium-timeliness (e.g., gradients in backpropagation), or low-timeliness (e.g., slowly updated weights). Dependency features are identified by constructing a directed dependency model of the computation graph to assess the predecessor and successor relationships between tensors, thus evaluating their "urgency" and "scope of influence" in the computation process. The rate of change feature is calculated by maintaining tensor version identifiers and content summaries to determine the update frequency per unit time. and range of change This is used to determine whether the data belongs to high, medium, or low rate of change.

[0070] Furthermore, the system quantifies the characteristics of tensor structures, such as the number of dimensions. sparsity (Zero element ratio), memory continuity (Whether it is contiguous storage), etc.; In access pattern recognition, the system uses the access sequence length. Access Interval Distribution In terms of timeliness, the system categorizes data into high, medium, and low levels based on data type and assigns a corresponding transmission queue priority to each level. And QoS parameters; during dependency graph construction, the system calculates the "urgency" of each transmission task. and "scope of influence" Used for scheduling priority; in rate of change evaluation, the system calculates the frequency of change. ,in The number of changes per unit time. The time window length and the magnitude of change This is used to measure the significance of data updates.

[0071] In this embodiment of the invention, this step is widely applied in distributed deep learning training systems, especially in the training of large-scale models (such as Transformer, GNN, etc.). The system perceives the data characteristics of different layer types (such as convolutional layers, fully connected layers, and attention layers) to provide differentiated strategies for data transmission in different scenarios. For example, in convolutional layers, the system identifies channel-first access patterns, providing a basis for subsequent channel-parallel transmission strategies; in attention layers, the system identifies block-based access characteristics, supporting block-first sharding strategies.

[0072] This step significantly improves the accuracy and adaptability of transmission decisions through precise extraction of multi-dimensional features. The system can select the most suitable transmission mode based on the tensor's structural characteristics (e.g., dense tensors are suitable for RDMA Write, sparse tensors are suitable for RDMA Read), optimize transmission order and parallelism based on access patterns, allocate transmission priorities based on timeliness, optimize scheduling strategies based on dependencies, and reduce redundant transmissions based on the rate of change. This multi-dimensional perception mechanism provides a solid data foundation for subsequent intelligent mapping of transmission modes, data sharding optimization, and real-time switching, enabling improvements in training performance. and resource utilization optimization The key premise.

[0073] Further, step S1 includes:

[0074] S11 captures tensor creation and manipulation events through API interceptors, parses the tensor's dimension, sparsity, and memory continuity features, and constructs feature vectors containing dense and sparse tensors.

[0075] Specifically, in some implementations, the tensor structure analysis technique of this invention captures the creation and operation events of tensors in deep learning frameworks through API interceptors, thereby extracting key features such as tensor dimension, sparsity, and memory continuity. Based on dynamic monitoring of tensor storage structures and combined with memory access pattern analysis, this technique constructs feature vectors containing both dense and sparse tensors, providing data support for subsequent RDMA transfer mode selection.

[0076] Specifically, the system captures tensor creation, copying, slicing, and concatenation operations in real time by inserting lightweight API interceptors into deep learning frameworks (such as PyTorch or TensorFlow). The interceptors record the tensor's shape, data type, storage format (e.g., contiguous or non-contiguous), and the proportion of non-zero elements (sparseness). The formula for calculating sparsity is: ,in This represents the number of zero elements in the tensor. This indicates the total number of elements in the tensor. This metric is used to determine whether a tensor is suitable for sparse coding transmission.

[0077] Furthermore, the system analyzes the memory continuity of tensors to determine whether they are stored contiguously or in a scattered manner. For tensors stored contiguously, the system can use RDMA Write for batch transfer to improve throughput; while for tensors stored non-contiguously, the system tends to use RDMA Read or Send / Recv to accommodate their irregular memory access patterns.

[0078] The feature vector constructed in this step contains multiple dimensions, such as tensor dimension, sparsity, memory continuity flag (0 or 1), and storage format (e.g., float32, float16, int8), forming a structured feature description used by the subsequent transmission pattern decision engine for pattern matching and optimization. This technique is widely applicable in distributed training scenarios such as parameter synchronization, gradient transmission, and activation value transfer. Especially when dealing with large-scale sparse models (such as embedding layers in recommender systems), it can significantly reduce invalid data transmission, improve communication efficiency, and thus optimize overall training performance.

[0079] S12 uses a lightweight access tracking mechanism to record the access sequence of tensor elements, identify sequential access, random access and aggregate access modes, and map them to the corresponding RDMA operation type according to the access mode.

[0080] Specifically, this step uses a lightweight access tracking mechanism to record the access sequence of tensor elements, identify sequential access, random access, and aggregated access patterns, and map the access patterns to the corresponding RDMA operation types. This is the core component of the multi-dimensional data feature perception mechanism in this invention. Its technical implementation is based on the dynamic monitoring and pattern recognition of tensor access behavior during deep learning training, thereby providing a key decision basis for subsequent RDMA transmission mode selection.

[0081] In this embodiment of the invention, this step involves deploying a lightweight access tracking module within a deep learning framework to intercept tensor read and write operations and record their access sequences. The tracking mechanism employs a low-overhead, event-driven approach, capturing the access location, timestamp, and access type (read / write) of tensor elements through API hooks or kernel-level memory access monitoring, and storing this information as an access log. The access log format typically includes a timestamp. Access address Access type and access length ,Right now ,in This indicates the total number of access events.

[0082] Furthermore, the system constructs an access pattern recognition model based on access logs, employing sliding window analysis and sequence pattern matching algorithms to identify three main access patterns: sequential access, random access, and aggregate access. Sequential access typically manifests as read / write operations on contiguous addresses, with access intervals... Smaller and address difference Fixed step size; random access is characterized by irregular address differences and large access intervals; aggregated access is characterized by multiple access requests occurring in a short period of time, usually related to reduction operations (such as AllReduce).

[0083] Optionally, the system maps the identified access patterns to RDMA operation types. For example, sequential access patterns are preferentially mapped to RDMA Write operations to achieve efficient batch transmission; random access patterns are mapped to RDMA Read operations to support on-demand retrieval; and aggregated access patterns are mapped to Send / Receive or Atomic operations to accommodate synchronization and atomic update requirements. This mapping relationship can be dynamically optimized based on preset rules or through machine learning models, thereby improving transmission efficiency and resource utilization.

[0084] This step is widely used in distributed training scenarios, especially in Convolutional Neural Networks (CNNs) and Transformer models, where tensor access patterns differ significantly across layers. This step enables fine-grained control over communication behavior. Its technical benefits include significantly reducing communication latency, improving bandwidth utilization, and minimizing unnecessary data transmission, thereby enhancing overall training throughput and convergence efficiency.

[0085] S2, identify the current training phase, which includes a forward propagation phase, a backward propagation phase, and a parameter update phase.

[0086] Specifically, in some implementations, training phase identification techniques utilize the API interception mechanism of deep learning frameworks, combined with memory access patterns and computational resource usage characteristics, to accurately identify the three phases of forward propagation, back propagation, and parameter update. This technique categorizes operations based on typical behavioral characteristics during deep learning training. For example, in the forward propagation phase, the system primarily exhibits read-intensive operations, involving access to model weights and input data; in the back propagation phase, the system exhibits computation-intensive operations, requiring gradient calculations and access to activation values ​​generated during forward propagation; and in the parameter update phase, the system primarily performs write-intensive operations, involving the synchronization and updating of model parameters.

[0087] Furthermore, this identification process is typically implemented through an event-driven mechanism. The system sets up key event hooks (such as `forward()`, `backward()`, `optimizer.step()`, etc.) during the training process, inserting monitoring logic before and after these function calls to record the current stage's computation graph state, memory access patterns, and resource usage. For example, during the backpropagation stage, the system detects the generation and propagation path of the gradient tensor, combining this with memory read / write patterns to identify the characteristics of this stage. During the parameter update stage, the system observes the optimizer's modification behavior of the model parameters to determine whether to enter that stage.

[0088] Optionally, the system can also combine hardware performance counters and software-level log analysis to perform multi-dimensional verification of the training phase. For example, by analyzing metrics such as CPU / GPU utilization, memory bandwidth usage, and RDMA queue status, it can help determine which training phase is currently in. Specifically, the forward propagation phase typically has high memory read bandwidth requirements, while the parameter update phase exhibits a high write operation frequency.

[0089] This step plays a crucial role in the overall system decision-making, providing contextual information for subsequent RDMA transmission mode selection. By identifying the current training phase, the system can dynamically adjust its transmission strategy. For example, it can prioritize RDMA Read during the forward propagation phase to reduce remote write overhead, and prioritize RDMA Write or Atomic operations during the parameter update phase to ensure synchronization efficiency. This phase-aware mechanism significantly improves the accuracy of transmission decisions, thereby optimizing the overall communication efficiency and resource utilization of distributed AI training.

[0090] Further, step S2 includes:

[0091] S21, by intercepting the forward, backward, and optimizer.step API calls of the deep learning framework, and combining memory access patterns and computational resource usage characteristics, distinguishes between read-intensive operations in the forward propagation phase, computation-intensive operations in the backward propagation phase, and write-intensive operations in the parameter update phase.

[0092] Specifically, in some implementations, the training phase identification technology of this invention intercepts key API calls such as forward, backward, and optimizer.step of deep learning frameworks, and combines memory access patterns and computational resource usage characteristics to accurately identify the three main phases of the training process (forward propagation, backward propagation, and parameter update). Based on typical behavior patterns of deep learning training, this technology employs an event-driven monitoring mechanism to capture key function call sequences during model execution in real time, and determines the current training phase through context analysis.

[0093] Specifically, during the forward propagation phase, the system primarily observes read operations on model weights and input data, exhibiting read-intensive characteristics. This typically involves loading and computing a large number of tensors. In this phase, the system prioritizes RDMA Read operations to reduce remote read latency. During the backpropagation phase, the system detects gradient computation and activation value access, exhibiting computationally intensive characteristics. This involves numerous floating-point operations and reading intermediate results. In this phase, the system tends to use a hybrid mode, combining RDMA Write and Send / Recv operations to balance latency and throughput. During the parameter update phase, the system identifies write operations on model parameters, exhibiting write-intensive characteristics. This typically involves gradient synchronization and parameter updates. In this phase, the system prioritizes RDMA Write or Atomic operations to ensure data consistency and reduce CPU intervention.

[0094] Furthermore, the system constructs a multi-dimensional feature vector for stage identification by analyzing memory access patterns (such as read frequency, write pattern, and access locality) and computational resource usage characteristics (such as GPU utilization, computation queue depth, and memory bandwidth usage). For example, the memory read frequency during the forward propagation stage... Typically higher than the write frequency The parameter update phase, on the other hand, proceeds in the opposite way. The system sets feature thresholds (such as...) (When the time is determined to be forward propagation), combined with the API call sequence within the time window, high-precision stage identification is achieved.

[0095] This step plays a crucial role in the invention, providing a foundation for subsequent intelligent mapping of transmission modes. By accurately identifying the training phase, the system can dynamically adjust the RDMA operation mode, significantly improving communication efficiency, reducing idle waiting time of computing resources, and thus increasing overall training throughput and resource utilization.

[0096] S22, select the transmission mode according to the characteristics of the identified stage, wherein RDMA Read is preferred in the forward propagation stage, a hybrid mode is used in the backward propagation stage, and RDMA Write or Atomic operation is preferred in the parameter update stage.

[0097] Specifically, the step of selecting the transmission mode based on the identified stage characteristics in this invention is the core control logic for optimizing the communication efficiency of distributed AI training. This technology establishes a mapping relationship between the stage characteristics of the deep learning training process and the performance differences of RDMA operation modes. By dynamically identifying the training stage and matching the optimal transmission mode, it reduces communication latency and increases throughput.

[0098] In embodiments of this invention, the system intercepts key APIs (such as forward, backward, optimizer.step, etc.) in the deep learning framework and combines memory access patterns and computational resource usage characteristics to accurately identify whether the current stage is forward propagation, backward propagation, or parameter update. The forward propagation stage mainly involves reading model weights and input data, which is a read-intensive operation. Therefore, the system prioritizes RDMA Read mode to reduce the overhead of remote writes and improve data acquisition efficiency. The backward propagation stage involves gradient calculation and activation value return, which is a mixed read-write scenario. The system adopts a hybrid mode, dynamically selecting RDMA Write or Send / Recv based on data size and access mode to balance latency and throughput. The parameter update stage primarily involves writing model parameters. The system prioritizes RDMA Write or Atomic operations. Atomic operations are suitable for scenarios requiring atomic updates (such as gradient accumulation), while RDMA Write is used for batch parameter synchronization to reduce CPU intervention and improve transmission efficiency.

[0099] Key parameters involved in this step include: the confidence threshold for identification during the training phase (e.g., API call sequence matching rate must reach over 90% before a phase is confirmed), the minimum data block size for transmission mode switching (e.g., 4KB as the lower bound for medium data blocks), the priority level of RDMA operations (e.g., high, medium, and low), and QoS parameters (e.g., bandwidth allocation ratio, latency tolerance, etc.). Based on these parameters, the system constructs a mode-phase mapping matrix and, combined with a data size-based hierarchical decision-making mechanism, achieves intelligent selection of the transmission mode.

[0100] This step is widely used in distributed deep learning training systems, especially in multi-node, multi-GPU cluster environments. For example, when training large language models or recommendation systems, the forward propagation phase requires frequent reading of weights, the backpropagation phase requires efficient gradient transmission, and the parameter update phase requires ensuring synchronous consistency. Through the intelligent mapping mechanism in this step, the system can significantly reduce communication latency and improve training throughput, especially in high-bandwidth, low-latency InfiniBand or RoCE networks.

[0101] Furthermore, the technical benefits of this step are reflected in the following: in a typical ResNet-50 training task, the communication latency in the forward propagation stage can be reduced by approximately 25%, the throughput in the backpropagation stage can be increased by up to 35%, and the synchronization efficiency in the parameter update stage can be improved by more than 20%. Through stage-aware transmission mode selection, the system achieves efficient collaboration between communication and computation, significantly improving the overall performance and resource utilization of distributed AI training.

[0102] S3, based on the multi-dimensional data features and the training phase, select the optimal RDMA transmission mode that matches the current training scenario through an adaptive decision matrix. The optimal RDMA transmission mode includes one or more combinations of RDMA Write, RDMA Read, Send / Receive, and Atomic operations.

[0103] Specifically, this step, based on the aforementioned multi-dimensional data features and the training stage, selects the optimal RDMA transmission mode matching the current training scenario through an adaptive decision matrix. This is one of the core mechanisms for achieving efficient distributed AI training communication in this invention. Its technical implementation principle is based on real-time perception of multi-dimensional data features during deep learning training and dynamic identification of training stages. Combined with a preset mode-stage mapping matrix, it achieves intelligent selection and combination of RDMA Write, RDMA Read, Send / Receive, and Atomic operations.

[0104] In this embodiment of the invention, the system first extracts key features of the current transmission task through a multi-dimensional data feature perception module, including tensor dimension, sparsity, memory continuity, access pattern (sequential, random, aggregated), timeliness level, data dependency, and rate of change. These features are encoded into feature vectors and input into the decision engine. Simultaneously, the training phase identification module intercepts API calls from deep learning frameworks (such as forward, backward, optimizer.step, etc.) and, combined with computational resource usage features, identifies whether the current stage is forward propagation, backward propagation, or parameter update.

[0105] Optionally, the system maintains an adaptive decision matrix, which takes the training phase and data features as input dimensions and outputs recommended RDMA operation modes and their priorities. For example, in the forward propagation phase, model weights are typically read-intensive, and the system prioritizes RDMA Read operations; while in the backpropagation phase, gradient data has high timeliness and high update frequency, and the system tends to use RDMA Write or Atomic operations to ensure synchronization. Furthermore, the system also performs hierarchical decisions based on data block size (e.g., <4KB, 4KB-1MB, >1MB) to balance latency and throughput.

[0106] Furthermore, this step introduces a dynamic learning mechanism. The system continuously records the actual transmission performance (such as latency, throughput, and CPU utilization) for each feature-pattern combination and optimizes the decision matrix through A / B testing, enabling it to adapt to changes in different model structures, training strategies, and hardware environments. This mechanism follows the RDMA over ConvergedEthernet (RoCE) v2 protocol standard and is compatible with the InfiniBand network architecture, ensuring deployment in a variety of high-performance computing environments.

[0107] The technical advantage of this step lies in the fact that by jointly making decisions based on the training phase and data features, the system can significantly improve communication efficiency, reduce unnecessary transmission overhead, and increase the overlap between computation and communication, thereby achieving [advantages] in large-scale distributed training. This mechanism improves performance. Simultaneously, it enhances the system's adaptability, reduces reliance on manual configuration, and improves the overall stability and scalability of training.

[0108] Further, step S3 includes:

[0109] S31, maintain a dynamically updated mode-stage mapping matrix, divide the transmission task into small data blocks, medium data blocks and large data blocks according to the data block size, and correspond to RDMA Send / Recv, RDMA Read and RDMA Write operations respectively.

[0110] Specifically, in some implementations, the data size hierarchical decision-making technology of this invention achieves intelligent selection of RDMA operation modes for data blocks of different sizes by maintaining a dynamically updated mode-stage mapping matrix. Based on in-depth analysis of the performance characteristics of RDMA operations, this technology divides transmission tasks into three categories according to data block size: small data blocks (<4KB), medium data blocks (4KB-1MB), and large data blocks (>1MB), and maps them to RDMA Send / Recv, RDMA Read, and RDMAWrite operations respectively, thereby achieving optimal performance matching in data transmission of different scales.

[0111] In this embodiment of the invention, the system first obtains the size information of the data blocks to be transmitted through memory monitoring and task scheduling interfaces. Then, based on preset stratification thresholds (4KB and 1MB), the data blocks are classified, and the currently maintained mode-stage mapping matrix is ​​queried to determine the corresponding RDMA operation type. This matrix not only contains static mapping rules but also supports dynamic updates. The system performs A / B testing and adaptive adjustments to the mapping relationship based on actual transmission performance (such as latency and throughput) to adapt to the optimal operation selection under different hardware platforms and network environments.

[0112] Furthermore, the hierarchical thresholds used by the system are not fixed but dynamically adjusted based on the performance characteristics of the underlying network hardware (such as InfiniBand or RoCEv2). For example, in low-latency networks, the threshold for small data blocks can be appropriately increased to 8KB to reduce frequent Send / Recv operations; while in high-bandwidth, low-throughput scenarios, the threshold for large data blocks can be extended to 2MB to fully utilize the batch transmission advantages of Write operations. In addition, the system also supports data aggregation and fragmentation strategies, aggregating small data blocks to reduce the number of transmissions and fragmenting ultra-large data blocks for parallel transmission, thereby improving overall transmission efficiency.

[0113] This technology is widely applicable to scenarios such as parameter synchronization, gradient transfer, and activation value propagation in distributed deep learning training. For example, during the backpropagation phase, gradient data is typically of medium size, and the system prioritizes RDMA Read operations to balance latency and throughput; while during the parameter update phase, global gradients are often large blocks, and the system uses RDMA Write for efficient transmission. This technology is particularly suitable for multi-node, multi-GPU training clusters, significantly reducing communication overhead and improving training throughput.

[0114] Furthermore, the technical benefits of this step are reflected in significantly reduced communication latency and improved bandwidth utilization. By mapping data blocks of different sizes to the most suitable RDMA operations, the system can reduce unnecessary operational overhead and improve transmission efficiency. Experiments show that in typical distributed training scenarios, this technique can reduce communication latency by 20%-40% and increase overall training throughput by 15%-30%.

[0115] S32 quantifies the communication density of the task based on the communication-computation ratio (CCR). When CCR < 0.1, coarse-grained fragmentation is used to reduce communication overhead, and when CCR > 10, fine-grained fragmentation is used to improve parallelism.

[0116] Specifically, this step involves a dynamic data sharding strategy based on the communication-computation ratio (CCR), which is a core component of the deep learning algorithm-aware dynamic data sharding optimization module in this invention. Its technical principle is based on the balance theory of computational and communication resources. By quantifying the communication density of the task, it intelligently selects either coarse-grained or fine-grained sharding strategies, thereby optimizing the communication efficiency of distributed training under different computational load characteristics.

[0117] In this embodiment of the invention, the system first collects the computational density (ratio of computational load to data volume per unit) and communication overhead (including transmission latency, bandwidth usage, etc.) of the current task through the performance monitoring module, and calculates the CCR index accordingly, which is the ratio of communication volume required per unit of computation, as shown in the formula: ,in Indicates communication overhead. Indicates computational cost. When When the task is computationally intensive and communication overhead is relatively low, the system employs a coarse-grained data fragmentation strategy, dividing the data into larger blocks for transmission to reduce communication frequency and startup overhead, thereby lowering overall communication latency. Conversely, when... When task communication is intensive, the system adopts fine-grained fragmentation to divide the data into multiple small blocks for parallel transmission, thereby improving communication parallelism and bandwidth utilization.

[0118] This strategy is widely applicable to distributed training scenarios of varying scales in practical applications. For example, in large-scale matrix multiplication (such as QKV calculation in Transformer models), if the computational density is high, the system will use coarse-grained sharding to reduce communication frequency; while in small-batch synchronous updates (such as gradient synchronization in sparse models), if the communication overhead is significant, the system will use fine-grained sharding to improve parallel efficiency. Furthermore, the system supports dynamically adjusting the CCR threshold based on the hardware environment (such as RDMA bandwidth and latency characteristics) to adapt to the performance characteristics of different clusters.

[0119] The technical value of this step lies in introducing quantitative indicators. This approach achieves a dynamic balance between communication and computing resources, effectively reducing the impact of communication bottlenecks on training performance. Experiments show that this strategy can improve communication efficiency by 20%-40%, significantly reduce training latency, and increase overall throughput.

[0120] S4. The RDMA transmission mode is switched in real time according to the dynamic changes in the network environment. A gradual traffic migration strategy is adopted to ensure that the training data stream is not interrupted during the switching process, and the subsequent mode selection decision is optimized through a historical performance learning mechanism.

[0121] Specifically, the core control mechanism for achieving efficient and stable distributed AI training communication is to switch the RDMA transmission mode in real time according to the dynamic changes in the network environment, adopt a progressive traffic migration strategy to ensure that the training data stream is not interrupted during the switching process, and optimize the subsequent mode selection decision through a historical performance learning mechanism. Its technical implementation is based on the synergistic optimization of zero-interruption mode switching and historical performance learning.

[0122] In this embodiment of the invention, this step achieves seamless migration of RDMA operating modes through zero-interruption mode switching technology. The system first saves the current transmission state, including ongoing transmission tasks, queue status, and context information, ensuring that no data is lost or the training process is interrupted during the switch. Subsequently, the system pre-establishes RDMA connection resources for the target transmission mode, including queue pairs (QP), memory registration (MR), and completion channels (CQ), ensuring the new connection is fully ready. During the switch, the system employs a gradual traffic migration strategy, directing newly generated transmission requests to the new connection while ensuring the completion of tasks on the old connection, achieving a smooth transition from the old mode to the new mode. After confirming that all data has been safely transmitted and the new connection is operating stably, the system releases the old connection resources, thereby achieving a "zero-interruption" switch.

[0123] Furthermore, the system introduces a handover hysteresis mechanism and a cooldown period mechanism to avoid frequent handovers caused by short-term network fluctuations. The handover hysteresis mechanism requires that a handover be triggered only after multiple consecutive performance indicators (such as latency and throughput) exceed a preset threshold. The cooldown period mechanism sets a certain time window after a handover is completed and does not respond to new handover requests. In addition, the system records the performance of each mode switch, including transmission latency, through a historical performance learning mechanism. Throughput and resource utilization rate And based on these indicators, a decision-making model is built to optimize subsequent pattern selection strategies.

[0124] This step is widely applicable to large-scale distributed training scenarios, especially when network load changes dynamically during training, model structures are complex, and data features are diverse. For example, in the early stages of training when gradient updates are frequent, the system may switch from RDMA Write to Send / Recv to reduce latency; while in the later stages of training when gradients tend to be sparse, the system may switch to RDMA Read or sparse coding transmission to reduce communication overhead. This mechanism is well-adapted to multi-node clusters, heterogeneous computing environments (such as mixed deployment of GPUs and CPUs), and different network topologies (such as fat trees and rings).

[0125] The technical advantage of this step lies in its ability to adaptively switch between RDMA transmission modes without interrupting the training data stream by real-time sensing of network status and combining it with historical performance data. This significantly improves communication efficiency and training stability. Experiments show that this mechanism can reduce performance jitter caused by mode switching, increase overall training throughput by 15%-30%, and effectively reduce the risk of training interruption due to switching failures, thereby enhancing the system's robustness and intelligence.

[0126] The intelligent RDMA transmission mode selection method for distributed AI training according to the embodiments of the present invention can intelligently select the optimal RDMA transmission mode based on the data characteristics, stage and network environment of deep learning training, which significantly improves the communication efficiency and overall performance of distributed AI training.

[0127] Furthermore, S4 includes:

[0128] S41, save the current transmission state and pre-establish RDMA connection resources for the new mode, and guide new transmission requests to the new connection through a gradual traffic migration strategy, while ensuring that the old connection tasks are completed.

[0129] Specifically, this step saves the current transmission state and pre-establishes RDMA connection resources for the new mode. It guides new transmission requests to the new connection through a progressive traffic migration strategy, while ensuring that the old connection tasks are completed. This is the core implementation link of the zero-interruption mode switching technology in this invention, which aims to achieve smooth switching of RDMA transmission mode and avoid training interruption or data loss caused by mode switching, thereby ensuring the continuity and stability of distributed AI training.

[0130] In this embodiment of the invention, the system first saves the status of the currently ongoing transmission task through the RDMA connection state management module, including transmission queue status (such as QP status, CQ status), memory registration information (MR registration status, memory address mapping), completed channel status, and context information (such as transmission offset, remaining data amount, transmission identifier, etc.). This status information is serialized and cached in the system status table to ensure that unfinished transmission tasks can be resumed or continued during the handover process.

[0131] Subsequently, the system pre-establishes the RDMA connection resources required for the new mode in the background, including creating new queue pairs (QPs), registering new memory regions (MRs), configuring completion queues (CQs), and setting transmission parameters (such as maximum transmission unit (MTU), maximum receive data size (MRDS), and transmission window size). This process follows the RoCE v2 or InfiniBand protocol standard to ensure that the new connection meets the performance requirements of the current network environment.

[0132] In the incremental traffic migration strategy, the system employs a dual-channel parallel mechanism, gradually directing newly arriving transmission requests to new connections while continuing to process unfinished tasks on older connections. The migration process is scheduled based on the type and priority of the transmission requests; for example, high-priority weight update requests are preferentially transmitted through new connections, while low-priority activation value requests can be migrated with a delay. The migration ratio can be configured via system parameters (such as migration thresholds). Adjustments are made to balance switching speed and system stability.

[0133] After confirming that all old connection tasks have been completed or safely migrated, the system releases old connection resources, including closing QP, deregistering MR, cleaning up CQ, and completing channels. The entire switchover process is transparent to upper-layer applications and will not affect the continuity of the training process. In addition, the system has a built-in anomaly detection and rollback mechanism. If performance degradation or transmission anomalies are detected during the switchover, it can immediately roll back to the original mode to ensure system reliability.

[0134] This step plays a crucial role in the overall technical solution, serving as a key guarantee for real-time switching of transmission modes. Through state preservation, resource pre-establishment, and gradual migration, the system can achieve efficient and stable RDMA mode switching in dynamically changing training environments, thereby improving overall training efficiency and enhancing system robustness.

[0135] S42 introduces a switching hysteresis mechanism and a cooldown period mechanism, which triggers mode switching only when the performance indicators exceed the threshold multiple times in a row, and sets a cooldown period after switching to avoid frequent oscillations.

[0136] Specifically, the present invention introduces a switching lag mechanism and a cooling-off period mechanism. The mode switching is triggered only when the performance index exceeds the threshold multiple times in a row. The step of setting a cooling-off period after the switching to avoid frequent oscillations is the key control logic in the real-time switching module of transmission mode. It aims to improve system stability and decision robustness, prevent invalid mode switching caused by instantaneous performance fluctuations, thereby reducing system overhead and improving overall training efficiency.

[0137] In this embodiment of the invention, the mechanism is implemented by setting a switching hysteresis counter and a cooldown timer. The system continuously monitors performance metrics under the current RDMA transmission mode, including transmission latency. Throughput Resource utilization rate and error rate etc. When a certain performance index is continuous For example Exceeding a preset threshold (such as a delay threshold) The system only triggers a mode switching decision when the system experiences a lag. This lag mechanism avoids erroneous switching caused by momentary network jitter or computational load fluctuations, ensuring that the switching decision is based on a stable performance trend.

[0138] Furthermore, after completing a mode switch, the system will start a cooldown timer and set the cooldown duration. (For example The cooldown period is set to a certain number of seconds, during which no new handover requests will be responded to. This cooldown period is based on an assessment of the time required for RDMA connection establishment, resource release, and context switching, ensuring the system has sufficient time to operate stably after a handover and avoiding resource waste and performance degradation caused by frequent handovers.

[0139] Optionally, the system supports dynamic adjustment of the lag counter and cooldown parameters, adaptively configuring them based on the current training stage, model type, and network load. For example, in the early stages of training, the system can set a shorter lag counter ( ) and shorter cooling period ( The system can quickly respond to performance changes by setting a lag counter (seconds); and in the later stages of training, the lag counter can be extended ( ) and cooldown period ( (seconds) to reduce unnecessary switching.

[0140] In practical applications, this mechanism is particularly suitable for large-scale distributed training scenarios, such as training deep neural network models with billions of parameters. In multi-node RDMA communication, the network state may experience short-term anomalies due to load changes, congestion control, or hardware performance fluctuations. This mechanism can effectively filter noise and ensure that mode switching only occurs when necessary, thereby improving the overall communication efficiency and training stability of the system.

[0141] In summary, this step, by introducing a switching lag and cooling-off period mechanism, achieves intelligent control of RDMA transmission mode switching, effectively preventing system oscillations, improving the accuracy of decision-making and the robustness of the system. It is an important guarantee for achieving efficient and stable communication scheduling in the real-time transmission mode switching module of this invention.

[0142] The intelligent RDMA transmission mode selection method for distributed AI training according to the embodiments of the present invention can intelligently select the optimal RDMA transmission mode based on the data characteristics, stage and network environment of deep learning training, which significantly improves the communication efficiency and overall performance of distributed AI training.

[0143] S5 constructs a weighted graph model based on the network topology, where nodes represent computing devices, edges represent network connections, and weights represent bandwidth, latency, and congestion status. Based on the topology graph, it calculates the optimal communication path and allocates data fragments, so that communication-intensive fragments are preferentially transmitted through high-bandwidth, low-latency paths.

[0144] Specifically, in some implementations, constructing a weighted graph model based on the network topology is one of the core steps of the topology-aware fragmented routing technology in this invention. This technology is implemented based on the physical network structure of a distributed training cluster. It obtains the connectivity relationships between nodes using network discovery tools (such as OSPF, BGP, LLDP, or a custom topology discovery protocol) and models them as a weighted graph. ,in Represents the set of computing device nodes. This represents the set of network connection edges between nodes. This represents the set of weights on the edge. Weights It consists of multiple network performance metrics, including bandwidth. (Unit: Gbps) Latency (Units are milliseconds or microseconds) and congestion status (This can be expressed as the current link utilization percentage or queue depth).

[0145] Furthermore, the system employs shortest path algorithms from graph theory (such as Dijkstra's algorithm or A* algorithm) to calculate the optimal communication path between nodes, where the path cost function is defined as:

[0146] in , , These are adjustable weighting coefficients used to balance the impact of bandwidth, latency, and congestion status on path selection. In practical deployments, these coefficients can be dynamically adjusted based on the communication intensity of the training task and the characteristics of the network hardware.

[0147] Optionally, the system may also consider the computational load of nodes when constructing the graph model. and memory bandwidth This avoids allocating communication-intensive fragments to resource-constrained nodes. Based on this graph model, the system distributes data fragments to the optimal path, ensuring that communication-intensive fragments are preferentially transmitted through high-bandwidth, low-latency, and low-congestion links. This significantly reduces communication latency and improves the overall throughput and resource utilization of distributed training. This step is particularly important in large-scale distributed training, especially in multi-node, multi-level network topologies (such as fat trees, Dragonfly, and ring topologies), effectively avoiding network bottlenecks and achieving efficient collaboration between communication and computation.

[0148] The intelligent selection method for RDMA transmission mode for distributed AI training in this invention constructs a weighted graph model that reflects bandwidth, latency, and congestion status, and optimizes communication paths and data fragment allocation based on this topology graph. This allows communication-intensive data to be transmitted preferentially through high-bandwidth, low-latency paths, further improving the data transmission efficiency and system resource utilization among multiple nodes in distributed AI training.

[0149] Example 2

[0150] This invention relates to an intelligent RDMA transfer mode selection system for distributed AI training, such as... Figure 2 As shown, the system specifically includes:

[0151] like Figure 2 The diagram shows the overall architecture of the intelligent RDMA transmission mode selection system according to an embodiment of the present invention. This system is located between the deep learning framework and the RDMA transmission layer. It analyzes the training workload characteristics through an intelligent transmission decision engine and selects the optimal RDMA transmission mode.

[0152] Furthermore, the system specifically includes:

[0153] 1. Multi-dimensional data feature perception module, used to analyze data features during deep learning training through various technical means, providing a basis for decision-making in subsequent transmission mode selection.

[0154] Furthermore, the multi-dimensional data feature perception module includes:

[0155] (1) Tensor structure analysis unit, used to parse the dimension, shape and storage format of tensors by intercepting tensor operations of deep learning frameworks.

[0156] Specifically, such as Figure 3 As shown, this unit captures tensor creation and manipulation events through API interceptors and analyzes the internal structure of tensors. The system distinguishes between dense tensors (which store all elements in contiguous memory) and sparse tensors (which store only non-zero elements and indices), and analyzes memory layout (contiguous or scattered) and shape regularity. Finally, it constructs feature vectors containing information such as dimension, sparsity, and memory continuity to provide a basis for transmission decisions. Based on the analysis results, the system selects the most suitable transmission mode for tensors with different characteristics: dense, contiguous tensors are suitable for RDMA Write batch transmission; sparse tensors are more suitable for RDMA Read on-demand retrieval or dedicated compressed encoding transmission. This refined structural analysis can significantly improve transmission efficiency, especially for large-scale sparse models.

[0157] (2) Data access pattern recognition unit, used to identify different access patterns such as random access, sequential access, and aggregate access by monitoring tensor access operations.

[0158] Specifically, such as Figure 4 As shown, this unit employs a lightweight access tracking mechanism to record the access sequences of tensor elements and perform pattern analysis. The system identifies three main access patterns: sequential access (such as scanning the input feature map by a convolutional layer), random access (such as lookup in an embedding layer), and aggregation access (such as reduction operations). For different access patterns, the system automatically maps to the most suitable RDMA operation: sequential access prioritizes RDMA Write for efficient batch transmission; random access prioritizes RDMA Read for on-demand data acquisition; and aggregation access prioritizes RDMA Send / Recv for real-time processing. This access pattern awareness technology is significantly superior to traditional static rule selection, adapting to complex AI workload characteristics, reducing unnecessary data transmission, and improving data locality utilization.

[0159] (3) Data timeliness detection unit, used to distinguish the timeliness requirements of different types of data and provide priority basis for transmission scheduling.

[0160] like Figure 5 As shown, this unit implements intelligent transmission priority allocation based on the time sensitivity differences of different data types in deep learning training. The system analyzes the computation graph and data flow to identify data types and classify them into three timeliness levels: high-timeliness data (such as forward propagation activation values) needs to be transmitted immediately; medium-timeliness data (such as gradients) needs to be completed within the current iteration; and low-timeliness data (such as slowly updated weights) can be transmitted asynchronously. The system assigns corresponding RDMA transmission queue priorities and QoS parameters to each data type to ensure timely transmission of critical data while rationally scheduling low-priority transmissions to avoid network congestion. This timeliness-aware mechanism effectively prevents idle computing resources during training, improving overall training throughput.

[0161] (4) Data dependency graph construction unit, which is used to construct a dependency graph between data by analyzing the computation graph and to guide transmission scheduling.

[0162] like Figure 6 As shown, this unit constructs a data transmission dependency model by analyzing the computation graph of the deep learning framework. The system extracts operator nodes and their input-output tensor information to construct a directed dependency graph, and then applies a critical path analysis algorithm to identify critical paths in the computation process. The system calculates the "urgency" and "impact range" indicators for each transmission task, and allocates transmission priorities and selects transmission modes based on these indicators: transmissions on critical paths preferentially select low-latency operation modes (such as RDMA Write), while transmissions on non-critical paths can select high-throughput but higher-latency batch transmission modes. This dependency-aware scheduling mechanism effectively reduces blocking waiting time during training, optimizes the overlap between computation and communication, and improves overall training efficiency.

[0163] (5) Data change rate evaluation unit, used to quantify the update frequency and change magnitude of different types of data during the training process.

[0164] like Figure 7 As shown, this unit optimizes the transmission strategy to reduce redundant transmission by monitoring data changes during training. The system maintains version identifiers and content summaries for key tensors, and calculates the frequency of change (the number of significant changes per unit time) and the magnitude of change (the size of the difference in updates). Based on these metrics, the system classifies data into high-rate-of-change, medium-rate-of-change, and low-rate-of-change data, and selects corresponding transmission strategies: high-rate-of-change data is transmitted in its entirety; medium-rate-of-change data is transmitted incrementally; and low-rate-of-change data is transmitted on demand. This rate-of-change-aware transmission optimization significantly reduces network traffic, especially in the later stages of training large models when many parameters tend to stabilize, saving 30-50% of communication overhead while maintaining training convergence and accuracy.

[0165] 2. The intelligent mapping module for transmission modes perceived during the training phase is used to intelligently select the most suitable RDMA transmission mode based on the characteristics of different stages of deep learning training.

[0166] Furthermore, the intelligent mapping module for transmission patterns perceived during the training phase includes:

[0167] (1) Training phase identification unit, used to accurately identify the three phases of forward propagation, backward propagation and parameter update in deep learning training.

[0168] Specifically, such as Figure 8 As shown, this unit intercepts key API calls of deep learning frameworks (such as forward, backward, optimizer.step, etc.) and, combined with memory access patterns and computational resource usage characteristics, accurately identifies the three main training stages. The forward propagation stage is read-intensive, primarily reading model weights and input data; the backpropagation stage is computationally intensive, requiring gradient calculation and activation value access; and the parameter update stage is write-intensive, requiring updating and synchronizing model weights. The system selects the most suitable RDMA transfer mode based on the identified stage characteristics: forward propagation prioritizes RDMA Read; backpropagation uses a hybrid mode; and parameter updates prioritize RDMA Write or Atomic operations. This training stage-aware transfer optimization allows the system to dynamically adjust its transfer behavior as the training progresses, improving transfer efficiency and computational resource utilization.

[0169] (2) Mode-stage mapping matrix unit, used to preset the optimal RDMA operation mode for different training stages.

[0170] Specifically, such as Figure 9 As shown, this unit establishes a multi-dimensional mapping table to achieve intelligent matching between the training phase, data features, and RDMA transfer modes. The system maintains a decision matrix covering all key scenario combinations. For example, model weight access in the forward propagation phase tends to use RDMA Read, while gradient transfer in the backpropagation phase tends to use RDMAWrite. This matrix is ​​not statically fixed but evolves through continuous learning and optimization: the system records the actual performance of each mapping combination and tries alternatives through A / B testing, automatically updating the matrix content. This adaptive mapping mechanism makes transfer decisions more accurate and can automatically adjust with training progress and hardware environment changes, adapting to the special needs of different model structures and training phases.

[0171] (3) Data size hierarchical decision unit, used to automatically select the most suitable RDMA operation based on the data block size.

[0172] Specifically, such as Figure 10 As shown, this unit implements refined transmission mode selection based on the analysis of the impact of data transmission size on RDMA operation performance. The system divides data blocks into three levels: small data blocks (<4KB), medium data blocks (4KB-1MB), and large data blocks (>1MB), and these thresholds are dynamically adjusted according to network hardware characteristics. For small data blocks, the system prioritizes RDMA Send / Recv or Atomic operations, taking advantage of their low startup overhead and latency; for medium data blocks, the system tends to use RDMA Read to balance latency and throughput; for large data blocks, the system prioritizes RDMA Write to take advantage of its high throughput. The system also implements data aggregation and fragmentation strategies, intelligently aggregating small data blocks and automatically fragmenting and transmitting very large data blocks in parallel. This hierarchical decision-making technology enables the system to select the optimal operation mode for data transmission of different sizes, significantly improving overall transmission efficiency.

[0173] (4) Batch processing sensing and scheduling unit, used to dynamically adjust RDMA transmission parameters and operation mode according to batch size.

[0174] like Figure 11As shown, this unit dynamically optimizes RDMA transmission parameters and modes by monitoring changes in batch size during deep learning training. The system acquires the current batch size information, assesses memory pressure, and then dynamically adjusts transmission parameters: when the batch size increases, the RDMA queue depth, buffer size, and number of parallel transmissions are increased accordingly; when the batch size decreases, these parameters are reduced accordingly to avoid resource waste. The system selects the optimal operation mode based on the batch size: large batch training tends to use RDMA Write for batch transmission; small batch training tends to use low-latency RDMA Read or Send / Recv. The system also implements a computation-communication pipeline strategy, processing batch data in chunks and prefetching the next chunk while computing the current chunk. This batch-aware transmission scheduling effectively adapts to the needs of different training strategies, improving system resource utilization and training throughput.

[0175] (5) Priority-aware mapping unit, used to allocate high-priority RDMA queues for data transmission on the critical path.

[0176] Specifically, such as Figure 12 As shown, this unit implements differentiated priority processing for data transmission tasks by analyzing the training computation graph. The system uses a critical path analysis algorithm to identify critical computation paths in the training process and calculates the "criticality" metric for each data transmission task. Based on the criticality, the system divides transmission tasks into multiple priority levels and assigns them to corresponding priority RDMA queues. High-priority queues are configured in low-latency mode and employ preemptive scheduling; low-priority queues are configured in high-throughput mode and employ batch processing strategies. The system also configures corresponding QoS parameters for different priority queues, including bandwidth allocation ratios, latency guarantees, and queue depth. This priority mapping mechanism ensures that critical data can be transmitted in a timely manner, while non-critical data does not block network resources, significantly reducing computational waiting time during training and improving overall training efficiency and hardware utilization.

[0177] 3. A deep learning algorithm-aware dynamic data fragmentation optimization module, used to optimize the data fragmentation strategy for RDMA transmission based on the characteristics of deep learning algorithms.

[0178] Furthermore, the deep learning algorithm-aware dynamic data sharding optimization module includes:

[0179] (1) Gradient sparsity-aware slicing unit, used to dynamically adjust the slicing granularity according to the sparsity of the gradient.

[0180] Specifically, such as Figure 13As shown, this unit implements an adaptive data sharding and encoding strategy based on the sparsity characteristics of gradients during deep learning training. The system calculates the sparsity (proportion of zero elements) of the gradient tensor and analyzes the distribution pattern, then dynamically adjusts the sparsity classification threshold according to the training stage and network layer type. The system classifies gradients into three categories: high-sparse gradients (>90% zero) typically appear during sparse regularization training or later stages of training; medium-sparse gradients (30%-90% zero) are common during regular training; and low-sparse gradients (<30% zero) mostly appear in the early stages of training. For high-sparse gradients, the system uses sparse encoding and fine-grained sharding, transmitting only non-zero elements and indices; for medium-sparse gradients, it uses hybrid encoding and adaptive sharding; and for low-sparse gradients, it uses dense encoding and coarse-grained sharding. This sparsity-aware sharding technique is particularly effective when training large sparse models, reducing communication by 50-80%.

[0181] (2) Hierarchical perception fragmentation strategy unit, used to adopt differentiated fragmentation strategies for different network layers.

[0182] Specifically, such as Figure 14 As shown, this unit optimizes data sharding by analyzing the structure and computational characteristics of different neural network layer types. The system identifies different types of neural network layers (such as convolutional layers, fully connected layers, and attention layers), analyzes their data access patterns and computational characteristics, and generates the most suitable sharding scheme. For convolutional layers, a "channel-first" sharding strategy is adopted, grouping convolutional kernel parameters of the same channel together, which is beneficial for cache utilization. For fully connected layers, a "row-first" sharding strategy is adopted, allowing each computation node to independently compute a portion of the output neurons. For attention layers, a "block-first" sharding strategy is adopted to optimize the locality of attention computation. The system also selects the most suitable RDMA transfer operation based on the layer type: convolutional layer parameters are typically transferred using RDMA Read in conjunction with caching; fully connected layers tend to use RDMA Write for whole-line transmission. This layer-aware sharding tightly matches data sharding with computational patterns, significantly improving computation-communication efficiency.

[0183] (3) Communication-computation ratio optimized sharding unit, used for adaptive sharding based on a balance between computation density and communication overhead.

[0184] Specifically, such as Figure 15As shown, this unit implements an adaptive data sharding strategy based on the theory of computation-communication balance. The system estimates the computational density (FLOPS / Byte) of operators and the communication overhead of different sharding strategies, and then calculates the "communication-computation ratio" (CCR) to measure the communication intensity of the task. The system classifies tasks into three categories: computationally intensive (CCR < 0.1), such as matrix multiplication; communication-computation balanced (0.1 ≤ CCR ≤ 10), such as medium-scale optimization; and communication-intensive (CCR > 10), such as mini-batch synchronization. For computationally intensive tasks, coarse-grained sharding is used to reduce communication overhead; for communication-intensive tasks, fine-grained sharding is used to increase parallelism; and for balanced tasks, an optimal balance point is found between the two. The system also dynamically adjusts the CCR threshold according to the hardware environment. This sharding optimization based on the communication-computation ratio improves resource utilization efficiency, enabling the system to automatically adapt to training tasks of different sizes and hardware environment characteristics.

[0185] (4) Topology-aware fragmented routing unit, used to make data fragmentation and routing decisions considering the network topology.

[0186] Specifically, such as Figure 16 As shown, this unit achieves coordinated optimization of data sharding and network routing by analyzing the cluster network topology. The system discovers the cluster's physical network topology using network probing tools and constructs a weighted graph model, where nodes represent computing devices, edges represent network connections, and weights represent performance metrics. Based on the topology graph, the system applies path planning algorithms to calculate the optimal communication path between nodes, considering factors such as bandwidth, latency, and congestion status. The system allocates data shards to the most suitable transmission paths, matching the data distribution with the network topology. For example, in a fat-tree network, the system tries to allocate communication-intensive shards to nodes under the same switch; in a ring topology, the system optimizes the global data flow. The system also supports dynamic topology adaptation: when network changes are detected, sharding and routing strategies are automatically adjusted. This topology-aware sharding significantly reduces network congestion and communication latency, and is particularly effective in large-scale distributed training.

[0187] (5) Mixed precision training sharding unit, used to design special sharding strategies for data with different precision.

[0188] Specifically, such as Figure 17As shown, this unit implements precision-aware data sharding and transmission optimization for mixed-precision training. The system identifies the precision type of the data, including FP32 (single precision), FP16 (half precision), BF16 (Brain Floating-Point), and INT8 (quantized integer). For different precisions, the system implements customized sharding strategies: FP16 data uses 2-byte aligned sharding to optimize memory access and transmission unit utilization; FP32 data uses 4-byte aligned standard sharding; and INT8 data uses 1-byte aligned compact sharding. The system selects appropriate transmission optimizations for different precisions: FP16 data prioritizes compressed transmission to further reduce communication volume; FP32 data uses standard transmission to ensure precision; and INT8 data prioritizes batch transmission to maximize throughput. The system also selects RDMA operation mode based on numerical stability requirements and supports dynamic precision adjustment. This mixed-precision-aware sharding technology adapts to the mixed-precision computing needs of modern deep learning, improving communication efficiency while ensuring training accuracy.

[0189] 4. Tensor property adaptive transmission module, used to adaptively select the optimal transmission method based on the characteristics of tensors.

[0190] Furthermore, the tensor property adaptive transfer module includes:

[0191] (1) Tensor compression sensing transmission unit, used to design a dedicated transmission optimization protocol for compression gradient.

[0192] Specifically, such as Figure 18 As shown, this unit designs dedicated transmission protocols for different types of tensor compression methods to maximize compression efficiency. The system first identifies the compression types used by deep learning frameworks, including quantization compression (such as INT8 / FP16 quantization), sparse compression (such as Top-K sparsity), and hybrid compression methods. The system estimates the compression ratio and data characteristics, selecting the optimal transmission strategy for each compression type: for quantization compression, a bit-aligned transmission strategy is adopted, optimizing memory layout and RDMA packet structure; for sparse compression, a dedicated sparse coding transmission format is used, transmitting only non-zero elements and their indices; for hybrid compression, layered coding transmission is used, employing different coding methods for different compression parts. The system also implements compression-aware RDMA operation selection: for high-compression-ratio data, RDMA Write batch transmission is prioritized; for specially formatted compressed data, Send / Recv may be selected for on-the-fly decompression at the receiving end. This compression-aware transmission technique significantly reduces communication overhead, making it particularly suitable for training large models in bandwidth-constrained environments.

[0193] (2) Large tensor layered transmission unit, used to realize multi-level decomposition and priority transmission of large tensors.

[0194] Specifically, such as Figure 19As shown, this unit addresses the efficiency and prioritization issues of large tensor transmission, making it particularly suitable for scenarios such as large language models. The system first performs multi-level decomposition of the large tensor, employing methods such as block partitioning, dimensionality splitting, or semantic grouping. Then, the system assesses the importance of each part, considering factors such as computational dependency, access frequency, and numerical characteristics. Based on this importance assessment, the system assigns transmission priorities to each part of the tensor, ensuring that critical parts are transmitted first. Finally, the system implements hierarchical transmission scheduling, with high-priority parts transmitted via a fast channel and low-priority parts transmitted asynchronously via a background channel. This mechanism supports progressive loading, allowing the receiving end to begin computation immediately after receiving some key data, without waiting for the complete tensor transmission. The system also implements access prediction, enabling the pre-transmission of tensor parts that are about to be accessed, further reducing computational waiting time and significantly improving the efficiency of training large models.

[0195] (3) Tensor semantic understanding unit, used to identify tensor types with different semantics such as weights, gradients, and activation values.

[0196] Specifically, such as Figure 20 As shown, this unit achieves precise matching of transmission modes by deeply understanding the semantic roles of tensors. The system intercepts deep learning framework APIs, extracting semantic information from variable naming, creation context, and usage patterns to identify different types of tensors: weight tensors (model parameters), gradient tensors (parameter update directions), and activation value tensors (intermediate computation results), etc. The system selects the most suitable transmission mode based on the semantic type: for weight tensors, a persistent transmission mode is used, supporting caching and version control, suitable for scenarios with repeated access; for gradient tensors, an aggregate transmission mode is used, supporting gradient accumulation and atomic operations, suitable for distributed optimization; for activation value tensors, an instant transmission mode is used, optimizing latency and timeliness, suitable for scenarios with strong computational dependencies. The system also implements a semantically aware memory management strategy, allocating appropriate buffers and lifecycle management methods for different types of tensors. This semantic understanding technology makes transmission decisions more in line with the inherent needs of deep learning algorithms, significantly improving communication efficiency and resource utilization.

[0197] (4) Tensor heat prediction unit, used to predict hot spot tensor data and arrange transmission in advance.

[0198] Specifically, such as Figure 21As shown, this unit achieves intelligent prefetching and cache optimization by predicting data access patterns. The system statistically analyzes the access frequency, temporal distribution, and access context of tensors to construct a popularity prediction model, which can employ statistical models or machine learning methods (such as Markov models or LSTM networks). Based on historical access patterns, the system predicts high-probability accessed hot data in the future and generates a prefetching strategy. For tensors predicted to be hot, the system prefetches them into the local cache via RDMA Read to reduce access latency; for cold data, the system may choose to delay loading or reduce cache priority. The system also implements a context-aware prefetching mechanism, adjusting the prediction model based on the currently executed operation sequence to improve prediction accuracy. This popularity prediction technique significantly reduces data waiting time, especially in training scenarios with regular access patterns (such as convolutional layer computation in CNNs, multi-head attention mechanisms in Transformers, etc.), enabling advance data preparation, avoiding idle waiting of computing units, and improving hardware utilization.

[0199] (5) Operator computation mode sensing unit, used to optimize the transmission mode according to the operator type.

[0200] Specifically, such as Figure 22 As shown, this unit implements customized optimization of the transmission strategy based on the computational characteristics of deep learning operators. The system identifies different types of computational operators, such as matrix multiplication, convolution, and reduction, through framework API interception or computation graph analysis. The system analyzes the data access patterns, computational parallelism, and memory locality characteristics of each operator, designing dedicated transmission optimization strategies for them: for matrix multiplication, block transmission optimization is adopted, dividing the matrix into small blocks suitable for caching to achieve pipelined processing of computation and transmission; for convolution, channel parallel optimization is adopted, prioritizing the transmission of data from different input / output channels to match the parallel computation mode of convolution; for reduction, tree-structured transmission optimization is adopted, designing a minimum communication tree structure to reduce the number of communication calls and latency. The system also implements operator fusion awareness, identifying combinations of continuously executed operators and designing end-to-end optimized transmission strategies for them to reduce intermediate data transmission. This operator-aware transmission optimization ensures a high degree of matching between communication and computation modes, significantly improving the overall efficiency of distributed training.

[0201] 5. Real-time transmission mode switching module, used to achieve smooth real-time switching of RDMA transmission mode to adapt to the dynamically changing training environment.

[0202] Furthermore, the real-time transmission mode switching module includes:

[0203] (1) Zero interruption mode switching unit, used to smoothly switch transmission modes without interrupting the data stream.

[0204] Specifically, such as Figure 23As shown, this unit enables a smooth switching of RDMA transmission modes without interrupting the data stream, ensuring the continuity of the training process. The system first saves the current transmission state, including ongoing transmission tasks, queue status, and context information. Then, the system pre-establishes the RDMA connection and resources for the new mode, including queue pairs, memory registration, and completion channels, ensuring the new connection is fully ready before switching begins. Subsequently, the system implements a gradual traffic migration: new transmission requests are directed to the new connection while ensuring the completion of transmissions on the old connection, achieving a smooth transition. Finally, after confirming that all data has been safely transmitted and the new connection is functioning correctly, the system releases the resources of the old connection. The entire process is transparent to upper-layer applications and will not interrupt the training process or cause data loss. The system also implements a rollback mechanism, which can safely restore the system to its original state if an anomaly is detected during the switching process, ensuring system stability and reliability.

[0205] (2) Mode switching trigger condition evaluation unit, used to precisely define the set of conditions when it is necessary to switch transmission modes.

[0206] Specifically, such as Figure 24 As shown, this unit defines a complete mode switching trigger condition evaluation mechanism to avoid unnecessary frequent switching. The system continuously monitors various performance indicators, including latency, throughput, resource utilization, and error rate, capturing trends in transmission performance. Based on preset thresholds or adaptively learned dynamic thresholds, the system determines whether the current mode is still the optimal choice, such as when latency exceeds the threshold or throughput is lower than expected. The system also checks the stability of the network environment to ensure that switching decisions are based on continuous performance trends rather than short-term fluctuations. After comprehensive evaluation, the system generates a switching decision, including whether to switch, the timing of the switch, and the target mode. The system introduces a switching hysteresis mechanism (e.g., triggering a switch only after multiple consecutive threshold exceedances) and a cooldown period mechanism (preventing new switches from being triggered for a certain period after a previous switch) to avoid oscillations caused by short-term fluctuations. This precise trigger condition evaluation ensures that the system only performs mode switching when truly needed, achieving a balance between performance optimization and system stability.

[0207] Example 3

[0208] This invention also provides an intelligent selection device for RDMA transmission mode for distributed AI training, such as... Figure 25 As shown, the device 10 includes:

[0209] The multi-dimensional data feature acquisition module is used to acquire multi-dimensional data features during the deep learning training process. The multi-dimensional data features include tensor structure features, data access pattern features, data timeliness features, data dependency relationship features, and data change rate features.

[0210] The training phase identification module is used to identify the current training phase, which includes the forward propagation phase, the backward propagation phase, and the parameter update phase.

[0211] An adaptive decision matrix module is used to select the optimal RDMA transmission mode that matches the current training scenario based on the multi-dimensional data features and the training phase. The optimal RDMA transmission mode includes one or more combinations of RDMA Write, RDMA Read, Send / Receive, and Atomic operations.

[0212] The dynamic transmission mode switching module is used to switch the RDMA transmission mode in real time according to the dynamic changes in the network environment. It adopts a gradual traffic migration strategy to ensure that the training data stream is not interrupted during the switching process, and optimizes the subsequent mode selection decision through a historical performance learning mechanism.

[0213] Example 4

[0214] To implement the methods of the above embodiments, the present invention also provides a computer device, which includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, so as to implement the various steps of the methods described above.

[0215] Example 5

[0216] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0217] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0218] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0219] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for intelligent selection of RDMA transmission mode for distributed AI training, characterized in that, Includes the following steps: S1, Obtain multi-dimensional data features during deep learning training. These multi-dimensional data features include tensor structure features, data access pattern features, data timeliness features, data dependency features, and data change rate features. The timeliness feature is obtained by analyzing the time window of data usage during the training phase, while the change rate feature is obtained by maintaining the version identifier and content summary of the tensor and calculating the update frequency and change magnitude per unit time. S2, identify the current training phase, which includes a forward propagation phase, a backward propagation phase, and a parameter update phase; S3, based on the multi-dimensional data features and the training phase, select the optimal RDMA transmission mode that matches the current training scenario through an adaptive decision matrix. The optimal RDMA transmission mode includes one or more combinations of RDMA Write, RDMA Read, Send / Receive, and Atomic operations. S4. The RDMA transmission mode is switched in real time according to the dynamic changes in the network environment. A gradual traffic migration strategy is adopted to ensure that the training data stream is not interrupted during the switching process, and the subsequent mode selection decision is optimized through a historical performance learning mechanism.

2. The method according to claim 1, characterized in that, The acquisition of multi-dimensional data features during deep learning training also includes: S11 captures tensor creation and operation events through API interceptors, parses the tensor's dimension, sparsity, and memory continuity features, and constructs a feature vector containing the tensor's dimension, sparsity, and memory continuity features. S12 employs a lightweight access tracking mechanism to record the access sequence of tensor elements and identify sequential access, random access, and aggregate access patterns.

3. The method according to claim 1, characterized in that, The identification of the current training phase also includes: S21, by intercepting the forward, backward, and optimizer.step API calls of the deep learning framework, and combining memory access patterns and computational resource usage characteristics, distinguishes between read-intensive operations in the forward propagation phase, computation-intensive operations in the backward propagation phase, and write-intensive operations in the parameter update phase. S22, in which RDMA Read is preferred in the forward propagation phase, a hybrid mode is used in the backward propagation phase, and RDMA Write or Atomic operation is preferred in the parameter update phase.

4. The method according to claim 1, characterized in that, The step of selecting the optimal RDMA transmission mode matching the current training scenario through an adaptive decision matrix based on the multi-dimensional data features and the training phase further includes: S31, maintain a dynamically updated mode-stage mapping matrix, and divide the transmission task into small data blocks, medium data blocks and large data blocks according to the data block size. Small data blocks correspond to RDMA Send / Receive operations, medium data blocks correspond to RDMARead operations, and large data blocks correspond to RDMA Write operations. S32 quantifies the communication density of the task based on the communication-computation ratio (CCR) and adjusts the data fragmentation granularity based on the CCR value. When CCR < 0.1, coarse-grained fragmentation is used to reduce communication overhead, and when CCR > 10, fine-grained fragmentation is used to improve parallelism.

5. The method according to claim 1, characterized in that, The method of dynamically switching RDMA transmission mode according to changes in the network environment also includes: S41, save the current transmission state and pre-establish RDMA connection resources for the new mode, and guide new transmission requests to the new connection through a gradual traffic migration strategy, while ensuring that the old connection tasks are completed; S42 introduces a switching hysteresis mechanism and a cooldown period mechanism, which triggers mode switching only when the performance indicators exceed the threshold multiple times in a row, and sets a cooldown period after switching to avoid frequent oscillations.

6. The method according to claim 1, characterized in that, Also includes: S5 constructs a weighted graph model based on the network topology, where nodes represent computing devices, edges represent network connections, and weights represent bandwidth, latency, and congestion status. Based on the topology graph, it calculates the optimal communication path and allocates data fragments, so that communication-intensive fragments are preferentially transmitted through high-bandwidth, low-latency paths.

7. A smart selection device for RDMA transmission mode for distributed AI training, characterized in that, include: The multi-dimensional data feature acquisition module is used to acquire multi-dimensional data features during the deep learning training process. The multi-dimensional data features include tensor structure features, data access pattern features, data timeliness features, data dependency features, and data change rate features. The timeliness feature is obtained by analyzing the time window of data usage during the training phase, while the change rate feature is obtained by maintaining the version identifier and content summary of the tensor and calculating the update frequency and change magnitude per unit time. The training phase identification module is used to identify the current training phase, which includes the forward propagation phase, the backward propagation phase, and the parameter update phase. An adaptive decision matrix module is used to select the optimal RDMA transmission mode that matches the current training scenario based on the multi-dimensional data features and the training phase. The optimal RDMA transmission mode includes one or more combinations of RDMAWrite, RDMA Read, Send / Receive, and Atomic operations. The dynamic transmission mode switching module is used to switch the RDMA transmission mode in real time according to the dynamic changes in the network environment. It adopts a gradual traffic migration strategy to ensure that the training data stream is not interrupted during the switching process, and optimizes the subsequent mode selection decision through a historical performance learning mechanism.

8. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the RDMA transfer mode intelligent selection method for distributed AI training as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements an intelligent selection method for RDMA transfer mode for distributed AI training as described in any one of claims 1-6.

Images