Distributed server node collaborative interconnection system and device for AI model training

Abstract

The invention discloses a distributed server node collaborative interconnection system for AI model training. The distributed server node collaborative interconnection system comprises a node layer, a collaborative control layer, a resource scheduling layer, a data transmission layer and a fault-tolerant guarantee layer. The node layer comprises calculation, storage and communication nodes and adapts to heterogeneous hardware requirements; the cooperative control layer guarantees the gradient consistency of nodes through a distributed controller and a dynamic synchronization mechanism; the resource scheduling layer adopts a load prediction algorithm based on an exponential smoothing method to dynamically allocate resources; the data transmission layer optimizes transmission through adaptive fragmentation and differential compression; the fault-tolerant guarantee layer detects faults in real time and resumes migration tasks in a breakpoint mode. The system solves the problems of scheduling bottleneck, transmission delay and poor fault tolerance in the prior art, improves the cooperation efficiency, the resource utilization rate and the reliability, and is suitable for large-scale AI model training.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of AI model training, in particular to a distributed server node cooperative interconnection system and device for AI model training. BACKGROUND

[0002] At present, with the rapid development of artificial intelligence technology, the training of large-scale AI models puts forward very high requirements on computing resources, data transmission efficiency and system stability. As the core infrastructure supporting large-scale AI training, the performance of the distributed server node cooperative interconnection system directly determines the efficiency, accuracy and cost of model training. However, the existing distributed server node cooperative interconnection technology still has many technical problems to be solved:

[0003] Firstly, the traditional system mostly adopts a centralized cooperative controller, and the task allocation and synchronization control of all nodes depend on a single controller. When the number of nodes exceeds the scale of hundreds, the controller is prone to request accumulation and response delay, which seriously affects the training efficiency.

[0004] Secondly, the existing system mostly adopts a static resource allocation strategy, that is, fixed resources are allocated according to the initial demand of the training task, and the resource configuration cannot be adjusted in real time according to the node load. When part of the nodes are overloaded due to sudden increase of data volume, it is easy to cause task lag. Part of the nodes are in a low load state for a long time, causing resource waste. AI model training involves transmission of massive training data and model parameters. The existing system mostly uses the general TCP / IP protocol for data transmission, which has the problems of multiple data copying times of the operating system kernel and high transmission delay.

[0005] In addition, the existing system mostly uses a simple heartbeat detection method to identify node faults, and the fault detection delay often exceeds 10s. When the fault is recovered, all unfinished tasks need to be executed again, resulting in long training task interruption time and high data loss risk.

[0006] Finally, large-scale AI training needs to ensure the consistency of gradient data and model parameters of each computing node. The existing system mostly adopts a fixed period synchronization strategy. When the load difference between nodes is large, it is easy for some nodes to wait for synchronization of other nodes after completing gradient calculation, which not only wastes resources but also may cause gradient deviation due to synchronization delay, affecting the model training accuracy.

[0007] In summary, the existing distributed server node cooperative interconnection system has significant defects in cooperation efficiency, resource scheduling, data transmission, fault tolerance and synchronization accuracy, and cannot meet the needs of large-scale and high-complexity AI model training. Therefore, a distributed server node cooperative interconnection system with efficient cooperation, dynamic scheduling, optimized transmission and reliable fault tolerance is urgently needed. SUMMARY

[0008] In view of the deficiencies of the prior art, the application provides a distributed server node cooperative interconnection system and device for AI model training, which solves the problems in the background art.

[0009] To achieve the above object, the application is implemented by the following technical scheme: a distributed server node cooperative interconnection system and device for AI model training, comprising a node layer, a cooperative control layer, a resource scheduling layer, a data transmission layer and a fault tolerance guarantee layer, each layer communicates through a standardized interface, realizes efficient cooperation and interconnection of distributed server nodes, and supports stable execution of AI model training tasks;

[0010] The node layer comprises at least one computing node, at least one storage node, at least one communication node and a node management unit; the computing node is configured to perform gradient calculation and parameter update operations of AI model training, is configured with a GPU / TPU acceleration module, supports FP16 / FP32 precision calculation, communicates with the cooperative control layer through an RPC protocol, uploads computing progress and gradient data in real time, and receives parameter update instructions; the storage node is configured to store training data, model parameters and task logs, adopts a distributed file system to build a storage cluster, supports data redundancy backup, establishes a high-speed data transmission channel with the computing node through an NVMe over Fabrics protocol, and reduces data reading delay; the communication node is configured to realize data forwarding and routing between nodes, is configured with a 25G / 100G Ethernet interface, reduces the number of data copying times by bypassing the operating system kernel, and controls the inter-node communication delay to the microsecond level; the node management unit is configured to collect real-time state information of each node, including CPU utilization, GPU memory occupancy, storage capacity usage, network bandwidth occupancy and hardware temperature, the collection frequency is 1 time / s, the state information is uploaded to the resource scheduling layer through an SNMP protocol, and the resource scheduling layer's resource adjustment instructions are received, and the hardware operating parameters of the node are dynamically configured;

[0011] The collaborative control layer includes a distributed collaborative controller cluster, a task synchronization unit, and a task allocation unit. The distributed collaborative controller cluster consists of at least three controller nodes and uses the Raft consensus algorithm to achieve data synchronization and master-slave node election among controllers. When the master controller fails, the slave controller can switch over within 1 second to avoid interruption of the collaborative control function. The task synchronization unit adopts a parameter server architecture to build a synchronization link. The computing nodes upload gradient data to the parameter server, which updates the model parameters through a gradient aggregation algorithm and then distributes the updated parameters to each computing node. The synchronization cycle is dynamically adjusted according to the amount of training batch data. When the amount of batch data exceeds the threshold set by the system adaptively, gradient data consistency is ensured. The task allocation unit receives AI training task requests, parses the computational and data requirements of the task, decomposes the task into several sub-tasks, records the dependencies between sub-tasks through a task dependency graph, and distributes subsequent sub-tasks only after the previous sub-task is completed. At the same time, according to the resource allocation results of the resource scheduling layer, the sub-tasks are allocated to the computing node with the highest matching degree. The matching degree is calculated as the overlap ratio between the resources required by the sub-task and the remaining resources of the node. The higher the overlap ratio, the higher the matching degree.

[0012] The resource scheduling layer includes a load acquisition unit, a load prediction unit, and a resource allocation unit; the load acquisition unit receives real-time status information uploaded by the node management unit, using 3D modeling. Outliers are removed based on the criteria, the preprocessed data is standardized, and stored in the scheduling database. The load forecasting unit uses a load forecasting algorithm based on exponential smoothing to construct a load forecasting model, as shown in the following formula:

[0013]

[0014] in for The node's predicted load value at any given time; This is a smoothing coefficient, with a value ranging from 0.3 to 0.7. for The actual load value of the node at any given time; The total number of nodes participating in load balancing; For the first The load weight of each node is set according to the node's computing power; for Time of the first The overall resource utilization rate of each node is the weighted average of CPU, GPU, storage, and network resources. The load prediction unit performs a prediction every 5 minutes and sends the results to the resource allocation unit.

[0015] The resource allocation unit adopts a greedy algorithm, preferentially allocates resources to nodes with a load rate < 70%, and if there are multiple candidate nodes, selects the node with the highest matching degree to the sub-task demand; meanwhile, the task priority is considered, and high-priority tasks are preferentially allocated in resource competition, and if the resources are insufficient, the non-core resource release of low-priority tasks is triggered, the task progress is saved before release, and the execution is restored after the high-priority task is completed;

[0016] The data transmission layer includes a data fragmentation unit, a data compression unit and a transmission control unit; the data fragmentation unit determines the fragmentation size by using an adaptive fragmentation algorithm according to the communication node MTU value and the data size, and the formula is as follows:

[0017]

[0018] Among them, is the data fragmentation size, is the fragmentation coefficient, the data is numbered when fragmented, and stored in the fragmentation buffer area, is the maximum transmission unit value of the communication node;

[0019] The data compression unit selects an algorithm according to the data type:

[0020] The image data adopts JPEG2000, the text data adopts LZ77, and the model parameters adopt quantization compression, and the compressed record type and parameters are recorded for decompression; the transmission control unit establishes a direct transmission channel by using the RDMA protocol, monitors the bandwidth and delay in real time, switches the path when the delay > 50ms, and avoids data overflow through flow control;

[0021] The fault tolerance guarantee layer includes a fault detection unit, a task migration unit and a data recovery unit; the fault detection unit sends a heartbeat request every 2 seconds, retries if the node does not respond within 1s, and determines a fault if it does not respond continuously for 3 times or the MD5 check of the state information is incorrect; at the same time, the consistency of the stored data is checked every 10 minutes, and the comparison is made with the redundant backup check code; the task migration unit queries the sub-task progress of the fault node, selects the node with the lowest load as the target, migrates the task by using incremental transmission, updates the task ownership after migration, and continues execution from the breakpoint on the target node; the data recovery unit recovers damaged data from the redundant backup, and synchronizes to the related node after recovery if the backup is damaged (by interpolation method);

[0022] The system further includes an initialization unit, which configures the parameters of each module when starting, detects the connectivity of the nodes, constructs the undirected graph topology, checks the initialization result, and alarms the administrator if abnormal.

[0023] Optionally, the load prediction unit of the resource scheduling layer further includes a load correction module, and the correction formula is as follows:

[0024]

[0025] wherein, is the corrected predicted load value of the node at the moment, is the uncorrected predicted load value of the node at the moment, is the correction coefficient, the value range is 0.1-0.3; is the prediction error at the moment, the calculation formula is as follows:

[0026]

[0027] wherein is the predicted load value of the node at the moment, is the actual load value of the node at the moment, the corrected comparison and , update the historical error database, used to adjust and ; the resource allocation unit also contains a resource conflict resolution module, when multiple high-priority tasks compete for the same node resource, the resource demand density of the task is calculated, and the task with high density is preferentially allocated, so as to reduce the resource occupation time length.

[0028] Optionally, the data compression unit of the data transmission layer also contains a compression effect evaluation module, the compression rate and time consumption are calculated, when the compression rate is lower than the threshold value or the time consumption is more than the preset value, the algorithm is automatically adjusted: the image is switched to WebP, the text is increased by LZ77 window, and the parameter is increased by quantization bit number; the transmission control unit also contains a multi-path transmission module, when the packet loss rate is >5%, 2-3 redundant paths are enabled, the flow is distributed through the flow distribution algorithm, and the transmission stability is improved.

[0029] Optionally, the task migration unit of the fault tolerance guarantee layer also contains a migration priority sorting module, which is sorted according to the remaining time, data volume and dependency level of the subtask; the checkpoint mechanism is used during migration, the subtask with an execution period <10 min stores a checkpoint every 1 min, and the subtask with an execution period >30 min stores a checkpoint every 5 min, and the transmission is continued from the nearest checkpoint when the transmission is interrupted; the fault tolerance guarantee layer also contains a fault warning module, which analyzes the historical fault data of the node, generates a warning and migrates a low-priority subtask when the CPU temperature >85℃ and the GPU memory usage rate >95% for 10 min.

[0030] Optionally, the task synchronization unit of the collaborative control layer also contains a gradient consistency verification module, which calculates the gradient variance of the convolutional neural network model >0.05, the recurrent neural network When the synchronization delay of a node is greater than 200 ms, the task synchronization unit temporarily reduces the task allocation amount of the node to reduce the synchronization waiting time.

[0031] Optionally, the node management unit of the node layer further comprises a heterogeneous node adaptation module for adapting x86 / ARM / RISC-V architecture computing nodes and SSD / HDD / memory-level nodes through a unified driving interface: ARM nodes optimize GPU drivers, and memory-level nodes enable direct reading mode.

[0032] Optionally, the data transmission layer further comprises a data encryption module, which adopts an AES-256 symmetric encryption algorithm, the key is dynamically distributed by a key management center, encryption is performed before compression after fragmentation, and the ciphertext contains an encryption identifier.

[0033] Optionally, the system further comprises a performance optimization module, including an offline optimization unit and an online optimization unit; the performance optimization module generates a report regularly to show trends, effects and improvement suggestions.

[0034] The device of the distributed server node cooperative interconnection system for AI model training comprises a box body, a panel cover is installed at the front end of the box body, a plurality of partition plates are horizontally inserted into the inside of the box body, server nodes for cooperative interconnection are installed above the partition plates.

[0035] The box body is welded by a plurality of rectangular panels, a plurality of sliding grooves matching the edges of the partition plates are formed on the inner side surfaces of the two side panels, and the outer side of the box body is also provided with heat dissipation openings.

[0036] The present application provides a distributed server node cooperative interconnection system and device for AI model training, which has the following beneficial effects:

[0037] The distributed server node cooperative interconnection system and device for AI model training comprehensively solve the core pain points of existing distributed systems in the AI training scene through hierarchical architecture design and multi-dimensional technical innovation, have significant technical advantages and practical value, and have the following beneficial effects:

[0038] The system realizes efficient cooperation of large-scale nodes through a distributed controller cluster and a dynamic synchronization mechanism. The distributed controller uses the Raft algorithm to ensure high availability, and the master-slave switching delay is controlled within 1s, avoiding the interruption of cooperation caused by single node failure. The dynamic synchronization period is flexibly adjusted according to the training data volume, reducing unnecessary synchronization waiting time and ensuring the consistency of gradients among computing nodes, especially in the training of trillion-parameter models, significantly shortening the overall training period. At the same time, the task allocation unit allocates sub-tasks based on the task dependency graph and resource matching degree, avoiding the efficiency loss caused by chaotic sub-task execution order or resource mismatch, further improving the orderliness and efficiency of node cooperation.

[0039] The system significantly improves the resource utilization of distributed servers through dynamic load prediction and refined resource scheduling. The load prediction unit uses an improved exponential smoothing algorithm combined with a load correction mechanism to provide accurate basis for resource allocation. The resource allocation unit combines task priority and node load status to realize dynamic adjustment and on-demand allocation of resources, avoiding the resource overload or idling problem caused by traditional static scheduling.

[0040] The system significantly improves data transmission efficiency and reduces latency through multi-dimensional optimization strategies for the massive data transmission requirements of AI training. The adaptive fragmentation algorithm dynamically adjusts the fragmentation size according to the network state, avoiding the transmission failure caused by excessively large single fragmentation or the high transmission overhead caused by excessively small single fragmentation. The differentiated compression algorithm selects the optimal compression scheme for different types of data, reducing network bandwidth occupancy while ensuring data integrity. The application of RDMA technology realizes direct data transmission between nodes, reducing communication latency from milliseconds to microseconds compared to traditional TCP / IP protocols. Combined with multi-path transmission and flow control mechanisms, stable transmission performance can be maintained even under high network load.

[0041] In terms of system fault tolerance and reliability guarantee, the system constructs a multi-level and full-process fault tolerance mechanism, greatly improving the fault tolerance ability and data security of the AI training task; the fault detection unit adopts the combination of heartbeat detection and state verification, the fault recognition delay is shortened to within 5 seconds, which is much lower than the traditional system of more than 10 seconds, effectively avoiding fault diffusion; the task migration unit shortens the migration time to within 30 seconds through priority sorting, breakpoint resume and checkpoint mechanism, and avoids a large waste of computing resources; the data recovery unit ensures the integrity of training data and model parameters through redundant backup and historical version recovery, even if data damage occurs, it can be recovered within min, avoiding training task failure due to data loss; at the same time, the fault warning module can identify potential fault risks in advance, reduce the impact of faults by pre-migrating low-priority tasks, and further improve the stability of the system;

[0042] In terms of system scalability and security, the system adopts a hierarchical architecture and standardized interface design, and each functional module can be independently expanded, so that new nodes or functions can be added without the need for large-scale modification of the existing system, and can easily adapt to the expansion of the scale from hundreds of nodes to thousands of nodes, meeting the needs of AI training tasks of different scales; the data encryption module and the RBAC access control module build a perfect security protection system to ensure the security of training data and model parameters during transmission and storage, prevent data leakage or unauthorized access, and meet the data security compliance requirements in the AI training scenario.

[0043] In summary, the present application improves the performance, efficiency and reliability of the distributed server node cooperative interconnection system through technical innovation in key links such as cooperative control, resource scheduling, data transmission and fault tolerance guarantee, effectively supports the training needs of large-scale and high-complexity AI models, has significant advantages in reducing training costs, shortening training cycles and ensuring training safety, and has wide application prospects and promotional value. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the system startup and initialization process schematic diagram in the application;

[0045] Figure 2 is the state collection and upload process schematic diagram of the node layer in the application;

[0046] Figure 3 is the load processing and resource allocation process schematic diagram of the resource scheduling layer in the application;

[0047] Figure 4 is the task management and synchronization process schematic diagram of the cooperative control layer in the application;

[0048] Figure 5 Optimized data transmission process schematic diagram of the data transmission layer in the application;

[0049] Figure 6 Full name fault protection process schematic diagram of the fault tolerance guarantee layer in the application;

[0050] Figure 7 Main view schematic diagram of the application.

[0051] In the figure: 1, box; 2, panel cover; 3, partition. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments of the application.

[0053] In the description of the application, unless otherwise specified, the meaning of "a plurality of" is two or more than two; the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "inner", "outer", "front end", "rear end", "head", "tail" and the like are based on the orientations or positional relationships shown in the drawings, and are only for the purpose of facilitating the description of the application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application. In addition, the terms "first", "second", "third" and the like are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0054] In the description of the application, it should be noted that, unless otherwise specified and limited, the terms "connected", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0055] Please refer to Figures 1 to 6 The application provides a technical solution: a distributed server node cooperative interconnection system and device for AI model training, which comprises a node layer, a cooperative control layer, a resource scheduling layer, a data transmission layer and a fault tolerance guarantee layer, each layer communicates through a standardized interface, realizes efficient cooperation and interconnection of distributed server nodes, and supports stable execution of AI model training tasks;

[0056] The node layer includes at least one computing node, at least one storage node, at least one communication node and a node management unit; the computing node is configured to perform gradient calculation and parameter update operations of AI model training, configure a GPU / TPU acceleration module, support FP16 / FP32 precision calculation, communicate with the collaborative control layer through an RPC protocol, upload calculation progress and gradient data in real time, and receive parameter update instructions; the storage node is configured to store training data, model parameters and task logs, build a storage cluster using a distributed file system, support data redundancy backup (the number of backup copies is set according to the importance of data, and training data is backed up in three copies and model parameters are backed up in five copies), establish a high-speed data transmission channel with the computing node through an NVMe over Fabrics protocol, and reduce data reading delay; the communication node is configured to realize data forwarding and routing between nodes, configure a 25G / 100G Ethernet interface, support RDMA technology, reduce the number of data copying times by bypassing the operating system kernel (from traditional 4 times of copying to 1 time of copying), and control the communication delay between nodes to be in the order of microseconds; and the node management unit is configured to collect real-time state information of each node, including CPU utilization, GPU memory occupancy, storage capacity usage, network bandwidth occupancy and hardware temperature, the collection frequency is 1 time / s, the state information is uploaded to the resource scheduling layer through an SNMP protocol, and the resource scheduling layer's resource adjustment instructions are received, and the hardware operating parameters of the node are dynamically configured;

[0057] The node management unit also includes a heterogeneous node adaptation module, which is configured to adapt x86 / ARM / RISC-V architecture computing nodes and SSD / HDD / storage level memory nodes through a unified driver interface: the ARM node optimizes the GPU driver, and the storage level memory enables direct reading mode;

[0058] The collaborative control layer includes a distributed collaborative controller cluster, a task synchronization unit, and a task allocation unit. The distributed collaborative controller cluster consists of at least three controller nodes, employing the Raft consensus algorithm to achieve data synchronization and master-slave node election among controllers. When the master controller fails, the slave controller can switch over within 1 second, preventing interruption of the collaborative control function. The task synchronization unit adopts a parameter server architecture, constructing a synchronization link of "computation node-parameter server-computation node." Computation nodes upload gradient data to the parameter server, which updates the model parameters using a gradient aggregation algorithm and then distributes the updated parameters to each computation node. The synchronization cycle is dynamically adjusted based on the amount of training batch data. When the batch data size is greater than a certain threshold, the synchronization period is adjusted accordingly. When the system adaptively sets a threshold (threshold = total number of nodes × maximum processing capacity per node / 2), it ensures the consistency of gradient data. The task allocation unit receives AI training task requests, analyzes the task's computational requirements (required computing power, memory capacity, training cycle) and data requirements (data volume, data type, access frequency), decomposes the task into several subtasks, records the dependencies between subtasks through a task dependency graph (stored using an adjacency list structure), and only issues subsequent subtasks after the previous subtask is completed. At the same time, according to the resource allocation results of the resource scheduling layer, the subtasks are allocated to the computing node with the highest matching degree. The matching degree is calculated as the overlap ratio between the resources required by the subtask and the remaining resources of the node. The higher the overlap ratio, the higher the matching degree.

[0059] The task synchronization unit also includes a gradient consistency verification module, which uses the variance test method to calculate the gradient variance. Convolutional Neural Network Model >0.05, Recurrent Neural Network When the value is greater than 0.1, the node is notified to retransmit the gradient; if it fails three times consecutively, the node parameters are reset to the latest values ​​from the parameter server. The task synchronization unit also includes a synchronization delay compensation module. When the synchronization delay of a node is greater than 200ms, its task allocation is temporarily reduced to decrease the synchronization waiting time.

[0060] The resource scheduling layer includes a load acquisition unit, a load prediction unit, and a resource allocation unit. The load acquisition unit receives real-time status information uploaded by the node management unit, uses the 3σ criterion to remove outliers (removing data that deviates from the mean by more than three standard deviations), standardizes the preprocessed data (mapping the data to the 0-1 range), and stores it in the scheduling database. The load prediction unit uses a load prediction algorithm based on exponential smoothing to construct a load prediction model, as shown in the following formula:

[0061]

[0062] in for The node's predicted load value at any given time; This is a smoothing coefficient, ranging from 0.3 to 0.7. The system dynamically adjusts it based on historical prediction errors (when the error is > 5%). When the increase is 0.1 and the error is <2%, (Decrease by 0.1) for The actual load value of the node at any given time; The total number of nodes participating in load balancing; For the first The load weight of each node is set according to the node's computing power; for Time of the first The overall resource utilization rate of each node is the weighted average of CPU, GPU, storage, and network resources (weights of 0.3, 0.4, 0.2, and 0.1, respectively). The load prediction unit performs a prediction every 5 minutes and sends the results to the resource allocation unit. The resource allocation unit uses a greedy algorithm to prioritize allocating resources to nodes with a load rate of <70%. If multiple candidate nodes exist, the node with the highest matching degree with the subtask requirements is selected. At the same time, task priorities (high, medium, and low) are considered. High-priority tasks are allocated first when there is resource contention. If resources are insufficient, non-core resources of low-priority tasks can be released. Before release, the task progress is saved, and execution resumes after the high-priority task is completed.

[0063] The load prediction unit also includes a load correction module, with the correction formula as follows:

[0064]

[0065] in, For the revised version Predicted load values ​​at specific time points; Before the correction Predicted load values ​​at specific time points This is a correction factor, with a value ranging from 0.1 to 0.3 (prediction error). hour , hour ); for The time prediction error is calculated using the following formula:

[0066]

[0067] for Predict load values ​​in real time. for Real-time load values, corrected and compared. and Update the historical error database for adjustment. and ); the resource allocation unit also contains a resource conflict resolution module, when multiple high-priority tasks compete for the same node resource, the resource demand density (demand / execution time) of the task is calculated, and the task with high density is preferentially allocated to reduce the resource occupation time length;

[0068] The data transmission layer includes a data slicing unit, a data compression unit, and a transmission control unit; the data slicing unit determines the slicing size according to the communication node MTU value and the data size using an adaptive slicing algorithm, and the formula is as follows:

[0069]

[0070] wherein, is the data slicing size, is the slicing coefficient (1-3, bandwidth occupancy rate <40% when =3, >80% when k=1), the data is numbered (including the sequence number, the total number of slices, and the CRC32 check code) during slicing, and is stored in the slicing buffer area, is the maximum transmission unit value of the communication node;

[0071] The data compression unit selects an algorithm according to the data type: JPEG2000 is used for image data (compression level 5 when the resolution is >10241024, and level 3 when <512512), LZ77 is used for text data (sliding window 4096 bytes), and quantization compression is used for model parameters (16-bit quantization when the dynamic range is >1000, and 8-bit quantization when <100). The compressed record type and parameters are recorded for decompression; the transmission control unit uses the RDMA protocol to establish a direct transmission channel, and monitors the bandwidth and delay in real time. When the delay is >50ms (20ms for real-time tasks and 100ms for offline tasks), the path is switched, and data overflow is avoided through flow control (adjusting the rate according to the receiving end buffer);

[0072] The data compression unit also contains a compression effect evaluation module, which calculates the compression rate (after compression / before compression) and the time consumption. When the compression rate is lower than the threshold value (0.3 for images, 0.2 for texts, and 0.4 for parameters) or the time consumption exceeds the preset value (100ms for images, 50ms for texts, and 30ms for parameters), the algorithm is automatically adjusted: WebP is switched for images, the LZ77 window is increased for texts, and the quantization bit number is increased for parameters; the transmission control unit also contains a multi-path transmission module, which enables 2-3 redundant paths when the packet loss rate is >5%, disperses the flow through a flow distribution algorithm (allocates data volume according to path bandwidth), and improves the transmission stability;

[0073] The data transmission layer also contains a data encryption module, which uses the AES-256 symmetric encryption algorithm, and the key is dynamically distributed by the key management center (SM4 algorithm protects the master key). Encryption is performed after slicing and before compression, and the ciphertext contains an encryption identifier;

[0074] The fault-tolerant guarantee layer includes a fault detection unit, a task migration unit and a data recovery unit; the fault detection unit sends a heartbeat request every 2 seconds, retries if the node does not respond within 1 second, and determines a fault if it does not respond for 3 consecutive times or the state information MD5 check is incorrect; at the same time, the consistency of stored data is checked every 10 minutes, and the comparison is made with the redundant backup check code; the task migration unit queries the fault node sub-task progress (from the task log), selects the node with the lowest load as the target, uses incremental transmission to migrate the task, updates the task ownership after migration, and the target node continues from the breakpoint; the data recovery unit recovers damaged data from the redundant backup, recovers from the historical version through interpolation method if the backup is damaged, and synchronizes to the related node after recovery;

[0075] The task migration unit also includes a migration priority sorting module, which sorts according to the remaining time of the sub-task (the shorter the priority is higher), the data volume (the smaller the higher), and the dependency level (the higher the impact is greater, then the higher); checkpoint mechanism is used during migration, checkpoint is stored every 1 minute for sub-tasks with an execution period of less than 10 minutes, and every 5 minutes for sub-tasks with an execution period of more than 30 minutes, and the transmission is continued from the nearest checkpoint when the transmission is interrupted; the fault-tolerant guarantee layer also includes a fault warning module, which analyzes the historical fault data of the node (load, temperature, etc.), generates a warning and migrates low-priority sub-tasks when the CPU temperature is greater than 85℃ and the GPU memory usage is greater than 95% for 10 minutes;

[0076] The system also includes an initialization unit, which configures module parameters (node IP, port, threshold, etc.) when starting, detects node connectivity, constructs an undirected graph topology (nodes are servers, edges are communication links, and weights are bandwidth), checks the initialization result, and alarms the administrator if there is an exception;

[0077] The system also includes a performance optimization module, including an offline optimization unit and an online optimization unit; the performance optimization module generates a report regularly, showing trends, effects and improvement suggestions.

[0078] Please refer to Figure 7 , the device of the distributed server node cooperative interconnection system for AI model training, comprising a box body 1, a panel cover 2 is installed at the front end of the box body 1, a plurality of partition plates 3 are horizontally inserted in the inside of the box body 1, and servers for node cooperative interconnection are correspondingly installed above the partition plates 3.

[0079] The box body 1 is welded by a plurality of rectangular panels, a plurality of sliding grooves matching the edges of the partition plates 3 are formed on the inner surfaces of the side panels, and the outer side of the box body 1 is also provided with a heat dissipation port.

[0080] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A distributed server node collaborative interconnection system for AI model training, characterized in that, It includes a node layer, a collaborative control layer, a resource scheduling layer, a data transmission layer, and a fault tolerance layer. Each layer communicates through standardized interfaces to achieve efficient collaboration and interconnection of distributed server nodes, supporting the stable execution of AI model training tasks. The node layer includes at least one computing node, at least one storage node, at least one communication node, and a node management unit; The computing node is used to perform gradient calculation and parameter update operations for AI model training. It is equipped with a GPU / TPU acceleration module, supports FP16 / FP32 precision calculation, communicates with the collaborative control layer through the RPC protocol, uploads calculation progress and gradient data in real time, and receives parameter update instructions. The storage nodes are used to store training data, model parameters, and task logs. A distributed file system is used to construct the storage cluster, supporting data redundancy backup. A high-speed data transmission channel is established with the computing nodes via the NVMe over Fabrics protocol to reduce data read latency. The communication nodes are used to implement data forwarding and routing between nodes, configured with 25G / 100G Ethernet interfaces. By bypassing the operating system kernel, the number of data copies is reduced, keeping inter-node communication latency to the microsecond level. The node management unit collects real-time status information of each node, including CPU utilization, GPU memory usage, storage capacity utilization, network bandwidth utilization, and hardware temperature, at a frequency of once per second. The status information is uploaded to the resource scheduling layer via the SNMP protocol, and the unit also receives resource adjustment instructions from the resource scheduling layer to dynamically configure the node's hardware operating parameters. The collaborative control layer includes a distributed collaborative controller cluster, a task synchronization unit, and a task allocation unit. The distributed collaborative controller cluster consists of at least three controller nodes and uses the Raft consensus algorithm to achieve data synchronization and master-slave node election among controllers. When the master controller fails, the slave controller can switch over within 1 second to avoid interruption of the collaborative control function. The task synchronization unit adopts a parameter server architecture to build a synchronization link. The computing nodes upload gradient data to the parameter server, which updates the model parameters through a gradient aggregation algorithm and then distributes the updated parameters to each computing node. The synchronization cycle is dynamically adjusted according to the amount of training batch data. When the amount of batch data exceeds the threshold set by the system adaptively, gradient data consistency is ensured. The task allocation unit receives AI training task requests, parses the computational and data requirements of the task, decomposes the task into several sub-tasks, records the dependencies between sub-tasks through a task dependency graph, and distributes subsequent sub-tasks only after the previous sub-task is completed. At the same time, according to the resource allocation results of the resource scheduling layer, the sub-tasks are allocated to the computing node with the highest matching degree. The matching degree is calculated as the overlap ratio between the resources required by the sub-task and the remaining resources of the node. The higher the overlap ratio, the higher the matching degree. The resource scheduling layer includes a load acquisition unit, a load prediction unit, and a resource allocation unit; the load acquisition unit receives real-time status information uploaded by the node management unit, using 3D modeling. Outliers are removed based on the criteria, the preprocessed data is standardized, and stored in the scheduling database. The load forecasting unit uses a load forecasting algorithm based on exponential smoothing to construct a load forecasting model, as shown in the following formula: ； in for The node's predicted load value at any given time; This is a smoothing coefficient, with a value ranging from 0.3 to 0.

7. for The actual load value of the node at any given time; The total number of nodes participating in load balancing; For the first The load weight of each node is set according to the node's computing power; for Time of the first The overall resource utilization rate of each node is the weighted average of CPU, GPU, storage, and network resources. The load prediction unit performs a prediction every 5 minutes and sends the results to the resource allocation unit. The resource allocation unit adopts a greedy algorithm, which prioritizes allocating resources to nodes with a load rate of <70%. If there are multiple candidate nodes, the node with the highest matching degree with the subtask requirements is selected. At the same time, task priority is considered. High-priority tasks are allocated first when there is resource contention. If resources are insufficient, non-core resources of low-priority tasks are released. Before release, the task progress is saved and execution is resumed after the high-priority task is completed. The data transmission layer includes a data fragmentation unit, a data compression unit, and a transmission control unit. The data fragmentation unit determines the fragment size based on the MTU value of the communication node and the data size using an adaptive fragmentation algorithm, as shown in the following formula: ； in, For data partition size, The partitioning coefficient is used to assign a number to the data during partitioning and store it in the partitioning cache. This is the maximum transmission unit value of the communication node; The data compression unit selects an algorithm based on the data type: Image data uses JPEG2000, text data uses LZ77, and model parameters are quantized and compressed. The compressed data type and parameters are recorded for decompression. The transmission control unit uses the RDMA protocol to establish a direct transmission channel, monitors bandwidth and latency in real time, switches paths when latency is >50ms, and avoids data overflow through flow control. The fault tolerance layer includes a fault detection unit, a task migration unit, and a data recovery unit. The fault detection unit sends a heartbeat request every 2 seconds. If the node does not respond within 1 second, it will retry. If it fails to respond for 3 consecutive times or the MD5 checksum of the status information is incorrect, a fault is determined. At the same time, the consistency of the stored data is checked every 10 minutes by comparing the redundant backup checksum. The task migration unit queries the progress of subtasks of the faulty node, selects the node with the lowest load as the target, migrates the task using incremental transmission, updates the task ownership after migration, and resumes execution from the breakpoint at the target node; the data recovery unit restores damaged data from redundant backups, and if the backup is damaged, it restores it from historical versions (using interpolation to restore, and then synchronizes it to the relevant nodes). The system also includes an initialization unit, which configures the parameters of each module, detects node connectivity, constructs an undirected graph topology, verifies the initialization results, and alerts the administrator if any abnormalities are found.

2. The system as described in claim 1, characterized in that, The load prediction unit of the resource scheduling layer also includes a load correction module, and the correction formula is as follows: ； in, For the revised version Predicted load values ​​at specific time points; Before the correction Predicted load values ​​at specific time points This is a correction factor, with a value ranging from 0.1 to 0.3; for The time prediction error is calculated using the following formula: ； in for Predict load values ​​in real time. for Real-time load values, corrected and compared. and Update the historical error database for adjustment. and The resource allocation unit also includes a resource conflict resolution module. When multiple high-priority tasks compete for the same node resource, the module calculates the resource demand density of the tasks and prioritizes allocating resources to tasks with higher density, thereby reducing the duration of resource occupation.

3. The system as described in claim 1, characterized in that, The data compression unit of the data transmission layer also includes a compression effect evaluation module, which calculates the compression rate and time consumption. When the compression rate is lower than the threshold or the time consumption exceeds the preset value, the algorithm is automatically adjusted: the image is switched to WebP, the text window is enlarged to LZ77, and the quantization bit depth is increased. The transmission control unit also includes a multi-path transmission module. When the packet loss rate is greater than 5%, 2-3 redundant paths are enabled, and the traffic is distributed through a traffic allocation algorithm to improve transmission stability.

4. The system as described in claim 1, characterized in that, The task migration unit of the fault tolerance layer also includes a migration priority sorting module, which sorts subtasks according to their remaining time, data volume, and dependency level. During migration, a checkpoint mechanism is used, storing checkpoints every 1 minute for subtasks with an execution cycle of <10 minutes and every 5 minutes for subtasks with an execution cycle of >30 minutes. If the transmission is interrupted, it resumes from the most recent checkpoint. The fault tolerance layer also includes a fault early warning module, which analyzes historical fault data of nodes. When the CPU temperature is >85℃ and the GPU memory usage is >95% for 10 minutes, an early warning is generated and low-priority subtasks are migrated.

5. The system as described in claim 1, characterized in that, The task synchronization unit of the collaborative control layer also includes a gradient consistency verification module, which calculates the gradient variance using the variance test method. Convolutional Neural Network Model >0.05, Recurrent Neural Network When the value is greater than 0.1, the node is notified to retransmit the gradient; if the value fails three times in a row, the node parameters are reset to the latest value from the parameter server; the task synchronization unit also includes a synchronization delay compensation module. When the synchronization delay of a node is greater than 200ms, its task allocation is temporarily reduced to reduce the synchronization waiting time.

6. The system as described in claim 1, characterized in that, The node management unit of the node layer also includes a heterogeneous node adaptation module, which adapts x86 / ARM / RISC-V architecture computing nodes and SSD / HDD / storage-class memory nodes through a unified driver interface: ARM nodes optimize GPU drivers, and storage-class memory enables direct read mode.

7. The system as described in claim 1, characterized in that, The data transmission layer also includes a data encryption module, which uses the AES-256 symmetric encryption algorithm. The key is dynamically distributed by the key management center, and encryption is performed after fragmentation and before compression. The ciphertext contains an encryption identifier.

8. The system as described in claim 1, characterized in that, The system also includes a performance optimization module, comprising an offline optimization unit and an online optimization unit; the performance optimization module generates reports periodically, showing trends, effects, and improvement suggestions.

9. An apparatus for a distributed server node collaborative interconnection system for AI model training as described in any one of claims 1 to 8, comprising a housing (1), characterized in that, The front end of the box (1) is equipped with a panel cover (2), and multiple partitions (3) are horizontally inserted inside the box (1). A server for node collaborative interconnection is installed above the partitions (3). The box (1) is made of multiple rectangular panels welded together. Multiple grooves that match the edges of the partition (3) are opened on the inner surface of the two side panels. The outer side of the box (1) is also provided with heat dissipation vents.

Images