Cluster communication method and system based on artificial intelligence

Abstract

The invention provides a cluster communication method and system based on artificial intelligence. The cluster communication method and system are applied to AI model training or cluster range communication in high-performance computing. The method comprises the steps that when data communication is carried out between working nodes, if the data size exceeds a first threshold value of dynamic adjustment, data is segmented into a plurality of slices, the number of slices, received by a first working node, of a second working node is inversely proportional to a first distance, and the number of slices sent to a third working node is inversely proportional to a second distance. The AI model collects cluster state information in real time, dynamically selects a distance-slice number mapping function from a function type set according to a network congestion condition, and outputs an optimal parameter; meanwhile, according to different link qualities and data priorities, differentiated slice granularities are selected in a self-adaptive mode. According to the invention, by introducing a dynamic mapping function selection mechanism and a differentiated slicing strategy, finer bandwidth utilization and resource allocation are realized, and the cluster communication efficiency and adaptability are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and particularly relates to a cluster communication method and system based on artificial intelligence. BACKGROUND

[0002] In the prior art, an AI model outputs a general "distance-slice quantity mapping function", the mapping function is fixed in form (such as exponential decay), and the AI only adjusts the parameters (such as the basic slice quantity and the decay rate). However, the real network situation is more complex, and different mapping functions should be applied to highly congested networks and idle networks to more finely utilize the bandwidth.

[0003] In addition, the size of all data slices in the prior art is uniform. However, in real application scenarios, different slice granularities should be applied to different link qualities and data priorities.

[0004] The fast learning ability of the AI model also needs to be continuously improved.

[0005] Therefore, there is an urgent need for a targeted cluster communication method and system based on artificial intelligence. SUMMARY

[0006] The purpose of the present application is to provide a cluster communication method and system based on artificial intelligence, which realizes more fine intelligent slicing strategies through dynamic mapping function types and differentiated slice granularities, and builds a meta-learning framework to enable the AI model to learn more quickly.

[0007] In a first aspect, the present application provides a cluster communication method based on artificial intelligence, which comprises:

[0008] In the process of performing a cluster-wide communication operation by a plurality of worker nodes, a first worker node receives first data from a second worker node and / or sends second data to a third worker node;

[0009] When the data in each worker node of the plurality of worker nodes is greater than a first threshold, the data in each worker node is divided into a plurality of slice data, the number of slice data in the second worker node included in the first data is inversely proportional to a first distance, and the number of slice data in the first worker node included in the second data is inversely proportional to a second distance;

[0010] The AI model collects real-time cluster state information, including the real-time bandwidth and delay of all links, the port buffer usage of the switch, and the current data size of the communication;

[0011] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered.

[0012] Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority;

[0013] The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

[0014] Secondly, this application provides a cluster communication system based on artificial intelligence, the system comprising: a first working node, a second working node and / or a third working node, and an AI model;

[0015] During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node;

[0016] When the data in each of the plurality of working nodes is greater than the first threshold, the data in each working node is divided into several slices of data. The number of slices of data in the second working node included in the first data is inversely proportional to the first distance, and the number of slices of data in the first working node included in the second data is inversely proportional to the second distance.

[0017] The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated;

[0018] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered.

[0019] Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority;

[0020] The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

[0021] Thirdly, this application provides a cluster communication system based on artificial intelligence, the system including a processor and a memory:

[0022] The memory is used to store program code and transmit the program code to the processor;

[0023] The processor is configured to execute any one of the methods described in the first aspect according to the instructions in the program code.

[0024] Fourthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a computing device, perform any of the methods described in the first aspect.

[0025] Fifthly, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in the first aspect.

[0026] Beneficial effects

[0027] This invention provides a cluster communication method and system based on artificial intelligence, applicable to cluster-wide communication in AI model training or high-performance computing. The method includes: during data communication between worker nodes, if the data volume exceeds a dynamically adjusted first threshold, the data is divided into several slices. The number of slices received by a first worker node from a second worker node is inversely proportional to a first distance, and the number of slices sent to a third worker node is inversely proportional to a second distance. The AI ​​model collects cluster status information in real time, dynamically selects a distance-slice number mapping function from a set of function types based on network congestion, and outputs the optimal parameters. Simultaneously, it adaptively selects differentiated slice granularity based on different link qualities and data priorities. This invention, by introducing a dynamic mapping function selection mechanism and a differentiated slice strategy, achieves more refined bandwidth utilization and resource allocation, significantly improving the efficiency and adaptability of cluster communication.

[0028] The method and system of the present invention have the following advantages and effects:

[0029] 1. Achieved dynamic optimization and scenario adaptation of mapping functions: By introducing a dynamic mapping function selection mechanism, the AI ​​model can select the optimal distance-slice number mapping relationship from various function types (such as exponential decay, linear decrease, step function, etc.) according to the real-time network status (such as congestion or idleness). This overcomes the limitation of traditional fixed function forms being unable to adapt to complex network environments, enabling communication strategies to accurately match the current network conditions and achieve more refined bandwidth utilization.

[0030] 2. Improved data transmission reliability and efficiency: Through a differentiated slicing granularity strategy, the AI ​​model can adaptively adjust the slice size based on link quality and data priority. Finer-grained slices are used for high-priority data or low-quality links to enhance transmission reliability; larger slices are used for ordinary data or high-quality links to reduce protocol overhead, thereby maximizing network utilization while ensuring data transmission quality.

[0031] 3. Enhanced system intelligence and adaptability: By introducing the meta-learning framework, the AI ​​model has the ability to learn quickly. When facing new cluster environments or workloads, it can quickly adapt to efficient communication strategies with only a small number of samples, greatly shortening the model convergence time and improving the system's adaptability and practicality in dynamic environments.

[0032] 4. Optimized system resource utilization: The dynamically adjusted trigger threshold mechanism ensures that the intelligent slicing algorithm is activated at the most suitable data volume, avoiding unnecessary data segmentation overhead; at the same time, intelligent function selection and data granularity control based on network status achieve the best balance between network resources and computing resources, comprehensively improving the overall system performance. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the present invention;

[0035] Figure 2 This is a system architecture diagram of the present invention. Detailed Implementation

[0036] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0037] The artificial intelligence-based cluster communication method provided in this application includes:

[0038] During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node;

[0039] When the data in each of the plurality of working nodes is greater than the first threshold, the data in each working node is divided into several slices of data. The number of slices of data in the second working node included in the first data is inversely proportional to the first distance, and the number of slices of data in the first working node included in the second data is inversely proportional to the second distance.

[0040] The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated;

[0041] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered.

[0042] This allows AI not only to adjust parameters but also to select the most suitable function from a variety of types. For example, when the network is highly congested, a "ladder" mapping function is used to allocate a fixed number of slices to nodes at different distances to reduce scheduling complexity; when the network is idle, a "linear" mapping function is used to utilize bandwidth more precisely.

[0043] The "function type selection" is treated as a discrete action and, together with the existing parameters, serves as the output action space of the AI ​​model.

[0044] Specific implementation steps may include:

[0045] Step 1: State Awareness and Characterization

[0046] The AI ​​model first obtains raw state information from a cluster monitoring system (such as Prometheus) or a network controller (such as an SDN controller) and processes it into feature vectors that the model can understand.

[0047] State characteristics may include: [data volume, latency of node AB, latency of node AC, ..., available bandwidth of node AB, port queue depth of core switch, ...].

[0048] Step 2: Model Output Action – Parameterizing the “Mapping Function” and “Threshold”

[0049] The core task of the model is to generate "actions." The model outputs a few key parameters that define this function.

[0050] The model's action output can be defined as:

[0051] Action parameter 1 = Base slices: The number of base slices allocated to the nearest node.

[0052] Action parameter 2 = Decay rate: The rate at which the number of slices decreases as the distance increases.

[0053] Generation of mapping functions:

[0054] With these two parameters, the mapping function is determined. For example, the model outputs Base_Slices = 8 and Decay_Rate = 0.5.

[0055] Therefore, for a node with a distance of 1, the number of slices to be allocated is 8*(0.5)^(1-1) = 8.

[0056] For a node with a distance of 2, the number of slices allocated is 8*(0.5)^(2-1) = 4.

[0057] For a node with a distance of 3, the number of slices allocated is 8*(0.5)^(3-1) = 2.

[0058] This approach significantly reduces the action space. Regardless of the number of nodes in the cluster, the model only needs to learn to output two parameters, instead of outputting a value for each node pair, making learning feasible and efficient.

[0059] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, and selects a suitable distance-slice number mapping function from the set of function types to output.

[0060] The first threshold is dynamically adjusted, and the model directly outputs the threshold as an action parameter.

[0061] The model's action output can be defined as:

[0062] Action parameter 3 = first threshold (Threshold)

[0063] For example, based on the current network conditions, the model determines that even small-volume communication can benefit from intelligent scheduling, and it might output Threshold = 4KB. This means that as long as the data volume exceeds 4KB, the intelligent slicing algorithm will be triggered.

[0064] Step 3: Environmental Implementation and Reward Feedback

[0065] Execution action: The scheduler uses the parameters output by the model (defined mapping function and threshold) to execute the current collection communication operation (such as All-Reduce).

[0066] Measurement reward: The system accurately measures and records the total time required to complete this communication.

[0067] Calculate the reward: Reward = -Total Time. The shorter the time, the greater the reward (the smaller the negative value).

[0068] Feedback learning: This "reward" signal is sent back to the AI ​​model (reinforcement learning agent). The model uses this feedback to update its internal parameters (neural network weights), with the goal of adjusting its policy so that it can output action parameters that will yield higher rewards (i.e., shorter communication times) when faced with similar situations in the future.

[0069] Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority;

[0070] The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

[0071] The above process intelligently determines the data volume at which the intelligent slicing algorithm is triggered, rather than using a fixed threshold. This enables fine-grained scheduling and is more adaptable to changing environments than fixed empirical formulas.

[0072] AI does not change the algorithm itself, but rather dynamically selects the optimal parameters for the algorithm. Traditional ring algorithms or HD algorithms are based on fixed rules. The addition of AI aims to dynamically select or generate the optimal communication strategy based on the real-time operating environment (network topology, traffic load, worker node status) and historical data.

[0073] In some preferred embodiments, a meta-learning framework is introduced during the construction of the AI ​​model. Each different cluster configuration is treated as an independent task, a multi-task training environment is built, and a meta-learning training loop is designed based on the MAML (Model-Agnostic Meta-Learning) algorithm. The task-specific inner loop is executed for rapid adaptation. When the system is deployed to a new cluster environment, a small amount of real-time data is collected, and the meta-learning model is quickly fine-tuned. The adapted model can then be used for optimization decisions in the new environment.

[0074] In some preferred embodiments, smaller slices are used for control information or critical data blocks with high reliability requirements to facilitate retransmission and error correction; larger slices are used for large-scale data blocks to reduce protocol overhead.

[0075] The AI ​​model can determine the granularity of slicing based on link quality and data priority. Along with the output mapping function, the AI ​​model also outputs a "slicing granularity strategy" to guide worker nodes in performing non-uniform slicing.

[0076] Figure 2The architecture diagram of the artificial intelligence-based cluster communication system provided in this application includes: a first working node, a second working node and / or a third working node, and an AI model;

[0077] During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node;

[0078] When the data in each of the plurality of working nodes is greater than the first threshold, the data in each working node is divided into several slices of data. The number of slices of data in the second working node included in the first data is inversely proportional to the first distance, and the number of slices of data in the first working node included in the second data is inversely proportional to the second distance.

[0079] The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated;

[0080] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered.

[0081] Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority;

[0082] The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

[0083] This application provides a cluster communication system based on artificial intelligence, the system comprising: a processor and a memory.

[0084] The memory is used to store program code and transmit the program code to the processor;

[0085] The processor is configured to execute the method described in any one of the embodiments of the first aspect according to the instructions in the program code.

[0086] This application provides a computer-readable storage medium for storing program code, which is executed by a processor to implement the method described in any one of the embodiments of the first aspect.

[0087] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and the program, when executed, may include some or all of the steps in the various embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0088] In a specific implementation, the present invention also provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the method described in any one of the embodiments of the first aspect.

[0089] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0090] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0091] The embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention.

Claims

1. A cluster communication method based on artificial intelligence, characterized in that, The method includes: During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node; When the data in each of the plurality of working nodes is greater than the first threshold, the data in each working node is divided into several slices of data. The number of slices of data in the second working node included in the first data is inversely proportional to the first distance, and the number of slices of data in the first working node included in the second data is inversely proportional to the second distance. The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated; Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered. Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority; The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

2. The method according to claim 1, characterized in that: In the process of building AI models, a meta-learning framework is introduced, treating each different cluster configuration as an independent task, constructing a multi-task training environment, and designing a meta-learning training loop based on the MAML (Model-Agnostic Meta-Learning) algorithm, executing task-specific inner loops for rapid adaptation; when the system is deployed to a new cluster environment, a small amount of real-time data is collected, and rapid fine-tuning is performed based on the meta-learning model, and the adapted model can be used for optimization decisions in the new environment.

3. The method according to claim 1, characterized in that: For control information or critical data blocks with high reliability requirements, smaller slices are used to facilitate retransmission and error correction; for large-scale data blocks, larger slices are used to reduce protocol overhead.

4. A cluster communication system based on artificial intelligence, characterized in that, The system includes: a first working node, a second working node and / or a third working node, and an AI model; During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node; When the data in each of the plurality of working nodes is greater than the first threshold, the data in each working node is divided into several slices of data. The number of slices of data in the second working node included in the first data is inversely proportional to the first distance, and the number of slices of data in the first working node included in the second data is inversely proportional to the second distance. The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated; Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered. Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority; The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

5. A cluster communication system based on artificial intelligence, characterized in that, The system includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute instructions in the program code to implement the method of any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-3.

7. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1-3.

Images