A cluster communication method and system based on artificial intelligence

Abstract

The application provides a kind of cluster communication method and system based on artificial intelligence, applied to AI model training or cluster range communication in high performance computing.The method comprises: when data communication is carried out between work nodes, if the data volume exceeds the first threshold dynamically adjusted, the data is divided into several slices, wherein the number of slices of the second work node received by the first work node is inversely proportional to the first distance, and the number of slices sent to the third work node is inversely proportional to the second distance.The AI model collects cluster state information in real time, dynamically selects the mapping function of distance-slice number from the function type set according to the network congestion condition, and outputs the optimal parameters; at the same time, according to different link quality and data priority, adaptive selection of differentiated slice granularity is carried out.The application realizes more refined bandwidth utilization and resource allocation by introducing dynamic mapping function selection mechanism and differentiated slice strategy, and significantly improves the efficiency and adaptability of cluster communication.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to a cluster communication method and system based on artificial intelligence. Background Technology

[0002] In existing solutions, the AI ​​model outputs a general "distance-slice number mapping function," with a fixed form (such as exponential decay). The AI ​​only adjusts its parameters (such as the base slice number and decay rate). However, real-world network conditions are more complex, with highly congested and idle networks requiring different mapping functions to utilize bandwidth more precisely.

[0003] Furthermore, in existing technologies, all data slices are of uniform size. However, in real-world applications, different slice granularities should be applied to address varying link qualities and data priorities.

[0004] The rapid learning ability of AI models also needs continuous improvement.

[0005] Therefore, there is an urgent need for a targeted AI-based cluster communication method and system. Summary of the Invention

[0006] The purpose of this invention is to provide a cluster communication method and system based on artificial intelligence, which realizes a more refined intelligent slicing strategy by dynamically mapping function types and differentiating slicing granularity, and builds a meta-learning framework to enable AI models to learn more quickly.

[0007] In a first aspect, this application provides a cluster communication method based on artificial intelligence, the method comprising:

[0008] During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node;

[0009] When the data in each of the plurality of working nodes is greater than the first threshold, the data in each working node is divided into several slices of data. The number of slices of data in the second working node included in the first data is inversely proportional to the first distance, and the number of slices of data in the first working node included in the second data is inversely proportional to the second distance.

[0010] The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated;

[0011] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered.

[0012] Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority;

[0013] The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

[0014] Secondly, this application provides a cluster communication system based on artificial intelligence, the system comprising: a first working node, a second working node and / or a third working node, and an AI model;

[0015] During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node;

[0016] When the data in each of the plurality of working nodes is greater than the first threshold, the data in each working node is divided into several slices of data. The number of slices of data in the second working node included in the first data is inversely proportional to the first distance, and the number of slices of data in the first working node included in the second data is inversely proportional to the second distance.

[0017] The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated;

[0018] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered.

[0019] Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority;

[0020] The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

[0021] Thirdly, this application provides a cluster communication system based on artificial intelligence, the system including a processor and a memory:

[0022] The memory is used to store program code and transmit the program code to the processor;

[0023] The processor is configured to execute any one of the methods described in the first aspect according to the instructions in the program code.

[0024] Fourthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a computing device, perform any of the methods described in the first aspect.

[0025] Fifthly, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in the first aspect.

[0026] Beneficial effects

[0027] This invention provides a cluster communication method and system based on artificial intelligence, applicable to cluster-wide communication in AI model training or high-performance computing. The method includes: during data communication between worker nodes, if the data volume exceeds a dynamically adjusted first threshold, the data is divided into several slices. The number of slices received by a first worker node from a second worker node is inversely proportional to a first distance, and the number of slices sent to a third worker node is inversely proportional to a second distance. The AI ​​model collects cluster status information in real time, dynamically selects a distance-slice number mapping function from a set of function types based on network congestion, and outputs the optimal parameters. Simultaneously, it adaptively selects differentiated slice granularity based on different link qualities and data priorities. This invention, by introducing a dynamic mapping function selection mechanism and a differentiated slice strategy, achieves more refined bandwidth utilization and resource allocation, significantly improving the efficiency and adaptability of cluster communication.

[0028] The method and system of the present invention have the following advantages and effects:

[0029] 1. Achieved dynamic optimization and scenario adaptation of mapping functions: By introducing a dynamic mapping function selection mechanism, the AI ​​model can select the optimal distance-slice number mapping relationship from various function types (such as exponential decay, linear decrease, step function, etc.) according to the real-time network status (such as congestion or idleness). This overcomes the limitation of traditional fixed function forms being unable to adapt to complex network environments, enabling communication strategies to accurately match the current network conditions and achieve more refined bandwidth utilization.

[0030] 2. Improved data transmission reliability and efficiency: Through a differentiated slicing granularity strategy, the AI ​​model can adaptively adjust the slice size based on link quality and data priority. Finer-grained slices are used for high-priority data or low-quality links to enhance transmission reliability; larger slices are used for ordinary data or high-quality links to reduce protocol overhead, thereby maximizing network utilization while ensuring data transmission quality.

[0031] 3. Enhanced system intelligence and adaptability: By introducing the meta-learning framework, the AI ​​model has the ability to learn quickly. When facing new cluster environments or workloads, it can quickly adapt to efficient communication strategies with only a small number of samples, greatly shortening the model convergence time and improving the system's adaptability and practicality in dynamic environments.

[0032] 4. Optimized system resource utilization: The dynamically adjusted trigger threshold mechanism ensures that the intelligent slicing algorithm is activated at the most suitable data volume, avoiding unnecessary data segmentation overhead; at the same time, intelligent function selection and data granularity control based on network status achieve the best balance between network resources and computing resources, comprehensively improving the overall system performance. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the present invention;

[0035] Figure 2 This is a system architecture diagram of the present invention. Detailed Implementation

[0036] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0037] The artificial intelligence-based cluster communication method provided in this application includes:

[0038] During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node;

[0039] When the data in each of the plurality of working nodes is greater than the first threshold, the data in each working node is divided into several slices of data. The number of slices of data in the second working node included in the first data is inversely proportional to the first distance, and the number of slices of data in the first working node included in the second data is inversely proportional to the second distance.

[0040] The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated;

[0041] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered.

[0042] This allows AI not only to adjust parameters but also to select the most suitable function from a variety of types. For example, when the network is highly congested, a "ladder" mapping function is used to allocate a fixed number of slices to nodes at different distances to reduce scheduling complexity; when the network is idle, a "linear" mapping function is used to utilize bandwidth more precisely.

[0043] The "function type selection" is treated as a discrete action and, together with the existing parameters, serves as the output action space of the AI ​​model.

[0044] Specific implementation steps may include:

[0045] Step 1: State Awareness and Characterization

[0046] The AI ​​model first obtains raw state information from a cluster monitoring system (such as Prometheus) or a network controller (such as an SDN controller) and processes it into feature vectors that the model can understand.

[0047] State characteristics may include: [data volume, latency of node AB, latency of node AC, ..., available bandwidth of node AB, port queue depth of core switch, ...].

[0048] Step 2: Model Output Action – Parameterizing the “Mapping Function” and “Threshold”

[0049] The core task of the model is to generate "actions." The model outputs a few key parameters that define this function.

[0050] The model's action output can be defined as:

[0051] Action parameter 1 = Base slices: The number of base slices allocated to the nearest node.

[0052] Action parameter 2 = Decay rate: The rate at which the number of slices decreases as the distance increases.

[0053] Generation of mapping functions:

[0054] With these two parameters, the mapping function is determined. For example, the model outputs Base_Slices = 8 and Decay_Rate = 0.5.

[0055] Therefore, for a node with a distance of 1, the number of slices to be allocated is 8*(0.5)^(1-1) = 8.

[0056] For a node with a distance of 2, the number of slices allocated is 8*(0.5)^(2-1) = 4.

[0057] For a node with a distance of 3, the number of slices allocated is 8*(0.5)^(3-1) = 2.

[0058] This approach significantly reduces the action space. Regardless of the number of nodes in the cluster, the model only needs to learn to output two parameters, instead of outputting a value for each node pair, making learning feasible and efficient.

[0059] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, and selects a suitable distance-slice number mapping function from the set of function types to output.

[0060] The first threshold is dynamically adjusted, and the model directly outputs the threshold as an action parameter.

[0061] The model's action output can be defined as:

[0062] Action parameter 3 = first threshold (Threshold)

[0063] For example, based on the current network conditions, the model determines that even small-volume communication can benefit from intelligent scheduling, and it might output Threshold = 4KB. This means that as long as the data volume exceeds 4KB, the intelligent slicing algorithm will be triggered.

[0064] Step 3: Environmental Implementation and Reward Feedback

[0065] Execution action: The scheduler uses the parameters output by the model (defined mapping function and threshold) to execute the current collection communication operation (such as All-Reduce).

[0066] Measurement reward: The system accurately measures and records the total time required to complete this communication.

[0067] Calculate the reward: Reward = -Total Time. The shorter the time, the greater the reward (the smaller the negative value).

[0068] Feedback learning: This "reward" signal is sent back to the AI ​​model (reinforcement learning agent). The model uses this feedback to update its internal parameters (neural network weights), with the goal of adjusting its policy so that it can output action parameters that will yield higher rewards (i.e., shorter communication times) when faced with similar situations in the future.

[0069] Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority;

[0070] The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

[0071] The above process intelligently determines the data volume at which the intelligent slicing algorithm is triggered, rather than using a fixed threshold. This enables fine-grained scheduling and is more adaptable to changing environments than fixed empirical formulas.

[0072] AI does not change the algorithm itself, but rather dynamically selects the optimal parameters for the algorithm. Traditional ring algorithms or HD algorithms are based on fixed rules. The addition of AI aims to dynamically select or generate the optimal communication strategy based on the real-time operating environment (network topology, traffic load, worker node status) and historical data.

[0073] In some preferred embodiments, a meta-learning framework is introduced during the construction of the AI ​​model. Each different cluster configuration is treated as an independent task, a multi-task training environment is built, and a meta-learning training loop is designed based on the MAML (Model-Agnostic Meta-Learning) algorithm. The task-specific inner loop is executed for rapid adaptation. When the system is deployed to a new cluster environment, a small amount of real-time data is collected, and the meta-learning model is quickly fine-tuned. The adapted model can then be used for optimization decisions in the new environment.

[0074] In some preferred embodiments, smaller slices are used for control information or critical data blocks with high reliability requirements to facilitate retransmission and error correction; larger slices are used for large-scale data blocks to reduce protocol overhead.

[0075] The AI ​​model can determine the granularity of slicing based on link quality and data priority. Along with the output mapping function, the AI ​​model also outputs a "slicing granularity strategy" to guide worker nodes in performing non-uniform slicing.

[0076] Figure 2The architecture diagram of the artificial intelligence-based cluster communication system provided in this application includes: a first working node, a second working node and / or a third working node, and an AI model;

[0077] During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node;

[0078] When the data in each of the plurality of working nodes is greater than the first threshold, the data in each working node is divided into several slices of data. The number of slices of data in the second working node included in the first data is inversely proportional to the first distance, and the number of slices of data in the first working node included in the second data is inversely proportional to the second distance.

[0079] The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated;

[0080] Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects a mapping function from the function type set to output the distance-slice number, and then outputs the optimal parameters, including: determining how many slices to allocate to nearby nodes and how many slices to allocate to distant nodes; dynamically adjusting the first threshold; and intelligently determining under what data volume the intelligent slicing algorithm is triggered.

[0081] Among them, the AI ​​model will also adaptively select differentiated slice granularity based on different link quality and data priority;

[0082] The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

[0083] This application provides a cluster communication system based on artificial intelligence, the system comprising: a processor and a memory.

[0084] The memory is used to store program code and transmit the program code to the processor;

[0085] The processor is configured to execute the method described in any one of the embodiments of the first aspect according to the instructions in the program code.

[0086] This application provides a computer-readable storage medium for storing program code, which is executed by a processor to implement the method described in any one of the embodiments of the first aspect.

[0087] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and the program, when executed, may include some or all of the steps in the various embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0088] In a specific implementation, the present invention also provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the method described in any one of the embodiments of the first aspect.

[0089] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0090] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0091] The embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention.

Claims

1. A cluster communication method based on artificial intelligence, characterized in that, The method includes: During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node; For each of the plurality of working nodes, when the amount of data to be communicated in that working node exceeds a first threshold, the AI ​​model adaptively selects a differentiated slice granularity based on different link qualities and data priorities, and divides the data to be communicated in that working node into several slices according to the selected slice granularity; the first data includes slice data from the second working node, and the number of slices included in the first data is inversely proportional to the first distance; the second data includes slice data sent from the first working node to the third working node, and the number of slices included in the second data is inversely proportional to the second distance; The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated; Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects and outputs a distance-slice number mapping function from the function type set, and then outputs the optimal parameters, which include the base slice number and the decay rate. The base slice number and the decay rate are used to determine the number of slices allocated to nearby nodes and the number of slices allocated to distant nodes through the selected mapping function; and dynamically adjusts the first threshold to determine the data volume under which slicing is triggered. The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

2. The method according to claim 1, characterized in that: In the process of building AI models, a meta-learning framework is introduced, treating each different cluster configuration as an independent task. A multi-task training environment is built, and a meta-learning training loop is designed based on the MAML algorithm. The task-specific inner loop is executed for rapid adaptation. When the system is deployed to a new cluster environment, a small amount of real-time data is collected, and the meta-learning model is quickly fine-tuned. The adapted model can then be used for optimization decisions in the new environment.

3. The method according to claim 1, characterized in that: For control information or critical data blocks with high reliability requirements, smaller slices are used to facilitate retransmission and error correction; for large-scale data blocks, larger slices are used to reduce protocol overhead.

4. A cluster communication system based on artificial intelligence, characterized in that, The system includes: a first working node, a second working node, a third working node, and an AI model; During the process of multiple worker nodes performing cluster-wide communication operations, the first worker node receives first data from the second worker node and / or sends second data to the third worker node; For each of the plurality of working nodes, when the amount of data to be communicated in that working node exceeds a first threshold, the AI ​​model adaptively selects a differentiated slice granularity based on different link qualities and data priorities, and divides the data to be communicated in that working node into several slices according to the selected slice granularity; the first data includes slice data from the second working node, and the number of slices included in the first data is inversely proportional to the first distance; the second data includes slice data sent from the first working node to the third working node, and the number of slices included in the second data is inversely proportional to the second distance; The AI ​​model collects cluster status information in real time, including: real-time bandwidth and latency of all links, port buffer utilization of switches, and the amount of data currently being communicated; Based on the above state information, the AI ​​model determines whether the current network is congested or idle, selects and outputs a distance-slice number mapping function from the function type set, and then outputs the optimal parameters, which include the base slice number and the decay rate. The base slice number and the decay rate are used to determine the number of slices allocated to nearby nodes and the number of slices allocated to distant nodes through the selected mapping function; and dynamically adjusts the first threshold to determine the data volume under which slicing is triggered. The first distance is the distance between the first working node and the second working node, and the second distance is the distance between the first working node and the third working node.

5. A cluster communication system based on artificial intelligence, characterized in that, The system includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute instructions in the program code to implement the method of any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-3.

7. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1-3.

Images