Set communication method and apparatus, computer device, readable storage medium and program product

Abstract

The application relates to a collection communication method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: acquiring network state information of each intelligent calculation center in an intelligent calculation network; performing decision processing on the network state information through a pre-trained reinforcement learning model to obtain a collection communication strategy for the intelligent calculation centers; and controlling data interaction between the intelligent calculation centers according to the collection communication strategy; wherein the pre-trained reinforcement learning model is trained by taking network state information as a state space and defining training performance of model training of the intelligent calculation network as a reward function. The method can improve collection communication efficiency, reduce network congestion, and further improve the training rate of model training across calculation centers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a set communication method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the rapid development of information technology, intelligent computing network has become an important development direction in the field of modern computing. Intelligent computing network integrates high-performance computing, big data analysis, artificial intelligence and other advanced technologies to realize rapid processing, intelligent analysis and efficient decision-making of massive data.

[0003] In the process of training a model in a network environment across intelligent computing centers, a large amount of communication is required between GPUs (graphics processing units), such as parameters, gradients, intermediate activation values, etc. These data transmission requirements have high requirements on network bandwidth and delay. When the network bandwidth is insufficient to meet the data transmission requirements, or the network delay is too high, network congestion may occur, thereby affecting the model training rate. SUMMARY

[0004] Therefore, it is necessary to provide a set communication method, device, computer equipment, computer readable storage medium and computer program product to solve the technical problem that network congestion easily occurs in the network environment across intelligent computing centers, affecting the model training rate.

[0005] In a first aspect, the present application provides a set communication method. The method comprises:

[0006] Obtaining network state information of each intelligent computing center in an intelligent computing network;

[0007] Performing decision processing on the network state information by a pre-trained reinforcement learning model to obtain a set communication strategy for the intelligent computing centers;

[0008] Controlling data interaction between the intelligent computing centers according to the set communication strategy;

[0009] The pre-trained reinforcement learning model is trained by defining the network state information as a state space and defining the training performance of the intelligent computing network as a reward function.

[0010] In one embodiment, the training process of the pre-trained reinforcement learning model comprises:

[0011] Defining the state space, action space and reward function of the reinforcement learning model, and initializing the model parameters;

[0012] In the current state, an action corresponding to the action space is selected according to a policy of the reinforcement learning model, and the action is executed to obtain a next state and feedback information of the intelligent computing network;

[0013] According to the feedback information and the reward function, a reward value is calculated;

[0014] According to the reward value, model parameters of the reinforcement learning model are updated until a training end condition is reached, and the pre-trained reinforcement learning model is obtained.

[0015] In one of the embodiments, the feedback information includes a throughput of model training of the intelligent computing network, a data amount affected by network delay, and a data amount affected by resource waste;

[0016] The reward value is calculated according to the feedback information and the reward function, including:

[0017] The total data amount of the data amount affected by network delay and the data amount affected by resource waste is calculated based on the reward function;

[0018] The difference between the throughput and the total data amount is calculated to obtain the reward value.

[0019] In one of the embodiments, each of the intelligent computing centers includes a plurality of computing nodes; and the policy of the reinforcement learning model includes:

[0020] For any two intelligent computing centers, the number difference between the number of computing nodes in the two intelligent computing centers is obtained;

[0021] If the number difference is less than a first threshold, the two intelligent computing centers are controlled to perform internal collective communication respectively, and then perform collective communication between the two intelligent computing centers.

[0022] In one of the embodiments, the policy of the reinforcement learning model further includes:

[0023] If the number difference is greater than a second threshold, the intelligent computing center with less number of computing nodes is controlled to perform internal collective communication, and the execution result is synchronized to the other intelligent computing center;

[0024] The other intelligent computing center is controlled to perform internal collective communication.

[0025] In one of the embodiments, the policy of the reinforcement learning model further includes:

[0026] When the data parallel mode is set for the intelligent computing network, different intelligent computing centers are set to perform processing tasks for different data sets.

[0027] In a second aspect, the present application also provides a collection communication device. The device comprises:

[0028] an information acquisition module configured to acquire network state information of each intelligent center in an intelligent network;

[0029] a policy determination module configured to perform decision processing on the network state information by using a pre-trained reinforcement learning model to obtain a collection communication policy for each intelligent center; wherein the pre-trained reinforcement learning model is trained by taking the network state information as a state space and defining a training performance of the intelligent network during model training as a reward function;

[0030] a collection communication module configured to control data interaction between the intelligent centers according to the collection communication policy.

[0031] In a third aspect, the present application also provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0032] acquiring network state information of each intelligent center in an intelligent network;

[0033] performing decision processing on the network state information by using a pre-trained reinforcement learning model to obtain a collection communication policy for each intelligent center;

[0034] controlling data interaction between the intelligent centers according to the collection communication policy;

[0035] wherein the pre-trained reinforcement learning model is trained by taking the network state information as a state space and defining a training performance of the intelligent network during model training as a reward function.

[0036] In a fourth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0037] acquiring network state information of each intelligent center in an intelligent network;

[0038] performing decision processing on the network state information by using a pre-trained reinforcement learning model to obtain a collection communication policy for each intelligent center;

[0039] controlling data interaction between the intelligent centers according to the collection communication policy;

[0040] The pre-trained reinforcement learning model is trained by taking the network state information as a state space and defining a training performance of the intelligent computing network during model training as a reward function.

[0041] In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the following steps:

[0042] obtaining network state information of each intelligent computing center in the intelligent computing network;

[0043] performing decision processing on the network state information by using a pre-trained reinforcement learning model to obtain a set communication strategy for the intelligent computing centers;

[0044] controlling data interaction between the intelligent computing centers according to the set communication strategy;

[0045] The pre-trained reinforcement learning model is trained by taking the network state information as a state space and defining a training performance of the intelligent computing network during model training as a reward function.

[0046] The set communication method, device, computer device, storage medium and computer program product described above use a reinforcement learning model pre-trained by taking network state information as a state space and defining a training performance of the intelligent computing network during model training as a reward function, and use the reinforcement learning model to generate a set communication strategy for each intelligent computing center according to network state information of each intelligent computing center in the intelligent computing network, so as to control data interaction between the intelligent computing centers. The method applies a reinforcement learning algorithm to determination of the set communication strategy, and generates the set communication strategy based on network state information of each intelligent computing center, so that the set communication strategy can dynamically adapt to network states of the intelligent computing network, thereby ensuring that each intelligent computing center can perform data interaction in an optimal communication manner, improving set communication efficiency, reducing network congestion, and further improving a training rate of model training across computing centers. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 An application environment diagram of the set communication method in one embodiment;

[0048] Figure 2 A flowchart of the set communication method in one embodiment;

[0049] Figure 3 A flowchart of the reinforcement learning model training step in one embodiment;

[0050] Figure 4 A flowchart of the reinforcement learning model training process in another embodiment;

[0051] Figure 5 This is a schematic diagram of the architecture of a reinforcement learning-based optimized ensemble communication algorithm in an intelligent computing network environment, as shown in one embodiment.

[0052] Figure 6 This is a structural block diagram of a collection of communication devices in one embodiment;

[0053] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0056] It is understandable that a large-scale distributed machine learning task involves multiple computing nodes distributed across different computing centers. These nodes need to frequently exchange and synchronize data for model training. However, due to the complexity and dynamism of the network environment, traditional ensemble communication algorithms may not achieve optimal performance under different network conditions, leading to prolonged training time and wasted resources. Since the traffic during model training has certain regularities, ensemble communication algorithms can be optimized based on reinforcement learning. Therefore, this application proposes a method for optimizing ensemble communication strategies based on reinforcement learning.

[0057] The collection communication method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with intelligent computing network 104 via a network, which includes multiple intelligent computing centers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, etc. Terminal 102 acquires network status information of each intelligent computing center in intelligent computing network 104; it performs decision processing on the network status information using a pre-trained reinforcement learning model to obtain a collective communication strategy for each intelligent computing center; and it controls the data interaction between the intelligent computing centers according to the collective communication strategy. This optimizes the collective communication algorithm for each intelligent computing center in the intelligent computing network, reduces the time spent on collective communication, and improves the model training rate across intelligent computing centers.

[0058] In one embodiment, as shown in Figure 2 A set communication method is provided, which is applied to Figure 1 the terminal 102 in the method, including the following steps:

[0059] Step S210, obtaining network state information of each intelligent center in the intelligent network.

[0060] The network state information is used to represent the performance of the network, and can include network bandwidth usage, delay, packet loss rate, GPU communication time, and parameter amount transferred, etc.

[0061] In a specific implementation, the network state information of each intelligent center in the intelligent network can be monitored by a network monitoring tool and sent to the terminal, so that the terminal can obtain the network state information of each intelligent center.

[0062] Step S220, processing the network state information by a pre-trained reinforcement learning model to obtain a set communication strategy for each intelligent center.

[0063] The pre-trained reinforcement learning model is trained by taking the network state information as the state space, and defining the training performance of the intelligent network as the reward function. Reinforcement learning is a machine learning technique that learns the optimal behavior strategy through interaction with the environment.

[0064] It can be understood that the intelligent center undertakes massive data processing and complex computing tasks, and the set communication algorithm plays a key role in it. Therefore, it is necessary to optimize the set communication algorithm because it directly affects the data transmission efficiency and resource utilization efficiency of the intelligent center. Therefore, in order to reduce data transmission delay, improve throughput, and ensure smooth data synchronization and collaborative work between different computing nodes. The present application proposes a method of using reinforcement learning to optimize the set communication algorithm, taking the network state as the state space, taking the data transmission mode (such as data transmission batch size, communication protocol, data compression level, etc.) as the action, taking the training performance of the intelligent network (affected by network performance indicators such as throughput, delay, etc.) as the reward function, training the reinforcement learning model, and determining the set communication strategy for each intelligent center through the pre-trained reinforcement learning model.

[0065] In specific implementation, the network state information of each intelligent computing center in the intelligent computing network can be input into the pre-trained reinforcement learning model. The reinforcement learning model is trained through interaction with the network environment, and thus an optimized set communication strategy for each intelligent computing center can be obtained. The set communication strategy can include communication control parameters such as data transmission sequence and data transmission rate between the intelligent computing centers.

[0066] In step S230, the data interaction between the intelligent computing centers is controlled according to the set communication strategy.

[0067] In specific implementation, after the set communication strategy is generated, the data interaction between the intelligent computing centers can be controlled according to the communication control parameters in the set communication strategy.

[0068] In the method, the monitoring frequency can be set in a time-based manner or by using other methods. When the network state of each intelligent computing center changes, the set communication strategy of each intelligent computing center is adaptively updated to control each intelligent computing center to communicate in accordance with the set communication strategy that meets its network state in a timely manner, thereby reducing the time occupied by set communication and improving the set communication efficiency, and thus improving the training rate of model training across computing centers.

[0069] In the set communication method, the reinforcement learning model is pre-trained with the network state information as the state space, and the training performance of the model training of the intelligent computing network is defined as the reward function. According to the network state information of each intelligent computing center in the intelligent computing network, the reinforcement learning model generates a set communication strategy for each intelligent computing center to control the data interaction between the intelligent computing centers. The method applies the reinforcement learning algorithm to the determination of the set communication strategy, generates the set communication strategy based on the network state information of each intelligent computing center, so that the set communication strategy can dynamically adapt to the network state of the intelligent computing network, thereby ensuring that each intelligent computing center can interact data in the optimal communication manner, improving the set communication efficiency, reducing network congestion, and further improving the training rate of model training across computing centers.

[0070] In an exemplary embodiment, as shown in Figure 3 the training steps of the pre-trained reinforcement learning model include:

[0071] In step S310, the state space, action space and reward function of the reinforcement learning model are defined, and the model parameters are initialized.

[0072] In step S320, an action corresponding to the action space is selected according to the policy of the reinforcement learning model in the current state, and the action is executed to obtain the next state and feedback information of the intelligent computing network.

[0073] In step S330, the reward value is calculated according to the feedback information and the reward function.

[0074] In step S340, the model parameters of the reinforcement learning model are updated according to the reward value until a training end condition is reached, and a pre-trained reinforcement learning model is obtained.

[0075] In a specific implementation, before the reinforcement learning model is trained, the environment of the reinforcement learning model is first defined, including a state space, an action space, a reward function, and a time interval.

[0076] The state space is defined as network state information, which can include the current network bandwidth utilization, the computing load of each node, the data transmission delay, the size of the ongoing collective communication task, and the like. These information can be used as the dimensions of the state space to form a vector representation of the state. For example, the state can be represented by a vector as follows: S = (bandwidth utilization, computing load 1, computing load 2, data transmission delay, task size).

[0077] The action space is defined as a data transmission method, and the action can be understood as an action performed in each state. The action can include adjusting the data transmission batch size, selecting different communication protocols (such as different variants of TCP (Transmission Control Protocol) or UDP (User Datagram Protocol)), adjusting the data compression level, and the like. The action can also be represented as a vector, for example: A = (batch size adjustment value, communication protocol selection, compression level adjustment value).

[0078] The reward function is used to evaluate the effect of each action. The reward function can be designed to be related to the training performance (such as training efficiency, stability, and the like) of the intelligent algorithm network during model training.

[0079] The time interval is used to determine the frequency of state updates. For example, it can be set to update the state once every 5 seconds, and the time interval T = 5s. This parameter needs to be adjusted according to the dynamic change speed of the network environment and the bearing capacity of the computing resources. Too fast can cause waste of computing resources, and too slow cannot capture the changes in the environment in time.

[0080] After the environment of the reinforcement learning model is defined, the model parameters can be initialized, including the weights of the neural network (if the algorithm uses a neural network), the learning rate, the discount factor, and the like. The learning rate can be initially set to a relatively large value, such as 0.001, and then adjusted according to the convergence during the training process. The discount factor is generally between [0.9, 1]. It determines the degree of influence of future rewards on current decisions, and is determined according to the time span of the task and the timeliness of the reward.

[0081] Further, the model training can be performed, and in the current state Next, select an action based on the policy of the reinforcement learning model. Then, perform the action. Observe the feedback from the environment to obtain the next state. Based on the feedback information from the intelligent computing network and the reward function, the reward value is calculated. .

[0082] Will( , , , The data is stored in the experience replay buffer. The size of the experience replay buffer can be set according to the needs of computing resources and the diversity of training data, and is generally set to several thousand to tens of thousands of experience samples. A batch of samples is randomly selected from the experience replay buffer to train the neural network of the reinforcement learning model. The parameters of the neural network are updated by minimizing the loss function, so that the Q-value estimation is more accurate.

[0083] Repeat the above steps, continuously updating the reinforcement learning model's policy, until the training termination condition is met, such as reward convergence or training time reaching its limit. Then, training ends, resulting in a pre-trained reinforcement learning model. The entire iterative training process is as follows: Figure 4 As shown.

[0084] In this embodiment, by defining the precise state and action of the reinforcement learning model, implementing an intelligent reward mechanism, and continuously optimizing the policy, efficient data transmission and resource management can be achieved in complex intelligent computing network environments, greatly improving network performance.

[0085] In one exemplary embodiment, the feedback information includes the throughput of the intelligent computing network for model training, the amount of data affected by network latency, and the amount of data affected by resource waste;

[0086] Step S330 above calculates the reward value based on the feedback information and the reward function, including: calculating the total amount of data affected by network latency and the amount of data affected by resource waste based on the reward function; calculating the difference between the throughput and the total amount of data to obtain the reward value.

[0087] In practice, the reward function can be defined as a function of the model training throughput (the amount of data processed per unit time) and the costs of network latency and resource waste. Specifically, it can be expressed as:

[0088]

[0089] Where R represents the reward value, and α and β are weighting coefficients that can be adjusted according to the actual situation.

[0090] When the reward value is calculated based on the reward function, the data volume affected by the network delay and the data volume affected by the resource waste are added to obtain a total data volume affected by both. Then, the reward value is obtained by subtracting the total data volume from the throughput of the model training.

[0091] In this embodiment, the reward value is calculated by considering the throughput of the model training, the data volume affected by the network delay, and the data volume affected by the resource waste. The reward value obtained not only reflects the basic performance of the network, but also takes into account factors such as delay and resource waste, making the decision strategy of the reinforcement learning model trained based on this more accurate and comprehensive, and improving the effect of the set communication strategy generated based on the reinforcement learning model subsequently.

[0092] It can be understood that long-distance communication should be avoided as much as possible when designing the strategy of the reinforcement learning model because the cross-region interconnection requires a long time. Since the set communication time is related to the number of nodes for communication, the intra-domain delay affected by the number of computing nodes of each intelligent calculation center and the delay of long-distance transmission need to be considered. For ease of illustration, the set communication algorithm rules are described below by taking the set communication of intelligent calculation centers in two different regions during model training as an example.

[0093] In an exemplary embodiment, each intelligent calculation center includes a plurality of computing nodes; and the strategy of the reinforcement learning model includes: for any two intelligent calculation centers, obtaining a quantity difference between the number of computing nodes in the two intelligent calculation centers; and if the quantity difference is less than a first threshold, controlling the two intelligent calculation centers to perform internal set communication respectively and then perform set communication between the two intelligent calculation centers.

[0094] The computing nodes can be GPU nodes.

[0095] In a specific implementation, the number of computing nodes in each intelligent calculation center can be obtained in advance. For any two intelligent calculation centers, if the quantity difference between the number of computing nodes of the two intelligent calculation centers is less than a first threshold, that is, the number of computing nodes of the two intelligent calculation centers is similar, the two intelligent calculation centers are controlled to perform internal set communication respectively and then perform long-distance set communication between the two intelligent calculation centers.

[0096] It can be understood that when the number of computing nodes of the two intelligent calculation centers is similar, the time for the two intelligent calculation centers to perform set communication is less different. Therefore, the two intelligent calculation centers are first controlled to perform internal set communication respectively, so that the data can be merged and integrated, and then long-distance set communication between the two intelligent calculation centers is performed, so that the amount of data that needs to be transmitted over a long distance can be reduced, the burden of cross-center communication can be reduced, the overall delay can be reduced, and the overall communication efficiency can be improved.

[0097] In another case, if the number difference between the number of computing nodes in the two intelligence calculation centers is greater than the second threshold value, the intelligence calculation center with less number of computing nodes is controlled to perform internal collective communication first, and the execution result is synchronized to the other intelligence calculation center; the other intelligence calculation center is controlled to perform internal collective communication.

[0098] In a specific implementation, for any two intelligence calculation centers, if the number difference between the number of computing nodes in the two intelligence calculation centers is greater than the first threshold value, that is, the number difference between the number of computing nodes in the two intelligence calculation centers is large, the intelligence calculation center with less number of computing nodes should perform internal collective communication first, and then synchronize the execution result to the other intelligence calculation center with more number of computing nodes through long-distance collective communication. Then, the intelligence calculation center with more number of computing nodes performs collective communication.

[0099] It can be understood that in the case of unbalanced number of computing nodes in the two intelligence calculation centers, the node with less number of computing nodes can complete the preliminary processing faster. If the two intelligence calculation centers perform collective communication respectively, long-distance collective communication needs to be performed after the intelligence calculation center with more number of computing nodes is processed. Therefore, in the embodiment, when the number of nodes in the two intelligence calculation centers is far apart, the intelligence calculation center with less number of computing nodes is first made to perform internal collective communication processing, and then the execution result is synchronized to the center with more number of nodes in time through long-distance collective communication. In this way, the intelligence calculation center with more number of computing nodes only needs to wait for a short time, so that long-time occupation of network resources can be avoided.

[0100] In the above embodiment, the communication strategy is optimized according to the number difference of computing nodes, which can effectively improve the overall performance and efficiency of the intelligence calculation network. When the number of nodes is similar, internal communication is performed first to reduce delay and improve throughput; when the number difference of nodes is large, the node with less number is processed first, resources and network load are reasonably utilized, and finally efficient long-distance data synchronization and computing task execution are realized. Therefore, the system can not only respond quickly, but also maintain high resource utilization.

[0101] In an example embodiment, the strategy of the reinforcement learning model further includes: when a data parallel mode is set for the intelligence calculation network, different intelligence calculation centers are set to perform processing tasks for different data sets.

[0102] The data parallelism (DP) refers to that each computing node runs the same model code, and the data set is split into multiple parts and allocated to different computing nodes for training.

[0103] It can be understood that in the data parallel mode, a large data set is usually divided into multiple small blocks, and these small blocks are allocated to different computing nodes for processing. Each node performs independent computation, and the final result needs to be aggregated through collective communication. When different computing nodes are physically far apart, long-distance communication will cause high latency and low bandwidth utilization. If the collective communication involves long-distance message passing across data centers or across regions, it can cause performance bottlenecks. Therefore, in order to reduce communication delay and improve efficiency, when designing a data parallel scheme, how to reduce or optimize these long-distance communications should be given priority.

[0104] Specifically, the embodiment adopts a way of matching the data division mode with the physical layout of computing resources, for example, distributing data nodes within each intelligence center instead of across multiple intelligence centers. That is, each computing node in each intelligence center processes the same data set, and the collective communication aggregation is performed within the intelligence center, so as to avoid the computing nodes of different intelligence centers performing the training task of the same data set.

[0105] In the embodiment, when setting the data parallel mode for the intelligence network, setting different intelligence centers to perform processing tasks for different data sets can avoid long-distance message passing across data centers, reduce the delay in data transmission by reducing long-distance communication, so that the computing nodes can complete the task more efficiently, thereby accelerating the overall computing rate.

[0106] In one embodiment, in order to facilitate those skilled in the art to understand the embodiments of the present application, the following will be described in conjunction with specific examples of the accompanying drawings. Referring to Figure 5 , a schematic diagram of an algorithm architecture for optimizing collective communication based on reinforcement learning in an intelligence network environment is shown, which includes a state monitoring module, a data acquisition module, a reinforcement learning module, a policy generation module, a collective communication module and a data transmission module, wherein:

[0107] The state monitoring module is used to monitor the network state of each intelligence center, including the network bandwidth usage, delay, packet loss rate, GPU communication time and parameter amount during model training and other key indicators.

[0108] The data acquisition module is used to obtain network state information from the state monitoring module and deliver it to the reinforcement learning module.

[0109] The reinforcement learning module uses reinforcement learning algorithm to learn and make decisions according to the network state information delivered by the data acquisition module, defines state space, action space and reward function, continuously optimizes the policy through interaction with the network environment, and outputs the decision result.

[0110] Policy generation module: based on the decision result output by the reinforcement learning module, generate the optimization strategy of the set communication.

[0111] Set communication module: execute the set communication strategy generated by the policy generation module, responsible for the transmission and synchronization of data between different intelligence calculation centers. According to the optimization strategy provided by the policy generation module, adjust the communication parameters such as data transmission rate and data transmission order.

[0112] Data transmission module: according to the data transmission rate strategy provided by the policy generation module, intelligently control the sending of data. When the network state changes, this module will adjust the sending behavior according to the new strategy.

[0113] Intelligence calculation center network environment: multiple intelligence calculation centers jointly constitute a multi-intelligence calculation center network environment, execute computing tasks and interact data.

[0114] It can be understood that an efficient set communication algorithm is crucial for large-scale distributed computing, deep learning model training and other tasks, which can accelerate the computing process, improve the overall performance and service quality of the intelligence calculation center, meet the growing computing demand, maintain an advantage in the fierce technology competition, and provide stronger support for the digital transformation and innovation of various industries. Therefore, the application method of the reinforcement learning algorithm in optimizing the set communication algorithm provided by the embodiment can promote the development of intelligence network technology, improve the network performance and stability, and provide more reliable network support for big data processing, intelligent decision-making and other fields.

[0115] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, as described above, at least part of the steps in the flowchart involved in each embodiment can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0116] Based on the same inventive concept, the embodiments of the present application also provide a set communication device for implementing the set communication method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more set communication device embodiments provided below can refer to the limitations of the set communication method described above, which will not be repeated here.

[0117] In one embodiment, as shown in FIG. 1, Figure 6 A collection communication device is provided, comprising:

[0118] An information acquisition module 610 is configured to acquire network state information of each intelligent center in the intelligent network.

[0119] A policy determination module 620 is configured to perform decision processing on the network state information by using a pre-trained reinforcement learning model to obtain a collection communication policy for each intelligent center. The pre-trained reinforcement learning model is trained by taking the network state information as a state space and defining a training performance of the intelligent network as a reward function.

[0120] A collection communication module 630 is configured to control data interaction between the intelligent centers according to the collection communication policy.

[0121] In one embodiment, the device further comprises a model training module configured to define a state space, an action space and a reward function of the reinforcement learning model, and initialize model parameters. In a current state, an action corresponding to the action space is selected according to a policy of the reinforcement learning model, and the action is executed to obtain a next state and feedback information of the intelligent network. A reward value is calculated according to the feedback information and the reward function. The model parameters of the reinforcement learning model are updated according to the reward value until a training end condition is reached, and a pre-trained reinforcement learning model is obtained.

[0122] In one embodiment, the feedback information comprises a throughput of the intelligent network for model training, a data volume affected by network delay and a data volume affected by resource waste. The model training module is further configured to calculate a total data volume of the data volume affected by network delay and the data volume affected by resource waste based on the reward function, and calculate a difference between the throughput and the total data volume to obtain the reward value.

[0123] In one embodiment, each intelligent center comprises a plurality of computing nodes. The policy of the reinforcement learning model comprises: for any two intelligent centers, acquiring a quantity difference between the number of computing nodes in the two intelligent centers; if the quantity difference is less than a first threshold, controlling the two intelligent centers to perform internal collection communication respectively, and then performing collection communication between the two intelligent centers.

[0124] In one embodiment, the policy of the reinforcement learning model further comprises: if the quantity difference is greater than a second threshold, controlling an intelligent center with fewer computing nodes among the two intelligent centers to perform internal collection communication first, and synchronizing the execution result to the other intelligent center; and controlling the other intelligent center to perform internal collection communication.

[0125] In one of the embodiments, the policy of the reinforcement learning model further includes: setting different intelligent calculation centers to perform processing tasks on different data sets when a data parallel mode is set for the intelligent calculation network.

[0126] The modules in the collection communication apparatus can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be invoked and executed by the processor to perform operations corresponding to the modules.

[0127] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 7 The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, mobile cellular network, NFC (near field communication), or other technologies. The computer program is executed by the processor to implement a collection communication method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0128] Those skilled in the art can understand that Figure 7 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In one embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0130] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0131] In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0132] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0133] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0134] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, as long as the combinations of technical features do not have contradictions, they shall be considered within the scope of the present disclosure.

[0135] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present application. It shall be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. An aggregation communication method characterized by, The method comprises: obtaining network state information of each intelligent center in an intelligent network; each of the intelligent centers comprises a plurality of computing nodes; processing the network state information through a pre-trained reinforcement learning model to obtain a set communication strategy for the intelligent centers; controlling data interaction between the intelligent centers according to the set communication strategy; wherein the pre-trained reinforcement learning model is trained by taking network state information as a state space, taking a data transmission mode as an action space, and defining a training performance of the intelligent network as a reward function, and the specific training process comprises: defining a state space, an action space and a reward function of the reinforcement learning model, and initializing model parameters; under a current state, selecting an action corresponding to the action space according to a strategy of the reinforcement learning model, and executing the action to obtain a next state and feedback information of the intelligent network; the feedback information comprises a throughput of the intelligent network for model training, a data volume affected by network delay and a data volume affected by resource waste; the strategy of the reinforcement learning model comprises: for any two intelligent centers, obtaining a quantity difference between the number of computing nodes in the two intelligent centers; if the quantity difference is less than a first threshold, controlling the two intelligent centers to perform internal set communication respectively, and then performing set communication between the two intelligent centers; calculating a total data volume of the data volume affected by network delay and the data volume affected by resource waste in the feedback information according to the reward function, calculating a difference value between the throughput and the total data volume to obtain a reward value; updating the model parameters of the reinforcement learning model according to the reward value until a training end condition is reached to obtain the pre-trained reinforcement learning model.

2. The method of claim 1, wherein, The strategy of the reinforcement learning model further comprises: if the quantity difference is greater than a second threshold, controlling the intelligent center with fewer computing nodes to perform internal set communication first, and synchronizing the execution result to the other intelligent center; controlling the other intelligent center to perform internal set communication.

3. The method according to claim 1 or 2, characterized in that, The strategy of the reinforcement learning model further comprises: when a data parallel mode is set for the intelligent network, each computing node in each intelligent center processes a same data set, and set communication is performed in the intelligent center.

4. The method of claim 1, wherein, The method further comprises: defining a time interval of the reinforcement learning model before training; the time interval is used to determine the frequency of state updating, and the time interval is adjusted according to the dynamic change speed of the network environment and the bearing capacity of the computing resources.

5. The method of claim 1, wherein, The reward function is: R = throughput - (α × data volume affected by network delay + β × data volume affected by resource waste) wherein R represents a reward value, and α and β are weight coefficients.

6. The method of claim 1, wherein, The model parameters comprise weights, learning rates and discount factors of a neural network.

7. An aggregation communication device, comprising: The device comprises: an information acquisition module configured to obtain network state information of each intelligent center in an intelligent network; each of the intelligent centers comprises a plurality of computing nodes; The policy determination module is configured to perform decision processing on the network state information by using a pre-trained reinforcement learning model to obtain a set communication policy for each of the intelligent calculation centers; wherein the pre-trained reinforcement learning model is trained by using the network state information as a state space, using a data transmission mode as an action space, and defining a training performance of the intelligent calculation network as a reward function. The set communication module is configured to control data interaction between the intelligent calculation centers according to the set communication policy. The device further includes a model training module configured to define a state space, an action space, and a reward function of a reinforcement learning model, and to initialize model parameters; in a current state, an action corresponding to the action space is selected according to a policy of the reinforcement learning model, and the action is executed to obtain a next state and feedback information of the intelligent calculation network; the feedback information includes a throughput of model training of the intelligent calculation network, a data amount affected by network delay, and a data amount affected by resource waste; the policy of the reinforcement learning model includes: for any two intelligent calculation centers, obtaining a quantity difference between the number of computing nodes in the two intelligent calculation centers; if the quantity difference is less than a first threshold, controlling the two intelligent calculation centers to perform internal set communication respectively, and then performing set communication between the two intelligent calculation centers; calculating a total data amount of the data amount affected by network delay and the data amount affected by resource waste in the feedback information according to the reward function, calculating a difference value between the throughput and the total data amount to obtain a reward value; updating the model parameters of the reinforcement learning model according to the reward value until a training end condition is reached to obtain the pre-trained reinforcement learning model.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the set communication method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the set communication method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the set communication method of any one of claims 1 to 6. The computer program is executed by the processor to implement the steps of the set communication method of any one of claims 1 to 6.

Images