Zero-bandwidth multi-agent training method and system based on multi-GPU distributed architecture

Abstract

The invention belongs to the field of artificial intelligence, and discloses a zero-bandwidth multi-agent training method and system based on a multi-GPU distributed architecture. The zero-bandwidth multi-agent training method comprises the following steps: based on a selected multi-agent coordination task, acquiring environment state information and a certain number of to-be-trained agents, and allocating and deploying the environment state information and the agents to a plurality of GPU devices; based on the environment state information, each GPU device calls a multi-GPU distributed multi-agent network to carry out iterative training on the distributed and deployed agents, and trained agents are obtained; the multi-GPU distributed multi-agent network comprises a visual feature extraction module, a zero-bandwidth communication compression module and an agent strategy network module which are connected in sequence. According to the technical scheme disclosed by the invention, the problems of long training time and performance bottleneck caused by low multi-agent training efficiency, overlarge communication overhead and insufficient utilization of computing resources in the prior art are solved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a zero-bandwidth multi-agent training method and system based on a multi-GPU (Graphics Processing Unit) distributed architecture. Background Technology

[0002] Multi-agent coordination, as a key research direction in the field of artificial intelligence, has broad application prospects in many fields such as search and rescue, environmental monitoring, and collaborative robots. However, existing technologies face problems such as low training efficiency, excessive communication overhead, and insufficient utilization of computing resources during the operation of multi-agent systems, resulting in lengthy training times and performance bottlenecks.

[0003] Explained, traditional multi-agent communication methods have significant drawbacks, which are a major factor contributing to excessive communication overhead. Traditional methods require the transmission of complete observation information or high-dimensional feature vectors, placing extremely high demands on network bandwidth. For example, in visual perception tasks, the feature vectors output by visual language models can reach 768 dimensions. Directly transmitting these high-dimensional feature vectors consumes a large amount of network bandwidth, severely limiting the system's real-time performance and scalability. Furthermore, traditional methods lack effective information compression and prioritization mechanisms, making efficient coordination impossible with limited communication resources. The complete feature vectors contain a large amount of redundant information that is not critical to coordination decisions, further reducing communication efficiency. Moreover, traditional methods have insufficient information processing capabilities in complex scenarios, indirectly affecting training efficiency. Their limited understanding of complex visual scenes makes it difficult to extract key coordination information from massive amounts of data, requiring agents to process more irrelevant data during coordination, increasing computational burden and prolonging training time. Finally, traditional methods lack adaptive communication strategies and cannot dynamically adjust communication strategies according to task requirements and environmental changes. When faced with complex and ever-changing task scenarios, they cannot allocate communication resources reasonably, resulting in low resource utilization. Limited communication resources are not optimally configured, and the overall system performance is greatly reduced. Summary of the Invention

[0004] The purpose of this invention is to provide a zero-bandwidth multi-agent training method and system based on a multi-GPU distributed architecture, so as to solve the technical problems of low training efficiency, excessive communication overhead, and insufficient utilization of computing resources in the existing multi-agent training technology, which result in long training time and performance bottlenecks.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, this invention provides a zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture, comprising the following steps: Based on the selected multi-agent coordination task, environmental state information and a certain number of agents to be trained are obtained, and the environmental state information and agents are distributed and deployed on multiple GPU devices. Based on the environmental state information, each GPU device calls a multi-GPU distributed multi-agent network to iteratively train the assigned agents. After the number of iterations reaches a preset threshold, the trained agents are obtained. The multi-GPU distributed multi-agent network comprises a visual feature extraction module, a zero-bandwidth communication compression module, and an agent policy network module connected in sequence. During each training round, the visual feature extraction module takes the agent's original visual observation data as input, performs semantic feature vector extraction, feature normalization, and historical observation information integration, and outputs a temporally fused feature representation. The zero-bandwidth communication compression module takes the temporally fused feature representation as input, performs visual feature compression, message discretization encoding, communication necessity probability estimation, and communication decision, and outputs a joint representation of the compressed message vector and the communication decision probability. The agent policy network module takes the joint representation of the compressed message vector and the communication decision probability as input and outputs a joint representation of the action selection probability distribution, state value estimation, and continuous action parameter probability distribution.

[0006] A further improvement to the technical solution of this invention is that the zero-bandwidth communication compression module includes: an extreme compression encoder, a communication value estimator, a message discretization module, and a communication decision module; wherein, The extreme compression encoder is used to input the feature representation of time-series fusion and perform feature compression, outputting a continuous compressed feature vector; the message discretization module is used to input the continuous compressed feature vector and perform discretization transformation, outputting a discrete compressed message vector. The communication value estimator is used to input the feature representation of time-series fusion and evaluate the necessity and expected benefits of communication in the current state, and output the communication value estimation result; the communication decision module is used to input the communication value estimation result and make a decision, and output the communication decision probability of whether to send a message.

[0007] A further improvement to the technical solution of this invention lies in that the extreme compression encoder comprises: a feature compression network, a batch normalization layer, an activation function layer, and a compression dimension adaptive module; wherein, The feature compression network is used to input the feature representation of temporal fusion and generate a compressed feature representation according to the set target dimension; the batch normalization layer is used to standardize the compressed feature distribution after the feature compression network; the activation function layer is used to introduce a nonlinear transformation and output a continuous compressed feature vector after the batch normalization layer. The compression dimension adaptive module is used to dynamically adjust the target dimension according to the task complexity.

[0008] A further improvement to the technical solution of this invention lies in that the agent policy network module includes: a message fusion module, a policy network, a value network, and an action parameter network; wherein, The message fusion module is used to input the normalized features of the agent's original visual observation data after feature normalization processing, as well as the joint representation of compressed message vector and communication decision probability, and perform fusion processing to output the fused feature representation. The policy network is used as input to the fused feature representation and outputs an action selection probability distribution. The value network is used as input to the fused feature representation and outputs a state value estimate. The action parameter network is used to input the action selection probability distribution and output a continuous action parameter probability distribution.

[0009] A further improvement to the technical solution of this invention lies in the process of allocating and deploying intelligent agents across multiple GPU devices. First, obtain the number of GPU devices, the number of agents, and the status information of each GPU device. Then, use a round-robin algorithm or load balancing strategy to distribute and deploy agents to each GPU device.

[0010] A further improvement of the technical solution of the present invention is that, in the process of distributing and deploying environmental state information to multiple GPU devices, each GPU device is configured with a preset number of parallel environment instances.

[0011] A further improvement of the technical solution of the present invention is that the selected multi-agent coordination task is a target search, path planning, resource allocation or cooperative navigation task.

[0012] A second aspect of the present invention provides a zero-bandwidth multi-agent training system based on a multi-GPU distributed architecture, comprising: The data acquisition and deployment unit is used to acquire environmental state information and a certain number of agents to be trained based on the selected multi-agent coordination task, and to distribute and deploy the environmental state information and agents on multiple GPU devices. The iterative training unit is used to call a multi-GPU distributed multi-agent network to iteratively train the assigned agents based on the environmental state information. After the number of iterations reaches a preset threshold, the trained agents are obtained. The multi-GPU distributed multi-agent network comprises a visual feature extraction module, a zero-bandwidth communication compression module, and an agent policy network module connected in sequence. During each training round, the visual feature extraction module takes the agent's original visual observation data as input, performs semantic feature vector extraction, feature normalization, and historical observation information integration, and outputs a temporally fused feature representation. The zero-bandwidth communication compression module takes the temporally fused feature representation as input, performs visual feature compression, message discretization encoding, communication necessity probability estimation, and communication decision, and outputs a joint representation of the compressed message vector and the communication decision probability. The agent policy network module takes the joint representation of the compressed message vector and the communication decision probability as input and outputs a joint representation of the action selection probability distribution, state value estimation, and continuous action parameter probability distribution.

[0013] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture as described in any one of the first aspects of the present invention.

[0014] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture as described in any one of the first aspects of the present invention.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The technical solution disclosed in this invention is a multi-GPU zero-bandwidth multi-agent coordinated training scheme. Through a pre-designed distributed training architecture and an extreme compression communication mechanism, it achieves collaborative learning, solving the technical problems of low training efficiency caused by excessively high inter-agent communication bandwidth requirements in existing technologies. Specifically, this invention distributes environmental state information and agents across multiple GPU devices for parallel training. This architecture fully utilizes the computing power of multiple GPUs, significantly improving the utilization rate of computing resources. It enables the training process to handle more data and tasks simultaneously, greatly shortening training time and solving the problem of excessively long training times caused by insufficient utilization of computing resources. Based on a visual feature extraction module, this invention extracts semantic feature vectors, performs feature normalization processing, and integrates historical observation information from the agent's original visual observation data, outputting a temporal fusion feature representation. Compared to traditional methods that transmit complete observation information or high-dimensional feature vectors, this module can extract key information, remove redundancy, reduce data dimensionality, and reduce the amount of data for subsequent communication and computation, laying the foundation for reducing communication overhead and improving training efficiency. Furthermore, a zero-bandwidth communication compression module performs visual feature compression, message discretization encoding, communication necessity probability estimation, and communication decision-making on the temporal fusion feature representation. It outputs a joint representation of the compressed message vector and the communication decision probability. This module effectively compresses information, removing a large amount of redundant information. Simultaneously, through communication necessity probability estimation and decision-making, it avoids unnecessary communication, significantly reducing communication overhead and solving the problem of low communication efficiency caused by the lack of information compression and prioritization mechanisms in traditional methods. Finally, an agent policy network module takes the joint representation of the compressed message vector and the communication decision probability as input and outputs a joint representation of the action selection probability distribution, state value estimation, and continuous action parameter probability distribution. This module can make reasonable decisions based on compressed and filtered key information under limited communication resources, improving the agent's information processing capability in complex scenarios. It solves the problem of insufficient information processing capability in complex scenarios affecting training efficiency in traditional methods, ultimately achieving improved multi-agent training efficiency, reduced communication overhead, and full utilization of computing resources, breaking through performance bottlenecks. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture provided in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the overall architecture of a multi-GPU distributed multi-agent network in an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the zero-bandwidth communication compression module in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of the structure of the extreme compression encoder in an embodiment of the present invention.

[0021] Figure 5 This is a schematic diagram of the structure of the GPU allocation management module in an embodiment of the present invention.

[0022] Figure 6 This is a schematic diagram of the intelligent agent policy network module in an embodiment of the present invention.

[0023] Figure 7 This is a schematic diagram of the structure of the parallel environment execution module in an embodiment of the present invention.

[0024] Figure 8 This is a schematic diagram of a zero-bandwidth multi-agent training system based on a multi-GPU distributed architecture provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0026] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0027] Please see Figure 1 and Figure 2 In this embodiment of the invention, a zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture is provided, comprising the following steps: Step 1: Based on the selected multi-agent coordination task, acquire environmental state information and a certain number of agents to be trained, and distribute the environmental state information and agents to multiple GPU devices. For example, the selected multi-agent coordination task can be a target search, path planning, resource allocation, or cooperative navigation task, etc., and the environmental state information includes comprehensive information such as the position, orientation, visual observation, and task progress of each agent. Therefore, the technical solution of this invention can be applied to various scenarios requiring multi-agent collaborative work, such as search and rescue, environmental monitoring, collaborative robots, and drone swarms.

[0028] Step 2: Based on the environmental state information, each GPU device calls a multi-GPU distributed multi-agent network to iteratively train the assigned agents. After the number of iterations reaches a preset threshold, the trained agents are obtained. Explained, this step uses a multi-GPU distributed multi-agent network to perform agent coordination learning and obtain training results. The training results can be output in the form of agent policy parameter updates, coordination performance indicators, and training convergence status. Among them, such as Figure 2 As shown, the multi-GPU distributed multi-agent network includes a visual feature extraction module, a zero-bandwidth communication compression module, and an agent policy network module connected in sequence. During each training round, the visual feature extraction module takes the agent's original visual observation data as input, performs semantic feature vector extraction, feature normalization, and historical observation information integration, and outputs a temporally fused feature representation. The zero-bandwidth communication compression module takes the temporally fused feature representation as input, performs visual feature compression, message discretization encoding, communication necessity probability estimation, and communication decision, and outputs a joint representation of the compressed message vector and the communication decision probability. The agent policy network module takes the joint representation of the compressed message vector and the communication decision probability as input and outputs a joint representation of the action selection probability distribution, state value estimation, and continuous action parameter probability distribution.

[0029] In a specific exemplary technical solution, the visual feature extraction module includes a VLM feature processor, a feature normalization layer, and a feature fusion layer connected in series; the zero-bandwidth communication compression module and the agent policy network module are both deployed on multiple GPU devices; interpretably, the VLM feature processor is used to input raw visual observation data and extract 768-dimensional semantic feature vectors through a pre-trained Vision-Language Model (VLM); the feature normalization layer is used to normalize the extracted features to ensure the consistency of feature distribution observed by different agents; the feature fusion layer is used to integrate historical observation information and output a temporally fused feature representation; in a further specific exemplary technical solution, the VLM feature processor uses the SmolVLM-256M-Instruct model for visual scene understanding, the feature normalization uses the LayerNorm method to avoid batch size limitations, and the feature fusion uses a combination of weighted averaging and attention mechanisms to process temporal information.

[0030] Please see Figure 3 The zero-bandwidth communication compression module in this embodiment of the invention includes an extreme compression encoder, a communication value estimator, a message discretization module, and a communication decision module. It uses an adaptive weight adjustment mechanism to compress the input 768-dimensional visual features into 1-8 dimensional message vectors, estimate the probability of communication necessity, perform message discretization encoding, and make a communication decision. Specifically, the extreme compression encoder is used to input a 768-dimensional VLM feature vector and compress the feature space from 768 dimensions to the target dimension through a multi-layer fully connected network and a nonlinear activation function. The communication value estimator is used to evaluate the necessity and expected benefits of communication in the current state. The message discretization module is used to convert the continuous compressed feature vector into discrete message symbols, improving the robustness of communication. The communication decision module is used to decide whether to send a message based on the communication value estimation result. The zero-bandwidth communication compression module outputs a joint representation of the compressed message vector and the communication decision probability.

[0031] Please see Figure 4 The extreme compression encoder in this embodiment of the invention includes a feature compression network, a batch normalization layer, an activation function layer, and a compression dimension adaptive module. The feature compression network takes a 768-dimensional VLM feature vector as input and generates a compressed representation of the corresponding dimension according to a set compression ratio. The batch normalization layer standardizes the compressed feature distribution to avoid gradient vanishing or exploding. The activation function layer introduces nonlinear transformations to enhance the network's expressive power. The compression dimension adaptive module dynamically adjusts the compression dimension according to task complexity, finding the optimal balance between extreme compression and information preservation. The output of the extreme compression encoder is a low-dimensional message vector after multiple nonlinear transformations.

[0032] In one embodiment of the present invention, a GPU allocation management module is further included, which adds a distributed coordination mechanism to the agent policy network module. Specifically, the GPU allocation management module includes an agent allocation policy network, a load balancing network, a device status monitor, and a dynamic migration controller. It uses a round-robin algorithm or a load balancing strategy to evenly allocate agents to each GPU, monitor the load status of each GPU, perform load balancing, and handle device failure migration. The agent allocation policy network is used to input the number of agents and GPU configuration information, and output the optimal allocation scheme. The load balancing network is used to input real-time load data of each GPU and output a load adjustment strategy. The device status monitor is used to input GPU hardware status information and output a device health status assessment. The dynamic migration controller is used to input allocation adjustment instructions and output agent migration operations. The GPU allocation management module outputs a joint representation of the optimized agent-GPU mapping relationship and load balancing weights.

[0033] Please see Figure 5 In one embodiment of the present invention, the GPU allocation management module performs multi-GPU distributed deployment of agents, primarily using a round-robin polling method for allocation, and performs device allocation operations for each agent. The GPU allocation management module includes an agent allocation strategy network and a load balancing network. The module output is the product of the agent-to-GPU mapping relationship and the device utilization rate, and the obtained allocation scheme is grouped by device. The inputs of the agent allocation strategy network and the load balancing network are the current system status information. The operations of the agent allocation strategy network are round-robin mapping, balanced allocation, and manual assignment, while the operations of the load balancing network are resource monitoring, dynamic adjustment, and performance optimization.

[0034] In this invention, zero-bandwidth communication technology based on multi-GPU distributed deep reinforcement learning is applied to the field of multi-agent coordination. A novel extreme compression encoder is proposed to significantly reduce communication overhead, and a GPU allocation management module is designed to realize parallel distributed training of agents. Based on this, a distributed training network is constructed to solve multi-agent coordination training tasks. Further, the extreme compression encoder disclosed in this invention is based on deep neural networks and information theory principles, with extremely large differences in the dimensions of input and output data. It requires no complex external dependencies, making it easy to apply to various multi-agent learning models and can be extended to other coordination tasks, such as distributed optimization and federated learning, showing broad application prospects. In summary, the zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture provided by this invention uses an extreme compression-based communication mechanism to significantly reduce communication overhead between agents, and proposes a multi-GPU distributed training architecture to further effectively accelerate training and optimize resource utilization, enabling efficient multi-agent coordination learning and improving training efficiency and final performance. Therefore, the technical solution disclosed in this invention can solve the technical problems of low training efficiency and excessive communication overhead in the prior art, which lead to bottlenecks in multi-agent coordination learning.

[0035] In one embodiment of the present invention, the extreme compression encoder extracts key information from a 768-dimensional VLM feature vector and compresses it into a message vector of the target dimension, converting the high-dimensional feature representation into a low-dimensional communication message. The extreme compression encoder is represented as:

[0036]

[0037]

[0038] In the formula, M is the compressed message vector, F is the input 768-dimensional VLM feature, C is the binary output of the communication decision, θ is the communication threshold, budget_available is the communication budget availability state; Linear is the linear transformation layer; ReLU is the modified linear unit activation function; Tanh is the hyperbolic tangent activation function; CommValueNet is the communication value estimation network; Sigmoid is the sigmoid activation function; AND is the logical AND operation.

[0039] Please see Figure 6 In one embodiment of the present invention, the agent policy network module includes a distributed policy network, a value estimation network, an action parameter network, and a message fusion mechanism. The multi-GPU distributed policy network is as follows: , In the formula, is the parameterized action policy distribution; a is the agent action; s is the current state observation; m is the received compressed message; PolicyNet is the policy network; ActionHead is the action output header; Concat is the feature concatenation operation; The distributed value estimation is as follows:

[0040] In the formula, This is a parameterized state value estimate; ValueHead is the value estimate output header. The Actor-Critic architecture, consisting of a policy network and a value network, outputs the following: , In the formula, For the distance parameter distribution, t is the distribution of rotational parameters. This represents the complete decision output of the intelligent agent.

[0041] In one embodiment of the present invention, the training process of a multi-GPU distributed multi-agent network includes: 1) Environment initialization: Deploy the training environment and agent to multiple GPU devices respectively, and set the number of parallel environments to 4-8 instances per GPU; 2) Agent allocation: Based on the set allocation strategy, N agents are allocated to K GPU devices, using either round-robin or load balancing strategies; 3) Distributed data collection: Each GPU performs parallel environmental interactions to collect observation, action, reward, and communication data; 4) Experience aggregation: Aggregate and synchronize the experience data collected by each GPU through the AllGather operation; 5) Gradient calculation: The gradients of the strategy and value network are calculated in parallel on each GPU using the PPO algorithm; 6) Parameter update: Synchronize gradients across GPUs using the AllReduce operation to update network parameters; 7) Performance evaluation: After each training cycle, calculate metrics such as success rate, reward, and communication efficiency to evaluate the training effect; 8) Model saving: If the validation performance improves, save the current optimal model parameters.

[0042] In a specific embodiment of the present invention, the training process specifically includes the following steps: Step 1: Construct a multi-agent coordinated task environment; the collected environment includes multiple agents, target objects, obstacles and communication constraints; the environment size is set to a 10x10 grid world, supporting 3-12 agents to act simultaneously; the dataset is randomly generated with task configurations of different difficulties; the environment instances are distributed and deployed to multiple GPU executors.

[0043] Step 2: Construct a multi-GPU distributed training network, including sequentially connected visual feature extraction, zero-bandwidth communication compression, agent policy network, GPU allocation management, and parallel environment execution modules (exemplarily, such as...). Figure 7 (as shown) The multi-GPU distributed multi-agent network comprises K GPU devices, each GPU containing M parallel environments, supporting distributed training for a total of N agents; the workload of each GPU includes the following computational process: Environment status acquisition: ; Feature extraction: , In the formula, obs_i represents the observation data of agent i, F_i represents the extracted 768-dimensional feature vector, and VLMProcessor is the visual language model processor; each feature extraction operation is semantically understood through a pre-trained SmolVLM model; feature standardization uses LayerNorm to avoid batch size limitations. Feature fusion integrates temporal and spatial information to enhance the richness of observation representations; GPU allocation management ensures balanced load across devices, with agents allocated to GPUs according to the round_robin strategy (GPU_id = agent_id mod K); Parallel environment execution supports running M=4-8 environment instances simultaneously on each GPU.

[0044] In this embodiment of the invention, the policy head in the agent policy network is a distributed policy network, the value head is a distributed value estimation network, and the communication module is a zero-bandwidth communication compression module. The zero-bandwidth communication compression operation is as follows: , In the formula, F_norm is the normalized feature vector, M_compressed is the compressed message vector, and CompressionNet is the compression network.

[0045] In this embodiment of the invention, the zero-bandwidth communication compression module compresses the agent's observed features from 768 dimensions to a 1-8 dimension message space using an extreme compression encoder; wherein, 1-dimensional compression achieves an extreme compression ratio of 768:1, and 8-dimensional compression achieves a high-efficiency compression ratio of 96:1; wherein, the compression network adopts an adaptive architecture according to the target dimension d: when d=1, a deep compression path of [768→256→64→16→1] is used, and when d=8, a standard compression path of [768→512→256→64→8] is used; wherein, the communication decision mechanism is as follows: should_communicate = (comm_value>threshold) AND (messages_sent <budget), In the formula, comm_value is the estimated value of communication, threshold is the communication threshold, messages_sent is the number of messages sent, and budget is the upper limit of the communication budget. The multi-GPU distributed strategy has been updated as follows: , In the formula, θ represents the policy network parameters, α represents the learning rate, AllReduce represents the gradient aggregation operation, and L_policy represents the policy loss function; the gradients of each GPU are aggregated through distributed communication to achieve synchronous parameter updates.

[0046] In this embodiment of the invention, the GPU allocation management module is responsible for the device allocation and load balancing of the intelligent agent, and the distributed training architecture adopts a combination of data parallelism and model parallelism. The agent allocation strategy is as follows: , In the formula, i The index is for the intelligent agent, num_gpus is the number of GPU devices, and GPU_assignment is the allocation result; The load balancing monitoring includes: load_balance_score = std(GPU_utilization) / mean(GPU_utilization), In the formula, GPU_utilization is the utilization vector of each GPU, std is the standard deviation, and mean is the mean; load rebalancing is triggered when load_balance_score > 0.2.

[0047] In this embodiment of the invention, the parallel environment execution module uses thread pools and asynchronous I / O technology to achieve efficient environment management, supporting the simultaneous running of multiple environment instances per GPU; Thread pool configuration: ; Asynchronous environment reset: ; Parallel action execution: ; Empirical data collection: ; Data synchronization transmission: .

[0048] In this embodiment of the invention, distributed training uses PyTorch DistributedDataParallel to wrap the agent network and NCCL backend for inter-GPU communication; the training process uses the PPO algorithm, which supports multiple rounds of policy optimization and value function updates; communication compression significantly reduces the bandwidth requirements between agents, and multi-GPU parallelism greatly improves the training speed.

[0049] This invention applies multi-GPU distributed computing technology to the field of multi-agent reinforcement learning, proposing a novel zero-bandwidth communication compression mechanism for large-scale multi-agent coordination tasks. This results in an efficient communication protocol based on extreme compression, upon which a multi-agent distributed training network is constructed to solve complex coordination learning tasks. Addressing the shortcomings of existing technologies in considering communication and computational efficiency, this invention fully considers extreme feature compression and multi-GPU parallel acceleration, effectively improving the learning efficiency of multi-agent coordination tasks while reducing training time and communication overhead. The proposed zero-bandwidth communication mechanism is based on deep neural networks and information compression theory, with standardized input / output interfaces, eliminating complex dependencies and facilitating application in various multi-agent learning frameworks. Furthermore, it can be extended to other distributed learning tasks, such as federated learning and distributed optimization, demonstrating broad application prospects.

[0050] In the following specific embodiments of the present invention: the multi-agent coordination task dataset used includes various scenarios such as target search, path planning, and resource allocation, supporting coordinated learning of 3-12 agents; the environment size is a 10x10 grid, containing dynamic obstacles and time-varying targets; the task is randomly divided into training and testing sets, and the environment undergoes state normalization and action space discretization during the preprocessing stage; the multi-agent network in this embodiment is initialized using Xavier orthogonal initialization; the running environment is a multi-GPU server configured with frameworks such as PyTorch and NCCL, capable of handling large-scale parallel computing and completing the construction and training of the model of this method. The training time for this embodiment on an Intel Xeon Gold CPU, 128GB of memory, and an 8×NVIDIA GeForce RTX3090 GPU cluster is approximately 6 hours.

[0051] In this embodiment of the invention, the specific implementation steps include: first, setting relevant training parameters, setting the optimizer used for network updates in this invention to the AdamW optimizer, setting its weight decay to 0.01, setting the initial learning rate to 3e-4, and setting the learning rate decay coefficient to 0.99. The learning rate adjustment strategy is set to exponential decay, with the learning rate multiplied by the decay coefficient after each training round. The multi-GPU distributed multi-agent network of this embodiment consists of five core modules: a visual feature extraction module for observation processing, a zero-bandwidth communication compression module for extreme compression, an agent policy network module for decision inference, a GPU allocation and management module for device coordination, and a parallel environment execution module for efficient simulation. The network input is multi-agent observation data, which is processed collaboratively by each module to output action decisions and value estimates. The output decision results undergo multiple rounds of iterative optimization and are finally passed to the environment executor to obtain training results containing coordination effects and performance indicators. Next, when training the network using a multi-GPU cluster, empirical data is collected in parallel from each environment each time, and the selected AdamW optimizer is used for parameter updates. Training is completed after 500 iterations on the dataset. Finally, the test task was input into the trained network for verification, yielding the results of the embodiment of the invention. From the training curves and performance metrics provided in the embodiment, it can be seen that the method proposed in this invention achieves excellent multi-agent coordinated learning results. In the given test task, the success rate reaches over 85%, the communication compression ratio reaches 768:1 to 96:1, and the training speed is 4-8 times faster than that of a single GPU, demonstrating the effectiveness of this invention in efficient multi-agent coordinated learning tasks.

[0052] In summary, this invention discloses a zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture. The zero-bandwidth communication compression module significantly reduces communication overhead between agents; the GPU allocation and management module accelerates parallel distributed training of agents. This invention introduces multi-GPU distributed computing into the field of multi-agent coordinated learning, effectively improving training efficiency. Each module is highly efficient, without excessive dependency, and possesses strong practicality. The zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture provided by this invention achieves an efficient multi-agent coordination strategy, solving the performance bottlenecks caused by low training efficiency and excessive communication overhead in the prior art.

[0053] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0054] Please see Figure 8 In this embodiment of the invention, a zero-bandwidth multi-agent training system based on a multi-GPU distributed architecture is provided, comprising: The data acquisition and deployment unit is used to acquire environmental state information and a certain number of agents to be trained based on the selected multi-agent coordination task, and to distribute and deploy the environmental state information and agents on multiple GPU devices. The iterative training unit is used to call a multi-GPU distributed multi-agent network to iteratively train the assigned agents based on the environmental state information. After the number of iterations reaches a preset threshold, the trained agents are obtained. The multi-GPU distributed multi-agent network comprises a visual feature extraction module, a zero-bandwidth communication compression module, and an agent policy network module connected in sequence. During each training round, the visual feature extraction module takes the agent's original visual observation data as input, performs semantic feature vector extraction, feature normalization, and historical observation information integration, and outputs a temporally fused feature representation. The zero-bandwidth communication compression module takes the temporally fused feature representation as input, performs visual feature compression, message discretization encoding, communication necessity probability estimation, and communication decision, and outputs a joint representation of the compressed message vector and the communication decision probability. The agent policy network module takes the joint representation of the compressed message vector and the communication decision probability as input and outputs a joint representation of the action selection probability distribution, state value estimation, and continuous action parameter probability distribution.

[0055] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used to execute operations of a zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture.

[0056] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture in the above embodiments.

[0057] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0058] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture, characterized in that, Includes the following steps: Based on the selected multi-agent coordination task, environmental state information and a certain number of agents to be trained are obtained, and the environmental state information and agents are distributed and deployed on multiple GPU devices. Based on the environmental state information, each GPU device calls a multi-GPU distributed multi-agent network to iteratively train the assigned agents. After the number of iterations reaches a preset threshold, the trained agents are obtained. The multi-GPU distributed multi-agent network comprises a visual feature extraction module, a zero-bandwidth communication compression module, and an agent policy network module connected in sequence. During each training round, the visual feature extraction module takes the agent's original visual observation data as input, performs semantic feature vector extraction, feature normalization, and historical observation information integration, and outputs a temporally fused feature representation. The zero-bandwidth communication compression module takes the temporally fused feature representation as input, performs visual feature compression, message discretization encoding, communication necessity probability estimation, and communication decision, and outputs a joint representation of the compressed message vector and the communication decision probability. The agent policy network module takes the joint representation of the compressed message vector and the communication decision probability as input and outputs a joint representation of the action selection probability distribution, state value estimation, and continuous action parameter probability distribution.

2. The zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture according to claim 1, characterized in that, The zero-bandwidth communication compression module includes: an extreme compression encoder, a communication value estimator, a message discretization module, and a communication decision module; wherein... The extreme compression encoder is used to input the feature representation of time-series fusion and perform feature compression, outputting a continuous compressed feature vector; the message discretization module is used to input the continuous compressed feature vector and perform discretization transformation, outputting a discrete compressed message vector. The communication value estimator is used to input the feature representation of time-series fusion and evaluate the necessity and expected benefits of communication in the current state, and output the communication value estimation result; the communication decision module is used to input the communication value estimation result and make a decision, and output the communication decision probability of whether to send a message.

3. The zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture according to claim 2, characterized in that, The extreme compression encoder includes: a feature compression network, a batch normalization layer, an activation function layer, and a compression dimension adaptive module; wherein... The feature compression network is used to input the feature representation of temporal fusion and generate a compressed feature representation according to the set target dimension; the batch normalization layer is used to standardize the compressed feature distribution after the feature compression network; the activation function layer is used to introduce a nonlinear transformation and output a continuous compressed feature vector after the batch normalization layer. The compression dimension adaptive module is used to dynamically adjust the target dimension according to the task complexity.

4. The zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture according to claim 1, characterized in that, The agent policy network module includes: a message fusion module, a policy network, a value network, and an action parameter network; wherein... The message fusion module is used to input the normalized features of the agent's original visual observation data after feature normalization processing, as well as the joint representation of compressed message vector and communication decision probability, and perform fusion processing to output the fused feature representation. The policy network is used as input to the fused feature representation and outputs an action selection probability distribution. The value network is used as input to the fused feature representation and outputs a state value estimate. The action parameter network is used to input the action selection probability distribution and output a continuous action parameter probability distribution.

5. The zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture according to claim 1, characterized in that, In the process of distributing and deploying agents across multiple GPU devices First, obtain the number of GPU devices, the number of agents, and the status information of each GPU device. Then, use a round-robin algorithm or load balancing strategy to distribute and deploy agents to each GPU device.

6. The zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture according to claim 1, characterized in that, In the process of distributing and deploying environmental state information across multiple GPU devices, each GPU device is configured with a preset number of parallel environment instances.

7. The zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture according to claim 1, characterized in that, The selected multi-agent coordination tasks are target search, path planning, resource allocation, or cooperative navigation.

8. A zero-bandwidth multi-agent training system based on a multi-GPU distributed architecture, characterized in that, include: The data acquisition and deployment unit is used to acquire environmental state information and a certain number of agents to be trained based on the selected multi-agent coordination task, and to distribute and deploy the environmental state information and agents on multiple GPU devices. The iterative training unit is used to call a multi-GPU distributed multi-agent network to iteratively train the assigned agents based on the environmental state information. After the number of iterations reaches a preset threshold, the trained agents are obtained. The multi-GPU distributed multi-agent network comprises a visual feature extraction module, a zero-bandwidth communication compression module, and an agent policy network module connected in sequence. During each training round, the visual feature extraction module takes the agent's original visual observation data as input, performs semantic feature vector extraction, feature normalization, and historical observation information integration, and outputs a temporally fused feature representation. The zero-bandwidth communication compression module takes the temporally fused feature representation as input, performs visual feature compression, message discretization encoding, communication necessity probability estimation, and communication decision, and outputs a joint representation of the compressed message vector and the communication decision probability. The agent policy network module takes the joint representation of the compressed message vector and the communication decision probability as input and outputs a joint representation of the action selection probability distribution, state value estimation, and continuous action parameter probability distribution.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the zero-bandwidth multi-agent training method based on a multi-GPU distributed architecture as described in any one of claims 1 to 7.

Images