Multi-agent reinforcement learning training method, system and equipment driven by unreal engine and medium

Abstract

The invention relates to a multi-agent reinforcement learning training method, system and device driven by an unreal engine and a medium. The method comprises the following steps: acquiring core attribute data of multiple agents, including real-time communication bandwidth, initial communication topology configuration, training stage progress, single agent computing power and cluster overall computing load state; analyzing the real-time communication bandwidth and the initial communication topology configuration to form an optimized topology configuration scheme; in combination with an optimized topology configuration scheme and a training stage progress, generating a cooperative training progress state by quantifying progress differences among intelligent agents; extracting load balance data according to the single agent computing capability and the cluster load state; integrating load balance data, a collaborative training progress state and an optimized topology configuration scheme, and constructing a multi-layer collaborative training framework; and integrated optimization is carried out to generate a cooperative training scheme for guiding multi-agent communication, training and resource allocation, so that the training efficiency is effectively improved, the resource load is balanced, and the cooperative synchronism is enhanced.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent reinforcement learning training, and in particular relates to multi-agent reinforcement learning training methods, systems, devices and media driven by Unreal Engine. Background Technology

[0002] With the rapid development of multi-agent reinforcement learning training technology, Unreal Engine-driven multi-agent reinforcement learning training technology has emerged. Leveraging Unreal Engine's powerful high-fidelity environment construction capabilities, real-time rendering performance, and flexible resource scheduling interfaces, this technology provides multi-agent training environments that closely resemble real-world application scenarios. By simulating complex dynamic scenarios, it enhances the environmental adaptability and collaborative decision-making capabilities of multi-agent agents, thereby promoting the practical application of multi-agent reinforcement learning in fields such as autonomous driving and intelligent robot collaboration.

[0003] Traditional multi-agent reinforcement learning training often employs a training mode based on simple simulation environments. In terms of communication topology configuration, it directly uses a fixed node connection structure, without considering the impact of real-time communication status on topology transmission efficiency. Regarding training progress coordination, it only achieves rough progress alignment through periodic synchronization of training data, lacking precise quantification and dynamic adjustment of differences in training progress among agents. In terms of resource allocation, it uses a static allocation strategy based on the initial computing power of each agent, without dynamic optimization based on real-time changes in the overall cluster computing load. Traditional methods first pre-set a fixed multi-agent communication topology, then allocate a fixed proportion of computing resources according to the initial computing power of each agent. During training, the training data of each agent is synchronized at fixed time intervals to maintain progress consistency, and training ends based on a pre-set number of training iterations or task completion indicators.

[0004] However, current traditional methods have many problems: First, fixed communication topologies cannot adapt to the dynamic fluctuations of real-time communication bandwidth. When the bandwidth supply capacity does not match the topology transmission demand, data transmission delays and link congestion are likely to occur, reducing the efficiency of training data interaction. Second, there is a lack of precise quantification of the differences in training progress among agents. Periodic synchronization alone is insufficient to eliminate progress deviations, causing some agents to slow down the overall training process due to lagging progress or waste computing resources due to advanced progress, resulting in poor collaborative synchronization. Third, static resource allocation strategies cannot respond to real-time changes in the overall computing load of the cluster, causing a mismatch between the load allocation of a single agent and the idle resource status of the cluster, resulting in load imbalance. This affects the computing efficiency of a single agent and reduces the overall computing resource utilization of the cluster, ultimately leading to low training efficiency and poor training results in multi-agent reinforcement learning. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, system, device, and medium for training multi-agent reinforcement learning driven by Unreal Engine that can solve the above problems.

[0006] Firstly, this application provides a multi-agent reinforcement learning training method driven by Unreal Engine, including:

[0007] Acquire core attribute data of multi-agent systems, including real-time communication bandwidth, initial communication topology configuration, training phase progress, single agent computing power, and overall cluster computing load status.

[0008] A coupling analysis is performed on the real-time communication bandwidth and the initial communication topology configuration to obtain the coupling analysis results. Based on the coupling analysis results, the communication topology structure between multiple agents is adjusted to obtain an optimized topology configuration scheme.

[0009] By combining the optimized topology configuration scheme with the training phase progress, the collaborative training progress status is generated by quantifying the progress differences among the agents.

[0010] Based on the computing power of a single agent and the overall computing load status data of the cluster, load balancing data is extracted from the computing resource pool associated with Unreal Engine. The load balancing data includes the load allocation ratio of a single agent, the amount of idle computing resources in the cluster, and the resource scheduling response latency.

[0011] By combining load balancing data, collaborative training progress status, and optimized topology configuration schemes, a collaborative training framework is constructed.

[0012] Based on the collaborative training framework, the collaborative training scheme is generated by integrating and optimizing the topology configuration scheme, collaborative training progress status and load balancing data. The collaborative training scheme is used to guide the communication, training and resource allocation of multiple agents.

[0013] In one embodiment, a coupling analysis is performed on the real-time communication bandwidth and the initial communication topology configuration to obtain the coupling analysis results. Based on the coupling analysis results, the communication topology among multiple agents is adjusted to obtain an optimized topology configuration scheme, including:

[0014] Extract the dynamic characteristic parameters of real-time communication bandwidth and calculate the bandwidth supply capacity based on the dynamic characteristic parameters. The dynamic characteristic parameters include bandwidth fluctuation amplitude, peak bandwidth duration and minimum available bandwidth.

[0015] The topology parameters of the initial communication topology configuration are parsed, and the topology transmission requirements are calculated based on the topology parameters. The topology parameters include the connection method of the agent nodes, the link transmission weight, and the node hierarchy.

[0016] The matching degree between bandwidth supply and topology transmission demand is calculated to obtain the coupling analysis results;

[0017] Based on the coupling analysis results, nodes and associated links with matching degrees below a preset threshold in the initial communication topology are identified as nodes to be optimized and related links to be optimized.

[0018] The nodes to be optimized and their associated links to be optimized are reconstructed and optimized to obtain reconstructed nodes and reconstructed associated links. Based on the computing power of the reconstructed nodes, the transmission weight allocation ratio of the reconstructed associated links is adjusted to obtain the reconstructed communication topology.

[0019] The optimization topology is formed by replacing the nodes and associated links to be optimized in the initial communication topology with a reconstructed communication topology.

[0020] Link connectivity detection is performed on the optimized topology to obtain the link connectivity detection results. Based on the link connectivity detection results, an optimized topology configuration scheme is generated according to the optimized topology that meets the preset connectivity standards.

[0021] In one embodiment, by combining the optimized topology configuration scheme with the training phase progress, a collaborative training progress state is generated by quantifying the progress differences among the agents, including:

[0022] Analyze the training progress of each agent to obtain progress parameters, including the number of training iterations, the percentage of task completion, and the amount of model parameter updates.

[0023] Based on the topology structure of the optimized topology configuration scheme, the cooperative association nodes corresponding to each agent are determined;

[0024] Based on the progress parameters of each agent and the corresponding collaborative nodes of each agent, the absolute difference in progress and the progress deviation rate between each pair of agents are calculated.

[0025] The absolute schedule difference and schedule deviation rate were standardized using a normalization method to obtain normalized schedule difference data.

[0026] Based on the preset progress difference threshold range, the normalized progress difference data is classified into levels to determine the training progress level of each agent.

[0027] By integrating the training progress levels, collaborative association nodes, and normalized progress difference data of each agent, a collaborative training progress status is generated.

[0028] In one embodiment, a collaborative training framework is constructed by combining load balancing data, collaborative training progress status, and optimized topology configuration scheme, including:

[0029] By combining load balancing data, collaborative training progress status, and optimized topology configuration schemes, the construction dimensions of the collaborative training framework are determined.

[0030] Based on the construction dimension, the hierarchical structure of the collaborative training framework is divided. The hierarchical structure includes a communication layer corresponding to the optimized topology configuration scheme, a progress coordination layer corresponding to the collaborative training progress status, and a resource allocation layer corresponding to the load balancing data.

[0031] Based on the hierarchical structure, load balancing data, collaborative training progress status, and optimized topology configuration scheme are mapped to the corresponding resource allocation layer, progress coordination layer, and communication layer in the hierarchical structure, respectively.

[0032] Based on the mapped resource allocation layer, progress coordination layer and communication layer, the data processing flow, transmission path and data interaction triggering conditions of each layer of the hierarchical structure are set according to the preset hierarchical interaction rules.

[0033] By integrating hierarchical structure, data processing flow, transmission path, and data interaction event triggering conditions, a collaborative training framework is formed.

[0034] In one embodiment, based on the collaborative training framework, and combining optimized topology configuration schemes, collaborative training progress status, and load balancing data, an integrated optimization is performed to generate a collaborative training scheme, including:

[0035] Optimized topology configuration parameters are extracted from the communication layer, collaborative training progress status parameters are extracted from the progress coordination layer, and load balancing data parameters are extracted from the resource allocation layer. The core parameters of optimized topology configuration, collaborative training progress status, and load balancing data parameters are then integrated to form a real-time input parameter set.

[0036] The real-time input parameter set is input into the pre-trained deep learning optimization model to obtain the parameter prediction results of the collaborative training scheme.

[0037] Based on the preset hierarchical interaction rules, the prediction results of the initial collaborative training scheme parameters are verified, and collaborative training scheme parameters that meet the communication link connectivity threshold, training progress difference level threshold, and load distribution balance threshold are selected.

[0038] Integrate the parameters of the selected collaborative training schemes to generate a collaborative training scheme.

[0039] In one embodiment, the hierarchical structure, data processing flow, transmission path, and data interaction event triggering conditions are integrated to form a collaborative training framework, which is implemented mathematically as follows:

[0040] F = w C ·Φ(C,T) C E C )+w P ·Φ(P,T) P E P )+w R ·Φ(R,T) RE R )+λ·Ψ(L,D C-P D P-R )

[0041] Where F is the mathematical representation of the collaborative training framework, w C w P and w R The weight coefficients for the communication layer, schedule coordination layer, and resource allocation layer are respectively, satisfying w C +w P +w R =1, C is the optimal topology configuration parameter in the communication layer, P is the collaborative training progress status parameter in the progress coordination layer, R is the load balancing data parameter in the resource allocation layer, and T C T P and T R These are the transmission path parameter sets for the communication layer, the schedule coordination layer, and the resource allocation layer, respectively. C E P and E R These are the data interaction trigger condition parameter sets for the communication layer, progress coordination layer, and resource allocation layer, respectively. Φ(·) is the intra-layer integration operator, used to couple and integrate the structural parameters, transmission path parameters, and trigger condition parameters of a single layer to satisfy... This represents element-level collaborative operations after parameter dimension matching, where λ is the constraint coefficient of the hierarchical interaction rule, with a value range of (0,1], Ψ(·) is the cross-level interaction integration operator, L is the preset hierarchical interaction rule parameter set, and D... C-P and D P-R These are cross-level data processing parameter sets for the communication layer and the schedule coordination layer, and for the schedule coordination layer and the resource allocation layer, respectively.

[0042] In one embodiment, load balancing data is extracted from the computing resource pool associated with Unreal Engine based on the computing capabilities of a single agent and the overall computing load status data of the cluster, including:

[0043] Based on the computing capabilities of a single agent and the overall computing load status of the cluster, a corresponding data mapping relationship is established between the computing resource pool associated with Unreal Engine;

[0044] Based on the corresponding data mapping relationship, raw load data related to the computing capabilities of a single agent and the overall computing load status of the cluster are filtered out from the computing resource pool associated with Unreal Engine;

[0045] The raw load data is quantified and calculated to obtain the load distribution ratio of a single agent, the amount of idle computing resources in the cluster, and the resource scheduling response delay data.

[0046] The load balancing data is formed by integrating the load distribution ratio of single agents, the amount of idle computing resources in the cluster, and the resource scheduling response delay data.

[0047] Secondly, this application also provides a Unreal Engine-driven multi-agent reinforcement learning training system, including:

[0048] The multi-agent data acquisition module acquires the core attribute data of the multi-agent, including real-time communication bandwidth, initial communication topology configuration, training phase progress, single agent computing power, and overall cluster computing load status.

[0049] The communication topology optimization module performs a coupling analysis on the real-time communication bandwidth and the initial communication topology configuration, obtains the coupling analysis results, and adjusts the communication topology structure between multiple agents based on the coupling analysis results to obtain an optimized topology configuration scheme.

[0050] The collaborative training optimization module combines the optimized topology configuration scheme with the training phase progress, and generates the collaborative training progress status by quantifying the progress differences between each agent.

[0051] The load data acquisition module extracts load balancing data from the computing resource pool associated with Unreal Engine based on the computing capabilities of a single agent and the overall computing load status data of the cluster. The load balancing data includes the load allocation ratio of a single agent, the amount of idle computing resources in the cluster, and the resource scheduling response latency.

[0052] The training framework building module combines load balancing data, collaborative training progress status, and optimized topology configuration schemes to construct a collaborative training framework.

[0053] The training scheme generation module, based on the collaborative training framework, integrates and optimizes the topology configuration scheme, collaborative training progress status, and load balancing data to generate a collaborative training scheme. The collaborative training scheme is used to guide the communication, training, and resource allocation of multiple agents.

[0054] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described Unreal Engine-driven multi-agent reinforcement learning training method.

[0055] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described Unreal Engine-driven multi-agent reinforcement learning training method.

[0056] The aforementioned Unreal Engine-driven multi-agent reinforcement learning training method, system, device, and medium acquire core attribute data such as real-time communication bandwidth, initial communication topology configuration, training phase progress, individual agent computing power, and overall cluster computing load status of the multi-agents. It then performs coupled analysis on real-time communication bandwidth and initial communication topology configuration to obtain an optimized topology configuration scheme, solving the problem that traditional fixed communication topologies cannot adapt to dynamic bandwidth fluctuations, leading to data transmission delays or congestion. Combining this optimized topology configuration scheme with the training phase progress, it quantifies the progress differences between agents to generate a collaborative training progress status, improving the poor collaboration problem caused by traditional methods that only periodically synchronize data and struggle to eliminate progress deviations. Based on the individual agent computing power and the overall cluster computing load status, it extracts load balancing data, constructs a collaborative training framework, and integrates and optimizes to generate a collaborative training scheme. This avoids the load imbalance problem caused by traditional static resource allocation failing to respond to changes in cluster load, effectively improving the efficiency of multi-agent reinforcement learning training, balancing resource load, and enhancing collaborative synchronization. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of the Unreal Engine-driven multi-agent reinforcement learning training method of the present invention.

[0059] Figure 2 This is a structural diagram of the Unreal Engine-driven multi-agent reinforcement learning training system of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] In one embodiment, such as Figure 1As shown, a multi-agent reinforcement learning training method driven by Unreal Engine is provided. This embodiment illustrates the application of this method to a terminal as an example. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and implemented through the interaction between the terminal and the server. In the implementation environment, the terminal device corresponds to a multi-agent node, such as an embedded computing unit on an autonomous vehicle or intelligent robot, responsible for collecting core attribute data such as communication bandwidth, training progress, and local computing status. The server side relies on the Unreal Engine platform and consists of a high-performance computing cluster, possessing simulation rendering and data processing capabilities, used to perform communication topology coupling analysis, progress coordination state generation, and load balancing optimization. The network infrastructure provides dynamically adjustable communication links, supporting bandwidth fluctuation adaptation. In application scenarios, such as the multi-vehicle collaborative training requirements in intelligent transportation, the terminal agent uploads the collected data to the server via 5G or wireless network. The server simulates complex dynamic scenarios based on the Unreal Engine environment, performs topology reconstruction and resource scheduling analysis, generates a collaborative training scheme, and then distributes it to the terminal to guide the communication interaction, training synchronization, and computing resource allocation among the multi-agents, improving overall training efficiency and load balancing.

[0062] In this embodiment, the method includes the following steps:

[0063] S01, acquire the core attribute data of the multi-agent system. The core attribute data includes real-time communication bandwidth, initial communication topology configuration, training phase progress, single agent computing power and overall cluster computing load status.

[0064] Here, an agent is a multiple independent intelligent computing unit participating in reinforcement learning training, which can be distributed across computing cluster nodes associated with Unreal Engine; real-time communication bandwidth refers to the real-time data transmission rate and available bandwidth resources of the communication link when multiple agents interact with training data; initial communication topology configuration refers to the preset agent node connection structure, link association relationship, and transmission parameter configuration before training starts; training phase progress refers to the phase position of each agent in the reinforcement learning training process and the relevant data on the degree of completion of training tasks; single agent computing power refers to the computing performance of a single agent, reflected in computing power specifications, storage capacity, and data processing rate; and the overall cluster computing load status refers to the overall resource occupancy of the computing cluster to which the multiple agents belong, including resource utilization and load intensity. In implementation, the aforementioned core attribute data can be obtained in real time through data acquisition modules and interfaces adapted to Unreal Engine: real-time communication bandwidth can be obtained by collecting dynamic transmission data of communication links between multiple agents from the network monitoring unit; the initial communication topology configuration can be retrieved from the topology configuration storage unit preset by the training system; the training phase progress can be determined by reading information such as iteration count records and task completion markers in the training logs of each agent; the computing power of a single agent is calculated by combining agent hardware parameters and real-time computing efficiency test results; the overall computing load status of the cluster is obtained by integrating the CPU, memory, GPU and other resource usage data of the servers in the cluster collected by the resource monitoring unit of the cluster management node.

[0065] S02, perform coupling analysis on real-time communication bandwidth and initial communication topology configuration, obtain coupling analysis results, and adjust the communication topology structure among multiple agents based on the coupling analysis results to obtain an optimized topology configuration scheme.

[0066] The coupling analysis involves correlating and adapting the real-time communication bandwidth with the core characteristic parameters of the initial communication topology configuration, focusing on evaluating the degree of matching between bandwidth supply capacity and topology transmission requirements. The coupling analysis results refer to the matching degree data obtained through this analysis, as well as the identification information of links and nodes to be optimized. The optimized topology configuration scheme refers to the multi-agent communication topology and corresponding transmission parameter configuration that, after adjustment based on the coupling analysis results, can adapt to the real-time bandwidth status. In implementation, based on the dynamic transmission data of real-time communication bandwidth (such as real-time available bandwidth value and bandwidth fluctuation range) and the structural parameters of the initial communication topology configuration, such as node connection relationships and link transmission weights, coupling analysis is conducted using a preset matching degree evaluation algorithm (such as calculating the difference between bandwidth supply and the transmission requirements of each link in the topology, and calculating the overall adaptation rate) to obtain matching degree results. Based on these results (such as associated links or nodes with matching degrees below a preset threshold), the node connection relationships and link transmission weights among the multi-agents are adjusted to form an optimized topology configuration scheme that can adapt to the real-time bandwidth status and ensure the efficiency of training data interaction.

[0067] S03 combines the optimized topology configuration scheme with the training phase progress, and generates a collaborative training progress status by quantifying the progress differences between each agent.

[0068] The collaborative training progress status is a comprehensive status data reflecting the collaborative training progress of multiple agents after integrating the collaborative relationships and quantified progress differences among the agents under the optimized topology configuration. In implementation, based on the optimized topology configuration scheme, the corresponding collaborative nodes for each agent under the current topology are determined. The training phase progress data of each agent is acquired through the training data acquisition module. Quantitative algorithms (such as calculating the absolute difference in progress and the deviation rate) are used to process the progress data of each agent and its real-time collaborative nodes to obtain quantified progress difference results. The collaborative node information of each agent, the quantified progress difference data, and the corresponding core information of the training phase progress are integrated to generate a collaborative training progress status that intuitively reflects the collaborative training progress of multiple agents.

[0069] S04 extracts load balancing data from the computing resource pool associated with Unreal Engine based on the computing capabilities of a single agent and the overall computing load status data of the cluster. The load balancing data includes the load allocation ratio of a single agent, the amount of idle computing resources in the cluster, and the resource scheduling response latency.

[0070] The Unreal Engine-associated computing resource pool is a collection of resources that are associated with Unreal Engine for resource scheduling and provide computing resource support for multi-agent reinforcement learning training. Load balancing data is a dataset reflecting the balanced distribution of computing resources among multiple agents and the efficiency of cluster resource idleness and scheduling. Specifically, the single-agent load allocation ratio is the proportion of cluster computing resources occupied by a single agent, the cluster idle computing resource amount is the scale of unused computing resources in the cluster, and the resource scheduling response latency is the response time of the cluster to an agent's resource scheduling request. In implementation, based on the single-agent computing capability and the overall cluster computing load status data, a corresponding data mapping relationship with the Unreal Engine-associated computing resource pool is established. Based on this mapping relationship, raw load data directly related to the single-agent computing capability and the overall cluster computing load status is selected from the computing resource pool. The raw load data is processed using a preset quantization algorithm to calculate the single-agent load allocation ratio, the cluster idle computing resource amount, and the resource scheduling response latency, respectively. These three types of data are then integrated to form the load balancing data.

[0071] S05, combining load balancing data, collaborative training progress status, and optimized topology configuration scheme, constructs a collaborative training framework.

[0072] The collaborative training framework is a structured system that integrates load balancing data, collaborative training progress status, and optimized topology configuration schemes, standardizing the communication, progress coordination, and resource allocation logic of multi-agent systems. In implementation, the framework construction dimensions are determined based on the core characteristics of load balancing data, collaborative training progress status, and optimized topology configuration schemes. Based on these dimensions, a communication layer corresponding to the optimized topology configuration scheme, a progress coordination layer corresponding to the collaborative training progress status, and a resource allocation layer corresponding to the load balancing data are defined. These three types of data are mapped to their respective layers. Combined with preset layer interaction rules (such as data transmission priority and interaction triggering conditions), the data processing flow and transmission path for each layer are set. The Unreal Engine's framework integration unit integrates the layer structure, data processing flow, transmission path, and interaction rules to form a collaborative training framework capable of supporting subsequent integration and optimization.

[0073] S06, based on the collaborative training framework, integrates and optimizes the topology configuration scheme, collaborative training progress status and load balancing data to generate a collaborative training scheme, which is used to guide the communication, training and resource allocation of multiple agents.

[0074] Integrated optimization, within the collaborative training framework, involves comprehensively adapting and adjusting core parameters of the optimized topology configuration scheme, collaborative training progress status, and load balancing data to achieve optimal synergy between multi-agent communication efficiency, training progress coordination, and resource load balancing. The collaborative training scheme refers to the specific execution plan formed after integrated optimization, which directly guides the communication interaction rules, training progress synchronization methods, and computational resource allocation strategies of multiple agents. In implementation, optimized topology configuration parameters, collaborative training progress status parameters, and load balancing data parameters can be extracted from corresponding levels of the collaborative training framework. Based on the Unreal Engine's optimization computing unit, preset integrated optimization logic is used to integrate and adapt these three types of parameters, ensuring that the parameters meet the basic requirements of efficient communication links, synchronized training progress, and balanced resource allocation. Compliance verification of the adapted parameters is performed according to preset standards (such as communication connectivity and progress difference thresholds). The integrated parameters that pass verification generate the collaborative training scheme, which is then transmitted to each agent through the Unreal Engine's command delivery interface, directly guiding their communication, training, and resource allocation operations.

[0075] The aforementioned Unreal Engine-driven multi-agent reinforcement learning training method acquires core attribute data such as real-time communication bandwidth and initial communication topology configuration, and performs coupled analysis on these two data points. It dynamically adjusts the communication topology to generate an optimized topology configuration scheme, ensuring that bandwidth supply matches topology transmission demand and avoiding data transmission delays or congestion. Combining the optimized topology configuration scheme with the training phase progress, it generates a collaborative training progress status by quantifying the progress differences among agents. This enables precise monitoring and dynamic adjustment of progress deviations, eliminating the shortcomings of traditional periodic synchronization and enhancing the collaborative synchronization among multiple agents. Based on the computing power of individual agents and the overall computing load status of the cluster, it extracts load balancing data and integrates it into the collaborative training framework. Through integrated optimization, it generates a collaborative training scheme, achieving dynamic optimization of computing resource allocation and improving cluster resource utilization and load balancing.

[0076] In one embodiment, a coupling analysis is performed on the real-time communication bandwidth and the initial communication topology configuration to obtain the coupling analysis results. Based on the coupling analysis results, the communication topology among multiple agents is adjusted to obtain an optimized topology configuration scheme, including:

[0077] S11, extract the dynamic characteristic parameters of real-time communication bandwidth and calculate the bandwidth supply capacity based on the dynamic characteristic parameters. The dynamic characteristic parameters include bandwidth fluctuation amplitude, peak bandwidth duration and minimum available bandwidth.

[0078] S12, parse the topology parameters of the initial communication topology configuration, and calculate the topology transmission requirements based on the topology parameters. The topology parameters include the connection method of the agent nodes, the link transmission weight, and the node hierarchy relationship.

[0079] S13, calculate the matching degree between bandwidth supply and topology transmission demand to obtain the coupling analysis results;

[0080] S14. Based on the coupling analysis results, identify nodes and related links in the initial communication topology that have a matching degree lower than a preset threshold and need to be optimized.

[0081] S15, reconstruct and optimize the node to be optimized and the associated link to be optimized to obtain the reconstructed node and the reconstructed associated link, and adjust the transmission weight allocation ratio of the reconstructed associated link according to the computing power of the reconstructed node to obtain the reconstructed communication topology.

[0082] S16, the nodes and associated links to be optimized in the initial communication topology are replaced by reconstructing the communication topology to form an optimized topology;

[0083] S17. Perform link connectivity detection on the optimized topology structure to obtain the link connectivity detection results. Based on the link connectivity detection results, generate an optimized topology configuration scheme according to the optimized topology structure whose connectivity meets the preset standards.

[0084] For example, bandwidth fluctuation amplitude, peak bandwidth duration, and minimum available bandwidth can be extracted from the bandwidth data of the communication link as dynamic feature parameters. A weighted algorithm (e.g., weighted summation with minimum available bandwidth weighted at 0.4, bandwidth fluctuation amplitude weighted at 0.3, and peak bandwidth duration weighted at 0.3) is used to calculate the bandwidth supply capacity. The initial communication topology configuration is parsed from the initial configuration file stored in the training system, and the agent node connection method, link transmission weight, and node hierarchy relationship are extracted as topology parameters. The topology transmission requirement is calculated based on the sum of the products of each link transmission weight and the preset data interaction amount between the corresponding node hierarchy. The matching degree between the bandwidth supply capacity and the topology transmission requirement is calculated (a ratio ≥ 1 indicates sufficient matching, and a ratio < 1 indicates insufficient matching). This matching degree value is the coupling analysis result. Based on a preset matching degree threshold (e.g., 0.8), all nodes and links in the initial communication topology are traversed, and nodes with a matching degree lower than 0.8 are selected as nodes to be optimized, and all their associated links are selected as associated links to be optimized. The nodes to be optimized are replaced with multi-agent... Within the computing cluster to which the energy source belongs, nodes with sufficient computing power and currently idle are designated as reconstruction nodes. For the links to be optimized, the link connection paths are adjusted to avoid bandwidth bottlenecks, forming reconstructed links. Computational capability parameters such as the number of floating-point operations per second of the reconstruction nodes are obtained. The transmission weight allocation ratio of the reconstructed links is adjusted according to the computing power ratio of the reconstruction nodes (e.g., if the computing power of reconstruction node A is twice that of node B, then the weight of the reconstructed link corresponding to A is set to twice that of B). This is integrated to form a reconstructed communication topology. This reconstructed communication topology is used to replace the nodes and links to be optimized in the initial communication topology one by one, retaining the parts of the initial topology that meet the matching requirements, forming a complete optimized topology. The ping command combined with port scanning tools can be used to perform connectivity testing on all links in the optimized topology, recording the stable transmission time and packet loss rate of each link to obtain the link connectivity test results. Optimized topologies with a connectivity rate ≥98% and no continuous packet loss are selected. Based on this structure, an optimized topology configuration scheme including node identifiers, link connection relationships, transmission weights, and communication protocols is generated.

[0085] In one embodiment, by combining the optimized topology configuration scheme with the training phase progress, a collaborative training progress state is generated by quantifying the progress differences among the agents, including:

[0086] S21, analyze the training progress of each agent to obtain progress parameters, including the number of training iterations, the percentage of task completion, and the amount of model parameter updates.

[0087] S22, Based on the topology structure of the optimized topology configuration scheme, determine the cooperative association nodes corresponding to each agent;

[0088] S23, based on the progress parameters of each agent and the corresponding collaborative association nodes of each agent, calculate the absolute difference in progress and the progress deviation rate between each pair of agents.

[0089] S24. The schedule absolute difference and schedule deviation rate are standardized using a normalization method to obtain normalized schedule difference data.

[0090] S25, based on the preset progress difference threshold range, classify the normalized progress difference data into levels and determine the training progress level of each agent.

[0091] S26, integrate the training progress levels, collaborative association nodes, and normalized progress difference data of each agent to generate collaborative training progress status.

[0092] Specifically, the training iteration count (the cumulative value of the iteration count field in the log), task completion percentage (calculated by dividing the number of completed training sub-tasks by the preset total number of sub-tasks; for example, completing 5 out of 10 sub-tasks results in a 50% completion rate), and model parameter update amount (obtained by summing the absolute values ​​of the differences between the current model weight parameters and the initial training weight parameters) can be extracted from the training progress of each agent as progress parameters. Based on the agent node connection relationships and link transmission weight data recorded in the optimized topology configuration scheme, agents with direct communication links to the target agent and link transmission weights not lower than a preset association threshold (set to 0.3) are selected. Other agent nodes (i.e., retaining only directly connected nodes with a transmission weight ≥ 0.3) are identified as the cooperative association nodes corresponding to the target agent. For each pair of agents with a cooperative relationship, the absolute difference in progress between them is calculated based on three types of progress parameters: the number of training iterations, the percentage of task completion, and the amount of model parameter updates. (For example, if agent A has 100 iterations and agent B has 80 iterations, the absolute difference in the number of iterations is 20. Similarly, the absolute difference in the percentage of task completion and the amount of model parameter updates are calculated, and the arithmetic mean of the three types of absolute differences is taken as the final absolute difference in progress between the pair of agents.) and the progress deviation rate are also calculated. The progress deviation rate is calculated by dividing the absolute difference in progress by the larger of the two progress parameter values ​​and then multiplying by 100%. For example, the progress deviation rate for the above iterations is 20 / 100×100%=20%. Similarly, the arithmetic mean of the deviation rates of the three types of parameters is taken as the final progress deviation rate. The Min-Max normalization method can be used to standardize the absolute difference in progress and the progress deviation rate between all pairs of agents, mapping the original absolute difference in progress and the progress deviation rate to the [0,1] interval to obtain normalized progress difference data. Based on the preset progress difference threshold interval (e.g., preset normalized progress difference data 0-0.2 for progress synchronization level, 0.2-0.5 for slight progress deviation level, etc.), the progress deviation rate is calculated. (0.5-1.0 represents a severe progress deviation level). The average of the normalized progress difference data between each agent and all its collaborative nodes is taken, and the training progress level of the agent is determined based on the threshold range to which the average value belongs. Using the unique identifier of the agent as an index, the training progress level of each agent, the corresponding list of collaborative nodes (including node identifiers and link transmission weights), and the normalized progress difference data between each collaborative node are integrated into structured data (such as JSON format, including fields such as agent ID, training progress level, collaborative nodes, and normalized progress difference) to generate a collaborative training progress status, which can be directly used for the construction of subsequent collaborative training frameworks.

[0093] In one embodiment, a collaborative training framework is constructed by combining load balancing data, collaborative training progress status, and optimized topology configuration scheme, including:

[0094] S31. Combining load balancing data, collaborative training progress status, and optimized topology configuration scheme, determine the construction dimensions of the collaborative training framework;

[0095] S32. Based on the construction dimension, the hierarchical structure of the collaborative training framework is divided. The hierarchical structure includes a communication layer corresponding to the optimized topology configuration scheme, a progress coordination layer corresponding to the collaborative training progress status, and a resource allocation layer corresponding to the load balancing data.

[0096] S33, based on the partitioned hierarchical structure, maps load balancing data, collaborative training progress status and optimized topology configuration scheme to the corresponding resource allocation layer, progress coordination layer and communication layer in the hierarchical structure respectively.

[0097] S34, based on the mapped resource allocation layer, progress coordination layer and communication layer, set the data processing flow, transmission path and data interaction triggering conditions of each level of the hierarchical structure according to the preset hierarchical interaction rules;

[0098] S35 integrates hierarchical structure, data processing flow, transmission path, and data interaction event triggering conditions to form a collaborative training framework.

[0099] For example, by analyzing the supporting role of load balancing data in resource allocation balance, the guarantee role of collaborative training progress status in training synchronization, and the impact of optimized topology configuration on communication efficiency, three core directions—communication adaptation, progress coordination, and resource balancing—can be extracted to determine the construction dimensions of the collaborative training framework. Based on the correspondence between the above construction dimensions and various data, the hierarchical structure of the collaborative training framework is divided. The layer directly related to optimized topology configuration and responsible for handling multi-agent communication link configuration is designated as the communication layer; the layer related to collaborative training progress status and responsible for managing the synchronization of training progress among agents is designated as the progress coordination layer; and the layer related to load balancing data... The layer responsible for scheduling computing resources is designated as the resource allocation layer. Based on this hierarchical structure, data mapping is performed. Parameters such as the single agent load distribution ratio, cluster idle computing resource quantity, and resource scheduling response latency from the load balancing data are stored in the preset data fields of the resource status management module in the resource allocation layer. The agent training progress level, collaborative associated node list, and normalized progress difference data from the collaborative training progress status are mapped to the node progress control unit in the progress coordination layer. Parameters such as the reconstructed node identifier, reconstructed associated link transmission weight, and link connectivity standard from the optimized topology configuration scheme are imported into the topology configuration unit of the communication layer, thus achieving data binding with the corresponding layer. Based on the three-tiered structure after mapping, and referring to preset hierarchical interaction rules (e.g., the communication layer prioritizes synchronizing topology changes to the progress coordination layer, and the progress coordination layer then synchronizes progress deviations to the resource allocation layer), the data processing flow for each layer is set. For example, the resource allocation layer needs to process data according to the process of receiving progress deviation signals from the progress coordination layer → calculating the amount of idle resources in the cluster → adjusting the load allocation ratio of single agents. The data transmission path between each layer is defined as the Unreal Engine's internal data bus. The topology data from the communication layer to the progress coordination layer is transmitted via the bus channel Topo-P, and the progress data from the progress coordination layer to the resource allocation layer is transmitted via the bus channel Prog-R. Data interaction triggering conditions are also set, such as those for the communication layer. When the link connectivity detection result is below 98%, topology adjustment data is transmitted to the progress coordination layer. When the progress deviation rate of the progress coordination layer exceeds 10%, progress warning data is transmitted to the resource allocation layer. The Unreal Engine's framework configuration tool is used to integrate and import the hierarchical structure (including the functional definitions of the three levels and the hierarchical relationship diagram), the data processing flow of each level (such as the signal reception-resource accounting-load adjustment step description of the resource allocation layer), the transmission path (including the bandwidth configuration and transmission protocol of the bus channels Topo-P and Prog-R), and the data interaction triggering conditions (including each trigger threshold and the data transmission instructions after triggering) to generate a framework file containing hierarchical association logic.

[0100] In one embodiment, based on the collaborative training framework, and combining optimized topology configuration schemes, collaborative training progress status, and load balancing data, an integrated optimization is performed to generate a collaborative training scheme, including:

[0101] S41, extract the optimization topology configuration scheme parameters from the communication layer, extract the collaborative training progress status parameters from the progress coordination layer, extract the load balancing data parameters from the resource allocation layer, and integrate the core parameters of the optimization topology configuration scheme, the core parameters of the collaborative training progress status, and the load balancing data parameters to form a real-time input parameter set;

[0102] S42, input the real-time input parameter set into the pre-trained deep learning optimization model to obtain the parameter prediction results of the collaborative training scheme;

[0103] S43, based on the preset hierarchical interaction rules, verify the prediction results of the initial collaborative training scheme parameters, and select collaborative training scheme parameters that meet the communication link connectivity threshold, training progress difference level threshold, and load distribution balance threshold.

[0104] S44 integrates the selected collaborative training scheme parameters to generate a collaborative training scheme.

[0105] Specifically, core parameters for optimizing topology configuration schemes (such as unique identifiers of reconstructed nodes, transmission weight values ​​of each reconstructed associated link, and preset standards for link connectivity) can be extracted from the communication layer of the collaborative training framework; core parameters for collaborative training progress status (such as training progress levels of each agent, normalized progress difference data between agents and collaborative associated nodes, and a list of collaborative associated nodes) can be extracted from the progress coordination layer; and core parameters for load balancing data (such as the current load allocation ratio of a single agent, the amount of real-time idle computing resources in the cluster, and the resource scheduling response delay) can be extracted from the resource allocation layer. A structured data table (fields including parameter type, parameter name, parameter value, and level) can be used to integrate these three types of core parameters, removing duplicate and redundant parameters, resulting in a real-time input parameter set containing 20-30 key parameters. This pre-trained deep learning optimization model is built on the Transformer architecture, and the training data is a historical parameter set (including historical optimized topology parameters and progress status) from multi-agent reinforcement learning training over the past 6 months. The training process involves matching the parameters and load data with the corresponding optimal collaborative training scheme parameters in a dataset. During training, the Adam optimizer and MSE loss function are used for iterative optimization until the loss value is below 0.01. The real-time input parameter set is input into the pre-trained model according to the format required by the model (numerical parameters are normalized to [0,1], and categorical parameters (such as training progress level) are one-hot encoded). The model output layer generates the collaborative training scheme parameter prediction results through fully connected layer mapping, including communication link adjustment suggestions (such as the adjustment value of a certain link transmission weight), training progress synchronization strategy (such as the training acceleration coefficient of lagging agents), and resource allocation adjustment ratio (such as the adjustment range of a certain agent's load ratio). Based on the hierarchical interaction rules preset by the collaborative training framework (such as communication layer parameters need to ensure link connectivity, progress layer parameters need to control progress deviation, and resource layer parameters need to achieve load balancing), specific verification thresholds are set: the communication link connectivity threshold is ≥99% after adjustment, and the training progress difference level threshold is ≤0 after adjustment of normalized progress difference.2 (corresponding to the progress synchronization level), the load distribution balancing threshold is the absolute value of the difference between the adjusted load distribution ratio of each agent and the average load ratio of the cluster ≤ 10%, and each parameter in the prediction result is checked one by one, such as checking whether the predicted link transmission weight meets the connectivity threshold after adjustment, whether the predicted training acceleration coefficient can reduce the normalized progress difference to within the threshold, and whether the predicted load ratio adjustment range meets the balancing threshold requirements. All collaborative training scheme parameters that meet the above three thresholds are selected, and parameters that do not meet the standards are removed (e.g., if the connectivity of a certain link is only 95% after adjustment, the adjustment parameter of that link is removed, and the model is called again to supplement the predicted parameters that meet the standards). The selected compliance parameters are categorized and integrated into three main modules: communication configuration, progress coordination, and resource allocation. The communication configuration module includes adjusted reconstructed node connection relationships, link transmission weight tables, and link connectivity assurance measures. The progress coordination module includes the training synchronization frequency of each agent, retraining strategies for lagging agents, and task allocation suggestions for leading agents. The resource allocation module includes the final load allocation percentage for each agent, priority for scheduling idle cluster resources, and resource scheduling response delay control measures. Organizing these modules into a structured document directly guides collaborative training schemes for multi-agent communication interaction, training synchronization, and resource allocation operations.

[0106] In one embodiment, S51, the hierarchical structure, data processing flow, transmission path, and data interaction event triggering conditions are integrated to form a collaborative training framework, which is implemented mathematically as follows:

[0107] F = w C ·Φ(C,T) C E C )+w P ·Φ(P,T) P E P )+w R ·Φ(R,T) R E R )+λ·Ψ(L,D C-P D P-R )

[0108] Where F is the mathematical representation of the collaborative training framework, w C w P and w R The weight coefficients for the communication layer, schedule coordination layer, and resource allocation layer are respectively, satisfying w C +w P +w R =1, C is the optimal topology configuration parameter in the communication layer, P is the collaborative training progress status parameter in the progress coordination layer, R is the load balancing data parameter in the resource allocation layer, and T C T P and T RThese are the transmission path parameter sets for the communication layer, the schedule coordination layer, and the resource allocation layer, respectively. C E P and E R These are the data interaction trigger condition parameter sets for the communication layer, progress coordination layer, and resource allocation layer, respectively. Φ(·) is the intra-layer integration operator, used to couple and integrate the structural parameters, transmission path parameters, and trigger condition parameters of a single layer to satisfy... This represents element-level collaborative operations after parameter dimension matching, where λ is the constraint coefficient of the hierarchical interaction rule, with a value range of (0,1], Ψ(·) is the cross-level interaction integration operator, L is the preset hierarchical interaction rule parameter set, and D... C-P and D P-R These are cross-level data processing parameter sets for the communication layer and the schedule coordination layer, and for the schedule coordination layer and the resource allocation layer, respectively.

[0109] For example, F is the mathematical representation of the framework (such as a 1×N vector, reflecting core indicators such as hierarchical adaptability); w C w P and w R Weights for the communication layer, schedule coordination layer, and resource allocation layer (satisfying w) C +w P +w R =1, can be set according to the scenario, such as w C =0.4, w P =0.3, w R =0.3); C, P, and R are the core parameters of each layer (e.g., C is the number of reconstructed nodes and the link weight vector, P is the agent progress level vector, and R is the load percentage and idle resource vector); T C T P and T R It is the transmission path matrix of each layer (recording bandwidth and delay), E C E P and E R These are the trigger condition vectors for each layer (recording thresholds such as connectivity rate and progress difference). Intra-layer integration operators. First X, T X E X Adjust to the same dimension, then multiply the corresponding elements (e.g., communication layer C and T). C E C (Multiply element-wise after matching dimensions); λ is the cross-layer constraint coefficient (value 0 < λ ≤ 1, such as 0.6), Ψ(L, D) C-P D P-R ) is a cross-layer integration operator (L is the interaction rule vector, D C-P D P-RFor the cross-layer process parameter vector, adjust the dimensions, sum them, and then multiply by λ. (According to the formula...)

[0110] F = w C ·Φ(C,T) C E C )+w P ·Φ(P,T) P E P )+w R ·Φ(R,T) R E R )+λ·Ψ(L,D C-P D P-R )

[0111] The calculation involves adding the corresponding elements of each part of the result to obtain F. If the indicators in F (such as fit and fluency) meet the standards, a collaborative training framework that meets the requirements is formed.

[0112] In one embodiment, load balancing data is extracted from the computing resource pool associated with Unreal Engine based on the computing capabilities of a single agent and the overall computing load status data of the cluster, including:

[0113] S61 establishes a corresponding data mapping relationship between the computing resource pool associated with Unreal Engine, based on the computing capabilities of a single agent and the overall computing load status of the cluster.

[0114] S62, based on the corresponding data mapping relationship, select the raw load data related to the computing capabilities of a single agent and the overall computing load status of the cluster from the computing resource pool associated with Unreal Engine;

[0115] S63 quantifies and calculates the raw load data to obtain the load allocation ratio of a single agent, the amount of idle computing resources in the cluster, and the resource scheduling response delay data.

[0116] S64 integrates data on the load distribution ratio of single agents, the amount of idle computing resources in the cluster, and the resource scheduling response latency to form load balancing data.

[0117] Specifically, based on the computing power of a single agent and the overall computing load status of the cluster, the corresponding dimensions of the data between the agent and the Unreal Engine's associated computing resource pool (a resource set that provides computing power support for multi-agent training, including multiple computing nodes and resource monitoring modules) can be determined. For example, the number of floating-point operations per second of a single agent corresponds to the computing power specification of a single node in the resource pool, and the CPU utilization of the cluster corresponds to the overall CPU resource utilization rate of the resource pool. This can be achieved by establishing a key-value pair mapping table (e.g., single agent computing power - resource pool node computing power specification, cluster CPU utilization - overall CPU utilization rate of the resource pool) to form a corresponding data mapping relationship between the agent and the computing resource pool. Based on the established corresponding data mapping relationship, the raw load data can be filtered from the real-time monitoring database of the computing resource pool through the Unreal Engine's resource pool data interface (e.g., the memory usage allocated by the node to the agent) related to the single agent's computing power, and the total CPU resource amount and total GPU resource amount of the resource pool related to the overall computing load status of the cluster. Data such as allocated resources are used as raw load data for subsequent processing. The selected raw load data is quantitatively calculated: the load allocation ratio of a single agent is calculated by "the amount of CPU / GPU resources currently occupied by a single agent ÷ the total amount of CPU / GPU resources in the cluster × 100%", the amount of idle computing resources in the cluster is calculated by "the total amount of CPU / GPU / memory resources in the resource pool - the corresponding amount of resources allocated to all agents" (the units are the number of cores, GB of video memory, and GB of memory, respectively), and the resource scheduling response latency is calculated by recording the timestamps of the agent sending the resource scheduling request and the computing resource pool completing the resource allocation, and taking the difference between the two (the unit is milliseconds). The calculated load allocation ratio of a single agent (recorded according to the unique identifier of the agent), the amount of idle computing resources in the cluster (recorded according to CPU, GPU, and memory categories), and the resource scheduling response latency (recorded according to a single scheduling request) are integrated into structured data (such as JSON format, containing fields such as agent ID-load ratio, number of idle CPU cores in the cluster, number of idle GPU video memory in the cluster, and scheduling response latency) to form complete load balancing data.

[0118] The aforementioned Unreal Engine-driven multi-agent reinforcement learning training method acquires core attribute data such as real-time communication bandwidth, initial communication topology configuration, training progress, individual agent computing power, and overall cluster computing load status. It then performs coupled analysis on real-time communication bandwidth and initial communication topology configuration, dynamically adjusting the topology to obtain an optimized configuration scheme. This solves the problem that traditional fixed communication topologies cannot adapt to dynamic bandwidth fluctuations, leading to data transmission delays or congestion. Combining this optimized topology configuration scheme with the training progress, it generates a collaborative training progress status by quantifying the progress differences between agents, accurately controlling progress deviations and improving the poor coordination problem caused by traditional methods that rely solely on periodic data synchronization. Based on individual agent computing power and the overall cluster computing load status, it extracts load balancing data from the Unreal Engine's associated computing resource pool. This data is then combined to construct a collaborative training framework and integrate optimizations to generate a collaborative training scheme. This achieves dynamic allocation of computing resources, avoiding the load imbalance problem caused by traditional static resource allocation failing to respond to changes in cluster load. This effectively improves the efficiency of multi-agent reinforcement learning training, balances resource load, and enhances collaborative synchronization.

[0119] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0120] Based on the same inventive concept, this application also provides an Unreal Engine-driven multi-agent reinforcement learning training system for implementing the Unreal Engine-driven multi-agent reinforcement learning training method described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more Unreal Engine-driven multi-agent reinforcement learning training system embodiments provided below can be found in the limitations of the Unreal Engine-driven multi-agent reinforcement learning training method described above, and will not be repeated here.

[0121] In one exemplary embodiment, such as Figure 2 As shown, a multi-agent reinforcement learning training system driven by Unreal Engine is provided, including:

[0122] The multi-agent data acquisition module 101 acquires the core attribute data of the multi-agent, including real-time communication bandwidth, initial communication topology configuration, training phase progress, single agent computing power, and overall cluster computing load status.

[0123] The communication topology optimization module 102 performs a coupling analysis on the real-time communication bandwidth and the initial communication topology configuration, obtains the coupling analysis results, and adjusts the communication topology structure between multiple agents based on the coupling analysis results to obtain an optimized topology configuration scheme.

[0124] The collaborative training optimization module 103 combines the optimized topology configuration scheme with the training phase progress, and generates the collaborative training progress status by quantifying the progress differences between each agent.

[0125] The load data acquisition module 104 extracts load balancing data from the computing resource pool associated with Unreal Engine based on the computing power of a single agent and the overall computing load status data of the cluster. The load balancing data includes the load allocation ratio of a single agent, the amount of idle computing resources in the cluster, and the resource scheduling response latency.

[0126] Training framework construction module 105 combines load balancing data, collaborative training progress status, and optimized topology configuration scheme to construct a collaborative training framework;

[0127] The training scheme generation module 106, based on the collaborative training framework, integrates and optimizes the topology configuration scheme, collaborative training progress status, and load balancing data to generate a collaborative training scheme. The collaborative training scheme is used to guide the communication, training, and resource allocation of multiple agents.

[0128] In one embodiment, the communication topology optimization module 102 is further configured to:

[0129] Extract the dynamic characteristic parameters of real-time communication bandwidth and calculate the bandwidth supply capacity based on the dynamic characteristic parameters. The dynamic characteristic parameters include bandwidth fluctuation amplitude, peak bandwidth duration and minimum available bandwidth.

[0130] The topology parameters of the initial communication topology configuration are parsed, and the topology transmission requirements are calculated based on the topology parameters. The topology parameters include the connection method of the agent nodes, the link transmission weight, and the node hierarchy.

[0131] The matching degree between bandwidth supply and topology transmission demand is calculated to obtain the coupling analysis results;

[0132] Based on the coupling analysis results, nodes and associated links with matching degrees below a preset threshold in the initial communication topology are identified as nodes to be optimized and related links to be optimized.

[0133] The nodes to be optimized and their associated links to be optimized are reconstructed and optimized to obtain reconstructed nodes and reconstructed associated links. Based on the computing power of the reconstructed nodes, the transmission weight allocation ratio of the reconstructed associated links is adjusted to obtain the reconstructed communication topology.

[0134] The optimization topology is formed by replacing the nodes and associated links to be optimized in the initial communication topology with a reconstructed communication topology.

[0135] Link connectivity detection is performed on the optimized topology to obtain the link connectivity detection results. Based on the link connectivity detection results, an optimized topology configuration scheme is generated according to the optimized topology that meets the preset connectivity standards.

[0136] In one embodiment, the collaborative training optimization module 103 is further configured to:

[0137] Analyze the training progress of each agent to obtain progress parameters, including the number of training iterations, the percentage of task completion, and the amount of model parameter updates.

[0138] Based on the topology structure of the optimized topology configuration scheme, the cooperative association nodes corresponding to each agent are determined;

[0139] Based on the progress parameters of each agent and the corresponding collaborative nodes of each agent, the absolute difference in progress and the progress deviation rate between each pair of agents are calculated.

[0140] The absolute schedule difference and schedule deviation rate were standardized using a normalization method to obtain normalized schedule difference data.

[0141] Based on the preset progress difference threshold range, the normalized progress difference data is classified into levels to determine the training progress level of each agent.

[0142] By integrating the training progress levels, collaborative association nodes, and normalized progress difference data of each agent, a collaborative training progress status is generated.

[0143] In one embodiment, the training framework building module 105 is further configured to:

[0144] By combining load balancing data, collaborative training progress status, and optimized topology configuration schemes, the construction dimensions of the collaborative training framework are determined.

[0145] Based on the construction dimension, the hierarchical structure of the collaborative training framework is divided. The hierarchical structure includes a communication layer corresponding to the optimized topology configuration scheme, a progress coordination layer corresponding to the collaborative training progress status, and a resource allocation layer corresponding to the load balancing data.

[0146] Based on the hierarchical structure, load balancing data, collaborative training progress status, and optimized topology configuration scheme are mapped to the corresponding resource allocation layer, progress coordination layer, and communication layer in the hierarchical structure, respectively.

[0147] Based on the mapped resource allocation layer, progress coordination layer and communication layer, the data processing flow, transmission path and data interaction triggering conditions of each layer of the hierarchical structure are set according to the preset hierarchical interaction rules.

[0148] By integrating hierarchical structure, data processing flow, transmission path, and data interaction event triggering conditions, a collaborative training framework is formed.

[0149] In one embodiment, the training scheme generation module 106 is further configured to:

[0150] Optimized topology configuration parameters are extracted from the communication layer, collaborative training progress status parameters are extracted from the progress coordination layer, and load balancing data parameters are extracted from the resource allocation layer. The core parameters of optimized topology configuration, collaborative training progress status, and load balancing data parameters are then integrated to form a real-time input parameter set.

[0151] The real-time input parameter set is input into the pre-trained deep learning optimization model to obtain the parameter prediction results of the collaborative training scheme.

[0152] Based on the preset hierarchical interaction rules, the prediction results of the initial collaborative training scheme parameters are verified, and collaborative training scheme parameters that meet the communication link connectivity threshold, training progress difference level threshold, and load distribution balance threshold are selected.

[0153] Integrate the parameters of the selected collaborative training schemes to generate a collaborative training scheme.

[0154] In one embodiment, the training framework construction module 105 is further configured to integrate the hierarchical structure, data processing flow, transmission path, and data interaction event triggering conditions through the following mathematical implementation to form a collaborative training framework:

[0155] F = w C ·Φ(C,T) C E C )+w P ·Φ(P,T) P E P )+w R ·Φ(R,T) R E R )+λ·Ψ(L,D C-P D P-R )

[0156] Where F is the mathematical representation of the collaborative training framework, w C w P and wR The weight coefficients for the communication layer, schedule coordination layer, and resource allocation layer are respectively, satisfying w C +w P +w R =1, C is the optimal topology configuration parameter in the communication layer, P is the collaborative training progress status parameter in the progress coordination layer, R is the load balancing data parameter in the resource allocation layer, and T C T P and T R These are the transmission path parameter sets for the communication layer, the schedule coordination layer, and the resource allocation layer, respectively. C E P and E R These are the data interaction trigger condition parameter sets for the communication layer, progress coordination layer, and resource allocation layer, respectively. Φ(·) is the intra-layer integration operator, used to couple and integrate the structural parameters, transmission path parameters, and trigger condition parameters of a single layer to satisfy... This represents element-level collaborative operations after parameter dimension matching, where λ is the constraint coefficient of the hierarchical interaction rule, with a value range of (0,1], Ψ(·) is the cross-level interaction integration operator, L is the preset hierarchical interaction rule parameter set, and D... C-P and D P-R These are cross-level data processing parameter sets for the communication layer and the schedule coordination layer, and for the schedule coordination layer and the resource allocation layer, respectively.

[0157] In one embodiment, the load data acquisition module 104 is further configured to:

[0158] Based on the computing capabilities of a single agent and the overall computing load status of the cluster, a corresponding data mapping relationship is established between the computing resource pool associated with Unreal Engine;

[0159] Based on the corresponding data mapping relationship, raw load data related to the computing capabilities of a single agent and the overall computing load status of the cluster are filtered out from the computing resource pool associated with Unreal Engine;

[0160] The raw load data is quantified and calculated to obtain the load distribution ratio of a single agent, the amount of idle computing resources in the cluster, and the resource scheduling response delay data.

[0161] The load balancing data is formed by integrating the load distribution ratio of single agents, the amount of idle computing resources in the cluster, and the resource scheduling response delay data.

[0162] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the Unreal Engine-driven multi-agent reinforcement learning training method as described above.

[0163] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0164] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0165] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A multi-agent reinforcement learning training method driven by Unreal Engine, characterized in that, The method includes: Acquire the core attribute data of the multi-agent system, including real-time communication bandwidth, initial communication topology configuration, training phase progress, single agent computing power, and overall cluster computing load status. A coupling analysis is performed on the real-time communication bandwidth and the initial communication topology configuration to obtain the coupling analysis results. Based on the coupling analysis results, the communication topology structure between multiple agents is adjusted to obtain an optimized topology configuration scheme. By combining the optimized topology configuration scheme with the training phase progress, a collaborative training progress status is generated by quantifying the progress differences among the agents. Based on the computing power of the single agent and the overall computing load status data of the cluster, load balancing data is extracted from the computing resource pool associated with Unreal Engine. The load balancing data includes the load allocation ratio of the single agent, the amount of idle computing resources in the cluster, and the resource scheduling response latency. By combining the load balancing data, collaborative training progress status, and the optimized topology configuration scheme, a collaborative training framework is constructed. Based on the aforementioned collaborative training framework, and by integrating and optimizing the optimized topology configuration scheme, the collaborative training progress status, and the load balancing data, a collaborative training scheme is generated. This collaborative training scheme is used to guide the communication, training, and resource allocation of multiple agents.

2. The method according to claim 1, characterized in that, The process of performing a coupling analysis on the real-time communication bandwidth and the initial communication topology configuration to obtain the coupling analysis results, and adjusting the communication topology among multiple agents based on the coupling analysis results to obtain an optimized topology configuration scheme, includes: Extract the dynamic characteristic parameters of the real-time communication bandwidth and calculate the bandwidth supply capacity based on the dynamic characteristic parameters. The dynamic characteristic parameters include bandwidth fluctuation amplitude, peak bandwidth duration and minimum available bandwidth. The topology parameters of the initial communication topology configuration are parsed, and the topology transmission requirements are calculated based on the topology parameters. The topology parameters include the connection method of the agent nodes, the link transmission weight, and the node hierarchy relationship. Calculate the matching degree between the bandwidth supply and the topology transmission requirements to obtain the coupling analysis results; Based on the coupling analysis results, identify nodes and associated links in the initial communication topology that have a matching degree lower than a preset threshold that need to be optimized. The nodes to be optimized and their associated links to be optimized are reconstructed and optimized to obtain reconstructed nodes and reconstructed associated links. Based on the computing power of the reconstructed nodes, the transmission weight allocation ratio of the reconstructed associated links is adjusted to obtain the reconstructed communication topology. The reconstructed communication topology is used to replace the nodes and associated links to be optimized in the initial communication topology to form an optimized topology. Link connectivity detection is performed on the optimized topology to obtain the link connectivity detection results. Based on the link connectivity detection results, the optimized topology configuration scheme is generated according to the optimized topology whose connectivity meets the preset standard.

3. The method according to claim 1, characterized in that, The step of combining the optimized topology configuration scheme with the training phase progress, and generating a collaborative training progress state by quantifying the progress differences among the agents, includes: The training progress of each agent is analyzed to obtain progress parameters, which include the number of training iterations, the percentage of task completion, and the amount of model parameter updates. Based on the topology structure of the optimized topology configuration scheme, the cooperative association nodes corresponding to each agent are determined; Based on the progress parameters of each agent and the corresponding collaborative nodes of each agent, the absolute difference in progress and the progress deviation rate between each pair of agents are calculated. The absolute difference in schedule and the schedule deviation rate are standardized using a normalization method to obtain normalized schedule difference data. Based on a preset progress difference threshold range, the normalized progress difference data is classified into levels to determine the training progress level of each agent. The training progress levels of each agent, the collaborative association nodes, and the normalized progress difference data are integrated to generate the collaborative training progress status.

4. The method according to claim 3, characterized in that, The process of constructing a collaborative training framework by combining the load balancing data, collaborative training progress status, and the optimized topology configuration scheme includes: By combining the load balancing data, the collaborative training progress status, and the optimized topology configuration scheme, the construction dimensions of the collaborative training framework are determined. Based on the construction dimension, the hierarchical structure of the collaborative training framework is divided, and the hierarchical structure includes a communication layer corresponding to the optimized topology configuration scheme, a progress coordination layer corresponding to the collaborative training progress status, and a resource allocation layer corresponding to the load balancing data. Based on the hierarchical structure, the load balancing data, the collaborative training progress status, and the optimized topology configuration scheme are respectively mapped to the corresponding resource allocation layer, progress coordination layer, and communication layer in the hierarchical structure. Based on the mapped resource allocation layer, progress coordination layer and communication layer, the data processing flow, transmission path and data interaction triggering conditions of each level of the hierarchical structure are set according to the preset hierarchical interaction rules. The hierarchical structure, data processing flow, transmission path, and data interaction event triggering conditions are integrated to form the collaborative training framework.

5. The method according to claim 4, characterized in that, The process of integrating and optimizing the collaborative training framework, the optimized topology configuration scheme, the collaborative training progress status, and the load balancing data to generate a collaborative training scheme includes: Optimized topology configuration parameters are extracted from the communication layer, collaborative training progress status parameters are extracted from the progress coordination layer, and load balancing data parameters are extracted from the resource allocation layer. The core parameters of the optimized topology configuration scheme, the core parameters of the collaborative training progress status, and the load balancing data parameters are then integrated to form a real-time input parameter set. The real-time input parameter set is input into the pre-trained deep learning optimization model to obtain the parameter prediction results of the collaborative training scheme. Based on the preset hierarchical interaction rules, the prediction results of the initial collaborative training scheme parameters are verified, and collaborative training scheme parameters that meet the communication link connectivity threshold, training progress difference level threshold, and load distribution balance threshold are selected. The selected collaborative training scheme parameters are integrated to generate the collaborative training scheme.

6. The method according to claim 4, characterized in that, The integration of the hierarchical structure, the data processing flow, the transmission path, and the data interaction event triggering conditions forms the collaborative training framework, which is implemented mathematically as follows: F=w C ·Φ(C,T C ,E C )+w P ·Φ(P,T P ,E P )+w R ·Φ(R,T R ,E R )+λ·Ψ(L,D C-P ,D P-R ) Where F is the mathematical representation of the collaborative training framework, w C w P and w R The weight coefficients for the communication layer, schedule coordination layer, and resource allocation layer are respectively, satisfying w C +w P +w R =1, C is the optimal topology configuration parameter in the communication layer, P is the collaborative training progress status parameter in the progress coordination layer, R is the load balancing data parameter in the resource allocation layer, and T C T P and T R These are the transmission path parameter sets for the communication layer, the schedule coordination layer, and the resource allocation layer, respectively. C E P and E R These are the data interaction trigger condition parameter sets for the communication layer, progress coordination layer, and resource allocation layer, respectively. Φ(·) is the intra-layer integration operator, used to couple and integrate the structural parameters, transmission path parameters, and trigger condition parameters of a single layer to satisfy... This represents element-level collaborative operations after parameter dimension matching, where λ is the constraint coefficient of the hierarchical interaction rule, with a value range of (0,1], Ψ(·) is the cross-level interaction integration operator, L is the preset hierarchical interaction rule parameter set, and D... C-P and D P-R These are cross-level data processing parameter sets for the communication layer and the schedule coordination layer, and for the schedule coordination layer and the resource allocation layer, respectively.

7. The method according to claim 1, characterized in that, The step of extracting load balancing data from the computing resource pool associated with Unreal Engine, based on the computing capabilities of the single agent and the overall computing load status data of the cluster, includes: Based on the computing capabilities of the single agent and the overall computing load status of the cluster, a corresponding data mapping relationship is established between the computing resource pool associated with Unreal Engine; Based on the corresponding data mapping relationship, raw load data related to the computing capabilities of a single agent and the overall computing load status of the cluster are filtered out from the computing resource pool associated with the Unreal Engine. The original load data is quantified and calculated to obtain the load allocation ratio of the single agent, the amount of idle computing resources in the cluster, and the resource scheduling response delay data. The load balancing data is formed by integrating the single agent load distribution ratio, the amount of idle computing resources in the cluster, and the resource scheduling response delay data.

8. A multi-agent reinforcement learning training system driven by Unreal Engine, characterized in that, The system includes: The multi-agent data acquisition module acquires the core attribute data of the multi-agent, including real-time communication bandwidth, initial communication topology configuration, training phase progress, single agent computing power, and overall cluster computing load status. The communication topology optimization module performs a coupling analysis on the real-time communication bandwidth and the initial communication topology configuration to obtain the coupling analysis results, and adjusts the communication topology structure between multiple agents based on the coupling analysis results to obtain an optimized topology configuration scheme. The collaborative training optimization module, by combining the optimized topology configuration scheme with the training phase progress, generates a collaborative training progress status by quantifying the progress differences between each agent. The load data acquisition module extracts load balancing data from the computing resource pool associated with Unreal Engine based on the computing capabilities of the single agent and the overall computing load status data of the cluster. The load balancing data includes the load allocation ratio of the single agent, the amount of idle computing resources in the cluster, and the resource scheduling response latency. The training framework construction module combines the load balancing data, collaborative training progress status, and the optimized topology configuration scheme to construct a collaborative training framework. The training scheme generation module, based on the collaborative training framework, integrates and optimizes the optimized topology configuration scheme, the collaborative training progress status, and the load balancing data to generate a collaborative training scheme. The collaborative training scheme is used to guide the communication, training, and resource allocation of multiple agents.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images