Deep reinforcement learning method and system based on multiple agents

Abstract

The invention relates to the technical field of reinforcement learning, in particular to a deep reinforcement learning method and system based on multiple agents. The method comprises the following steps of performing operation log extraction on multiple agents and constructing a multi-agent interaction graph to obtain the multi-agent interaction graph; carrying out resource release limited interval calculation among different load nodes on the multi-agent interaction map to obtain a node resource release limited interval; according to the node resource release limited interval, making a multi-agent resource competition strategy and carrying out multi-agent hierarchical convergence behavior coupling to obtain hierarchical convergence behavior coupling data; and constructing a multi-agent resource competition balance model according to the layered convergence behavior coupling data to obtain the multi-agent resource competition balance model, and sending the multi-agent resource competition balance model to a cloud platform to execute multi-agent deep reinforcement learning. According to the method, multi-agent collaboration is more perfect through optimization processing of multi-agent reinforcement learning.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a multi-agent based deep reinforcement learning method and system. Background Art

[0002] Agents interact in a shared environment and learn how to optimize their own behavior strategies to achieve the goal of the overall system through continuous trial and feedback. Deep reinforcement learning uses deep neural networks to process a large amount of state and action space, enhancing the perception and decision-making ability of agents in complex environments. In a multi-agent system, each agent must not only consider the impact of its own actions on the environment, but also the behavior of other agents and their impact. In order to achieve collaboration, agents need to share information and learn how to cooperate with each other; in competitive scenarios, agents need to learn adversarial strategies. MADRL is widely used in fields such as robot coordination control, autonomous driving, smart grid optimization, and financial market transactions. Through simulation and practical applications, it has demonstrated its strong ability and potential in solving multi-agent problems. Through multi-agent deep reinforcement learning, the adaptability and intelligence of the system can be effectively improved, laying the foundation for the realization of more intelligent and automated complex systems. However, a traditional multi-agent deep reinforcement learning method has the problem that it is difficult to balance resource competition among multiple agents and that multiple agents are prone to fall into an unstable state. Summary of the invention

[0003] Based on this, it is necessary to provide a multi-agent deep reinforcement learning method and system to solve at least one of the above technical problems.

[0004] To achieve the above object, a multi-agent deep reinforcement learning method is provided, the method comprising the following steps: Step S1: extracting the operation log of the multi-agent to obtain the multi-agent operation log data; performing multi-agent interaction structure analysis on the multi-agent operation log data to obtain the multi-agent interaction structure data; constructing a multi-agent interaction graph according to the multi-agent interaction structure data to obtain the multi-agent interaction graph; Step S2: perform multi-agent load node labeling on the multi-agent interaction graph to obtain multi-agent load node labeling data; perform resource release amount calculation between different nodes on the multi-agent load node labeling data to obtain load node resource release amount data; perform resource release limited interval calculation between different load nodes on the load node resource release amount data to obtain node resource release limited interval; formulate a multi-agent resource competition strategy according to the node resource release limited interval to obtain a multi-agent resource competition strategy; Step S3: Performing hierarchical deep learning processing according to the multi-agent resource competition strategy to obtain resource competition hierarchical deep learning data; performing multi-agent hierarchical convergence behavior coupling based on the resource competition hierarchical deep learning data to obtain hierarchical convergence behavior coupling data; Step S4: Construct a multi-agent resource competition balance model based on the multi-agent resource competition strategy and the hierarchical convergence behavior coupling data to obtain the multi-agent resource competition balance model, and send the multi-agent resource competition balance model to the cloud platform to perform multi-agent deep reinforcement learning.

[0005] By analyzing the operation log data of multiple agents, the present invention can deeply understand the interaction structure and mode between each agent. This helps to identify key interaction nodes, frequently occurring interaction events, and the time and space distribution rules of interaction. The interaction map constructed based on the interaction structure data can help find bottlenecks and optimization points in the system. By identifying and optimizing the key interactions between multiple agents, the overall efficiency and response speed of the system can be improved. Analyzing the interaction map helps to identify potential system failure points or unstable factors. By better managing and regulating the interactions between multiple agents, the possibility of system crash or failure can be reduced, and the stability and reliability of the system can be improved. According to the load node annotation data and the resource release data, the resource usage between different nodes can be accurately calculated. This helps to optimize the allocation and utilization of resources, avoid waste and excessive consumption of resources, and improve the overall resource utilization efficiency of the system. Through the calculation of the limited interval of resource release and the formulation of the multi-agent resource competition strategy, the resource competition between different nodes can be effectively managed and mediated. This helps to reduce the conflict and efficiency loss caused by resource competition and maintain the stability and predictability of system operation. According to the limited interval of node resource release, the resource competition strategy can be formulated to make the system more flexible and adaptable to different operating environments and workloads. This helps the system to quickly adjust and respond to changes and challenges and maintain efficient operation. Therefore, the present invention provides a multi-agent based deep reinforcement learning method that optimizes a traditional multi-agent based deep reinforcement learning method, solves the problem that it is difficult to balance resource competition among multiple agents and that multiple agents are prone to falling into an unstable state in a traditional multi-agent based deep reinforcement learning method, improves the ability to balance resource competition among multiple agents, and reduces the problem that multiple agents are prone to falling into an unstable state.

[0006] Preferably, step S1 comprises the following steps: Step S11: extracting the operation log of the multi-agent to obtain the multi-agent operation log data; Step S12: marking the task processing types between different agents on the multi-agent operation log data to obtain task processing type marking data; Step S13: performing multi-agent interaction structure analysis on the task processing type tag data according to the multi-agent operation log data to obtain multi-agent interaction structure data; Step S14: performing multi-scale characteristic analysis on the multi-agent interaction structure data to obtain multi-scale characteristic data of the interaction structure; Step S15: construct a multi-agent interaction graph based on the multi-agent interaction structure data and the multi-scale characteristic data of the interaction structure to obtain a multi-agent interaction graph.

[0007] The present invention obtains detailed multi-agent system operation data by extracting operation logs. These data include information such as activity records, operation behaviors, state changes, etc. of each agent, providing comprehensive basic data for the overall operation of the system. The operation log data is marked with task processing types, which can accurately identify and distinguish the types of tasks performed by different agents in the system. This helps to understand the impact of different tasks on system resources and interaction modes in a targeted manner during subsequent analysis and optimization. Based on the operation log and task processing type marking data, the multi-agent interaction structure is analyzed. This step can reveal the relationship network, information flow path and common interaction mode between agents. By deeply analyzing the interaction structure, potential optimization points and system bottlenecks can be found. Based on the interaction structure data, multi-scale characteristics are analyzed. This includes multiple levels from micro to macro, such as multi-dimensional characteristic evaluations such as time scale and space scale. This comprehensive analysis can help identify the dynamic characteristics and laws of agent interaction at different scales. A multi-agent interaction map is constructed based on the interaction structure data and multi-scale characteristic data. This map not only shows the relationship and interaction mode between agents, but also reflects their interaction characteristics at different time and space scales. By establishing the map, we can form an intuitive grasp and analysis basis for the overall operation of the system.

[0008] Preferably, step S2 comprises the following steps: Step S21: annotating the multi-agent load nodes of the multi-agent interaction graph according to the multi-agent operation log data to obtain the multi-agent load node annotated data; Step S22: sorting the multi-agent load node labeling data according to resource scarcity among different load nodes to obtain load node resource scarcity sorting data; Step S23: Calculate the resource release amount between different nodes of the multi-agent load node annotation data according to the load node resource shortage ranking data and the multi-agent operation log data to obtain the load node resource release amount data; Step S24: Calculate the resource release limited intervals between different load nodes on the load node resource release amount data to obtain the node resource release limited intervals; Step S25: evaluating the performance difference between different load nodes according to the load node resource release amount data and the node resource release limited interval, and obtaining node performance difference evaluation data; Step S26: Formulate a multi-agent resource competition strategy based on the node performance difference evaluation data and the limited interval of node resource release to obtain a multi-agent resource competition strategy.

[0009] The present invention marks the load nodes in the interaction map through the multi-agent operation log data. This step can accurately identify the key nodes in the system, that is, those nodes that undertake important tasks or resource loads in the operation of the system. According to the load node marking data, the resource scarcity between different load nodes is sorted. This sorting is based on the analysis of the system's demand for resources and supply capacity, and can help determine which nodes are more critical and tense in resource allocation. Combined with the resource shortage sorting data and the operation log, the amount of resources released by each load node is calculated. This step reveals the resource utilization efficiency of each node in the system and their contribution to the release of overall system resources. Based on the load node resource release, the limited interval of resource release between different load nodes is calculated. This calculation takes into account the time and space restrictions of node resource release, helping to optimize the timing and spatial performance of resource allocation. Based on the resource release amount and limited interval data, the performance difference between different load nodes is evaluated. This includes the performance differences of each node in the system in resource competition and allocation, thereby providing specific data support and basis for optimizing system performance. According to the performance difference evaluation data and the limited interval of node resource release, a multi-agent resource competition strategy is formulated. These policies can cover aspects such as resource priority allocation, dynamic adjustment, and load balancing to ensure stable operation and efficient performance of the system under different workloads and competition conditions.

[0010] Preferably, step S24 comprises the following steps: Step S241: performing resource release amount fluctuation analysis between different nodes according to the load node resource release amount data to obtain node resource release amount fluctuation data; Step S242: calculating the resource release fluctuation extreme value difference between nodes with different loads on the node resource release amount fluctuation data to obtain the resource release fluctuation extreme value difference; Step S243: performing local similar fluctuation structure analysis between nodes with different loads on the node resource release amount fluctuation data according to the resource release fluctuation extreme value difference to obtain local similar fluctuation structure data; Step S244: performing nonlinear iterative numerical approximation on the node resource release amount fluctuation data based on the local similar fluctuation structure data to obtain release amount iterative numerical approximation data; Step S245: performing local convex optimization processing on the local similar fluctuation structure data according to the release amount iterative numerical approximation data to obtain local fluctuation convex optimization data; Step S246: Calculate the resource release finite interval between nodes with different loads based on the release amount iterative numerical approximation data and the local fluctuation convex optimization data for the node resource release amount fluctuation data to obtain the node resource release finite interval.

[0011] The present invention can reveal the dynamic changes and fluctuations in resource utilization of different nodes by analyzing the fluctuations in resource release of load nodes. This helps to identify the changing patterns of resource requirements of nodes in different time periods, and then adjust the resource allocation strategy to cope with the changing needs. Calculating the extreme value difference of resource release fluctuations can quantify the volatility differences of resource release between different load nodes. This analysis can help determine which nodes are the most unstable in resource release and may require additional resource management and adjustment. According to the extreme value difference of resource release fluctuations, local similar fluctuation structure analysis can be performed to find similarity patterns or features between nodes in resource release. This helps to identify node groups with similar resource release patterns, so as to adopt more accurate resource allocation strategies. Using a nonlinear iterative numerical approximation method based on local similar fluctuation structure, the changing trend and pattern of node resource release can be more accurately predicted. This method can effectively process complex resource release data and provide more accurate numerical prediction results. On the basis of iterative numerical approximation of release amount, convex optimization processing is performed on the local similar fluctuation structure. This step can further optimize the data model to make it more consistent with the actual resource release situation and improve the accuracy and reliability of the prediction. Combining the iterative numerical approximation data of the release amount and the local fluctuation convexity optimization data, the limited intervals of resource release between nodes with different loads are calculated. These interval data can help determine the resource release restrictions and possible optimal release timing of each node, thereby optimizing resource utilization and management strategies.

[0012] Preferably, step S26 comprises the following steps: Step S261: Calculate the processing efficiency of the same task volume between different nodes based on the node performance difference evaluation data to obtain node processing efficiency difference data; Step S262: Establish resource allocation priorities among different nodes according to the node processing efficiency difference data and the node resource release limited interval, and obtain node resource allocation priority data; Step S263: Designing a node interactive polling mechanism according to the node resource allocation priority data and the node resource release limited interval to obtain a node interactive polling mechanism; Step S264: simulating the operation scenario of the node interactive polling mechanism to obtain polling operation scenario simulation data; extracting network congestion features according to the polling operation scenario simulation data to obtain polling scenario network congestion data; Step S265: performing control optimization processing on the node interaction polling mechanism according to the polling scenario network congestion data to obtain a node interaction polling optimization mechanism; Step S266: Formulate a multi-agent resource competition strategy based on the node interaction polling optimization mechanism and the node resource allocation priority data to obtain a multi-agent resource competition strategy.

[0013] The present invention can clarify the efficiency difference of different nodes in processing tasks by calculating the processing efficiency difference of the same task amount based on the node performance difference evaluation data. This data helps to identify nodes with higher and lower performance, and provides a basis for resource allocation strategy. Combined with the node processing efficiency difference data and the limited interval of resource release, the resource allocation priority between different nodes is determined. These priority data can ensure that more resources are allocated to nodes with higher efficiency or better performance within the limited interval of resource release, thereby improving the overall efficiency and performance of the system. Based on the node resource allocation priority data and the limited interval of resource release, an interactive polling mechanism between nodes is designed. This mechanism can ensure fair allocation and reasonable utilization of resources in a multi-agent system, avoiding the problem of overly fierce resource competition or uneven resource utilization. The operation scenario simulation of the node interactive polling mechanism is performed to obtain the polling operation scenario simulation data. Through these data, the network congestion features that may appear in the polling scenario, such as communication delay, packet loss rate, etc., can be extracted. These features are important basis for evaluating the effect and optimization of the polling mechanism. According to the polling scenario network congestion data, the node interactive polling mechanism is controlled and optimized. The optimized polling mechanism can more effectively deal with network congestion problems and ensure the stability and performance of the system under different load conditions. Combining the node interactive polling optimization mechanism and the node resource allocation priority data, a multi-agent resource competition strategy is formulated. These strategies include dynamic resource adjustment and load balancing measures, aiming to maximize the utilization efficiency and overall performance of system resources while ensuring fair allocation of resources among nodes.

[0014] Preferably, designing a node polling mechanism according to the node resource allocation priority data and the node resource release limited interval includes the following steps: According to the node resource allocation priority data, the limited difference of resource release between nodes with different priorities is calculated for the limited interval of node resource release to obtain limited difference data of resource release; Based on the limited difference data of resource release, the resource release stability between nodes with different priorities is evaluated to obtain the resource release stability evaluation data; According to the resource release limited difference data and the resource release stability evaluation data, the node resource allocation priority data is compensated for the resource release correlation factor to obtain the node resource allocation priority compensation data; The node interactive polling mechanism is designed according to the node resource allocation priority compensation data to obtain the node interactive polling mechanism.

[0015] The present invention calculates the limited difference of resource release between nodes with different priorities according to the node resource allocation priority data. This step aims to quantify the differences in resource release between different nodes and determine which nodes may need more frequent or less resource release to optimize resource utilization efficiency. Based on the limited difference data of resource release, the stability evaluation of resource release is performed. This evaluation can help determine whether the resource release mode of nodes with different priorities is stable, whether there are frequent fluctuations or instability. The stability evaluation data provides a basis for the subsequent design of the polling mechanism. In combination with the limited difference data of resource release and the stability evaluation data, the node resource allocation priority data is compensated. This process can adjust the resource allocation priority according to the differences and stability between nodes, ensure a more fair allocation of resources during the polling process, and avoid the imbalance of resource release. Based on the node resource allocation priority compensation data, an interactive polling mechanism between nodes is designed. This design takes into account the priority and stability characteristics of different nodes to ensure efficient utilization and fair allocation of resources in a multi-node system. The design of the polling mechanism should be able to maximize the overall performance of the system and reduce the risk of causing system bottlenecks.

[0016] Preferably, controlling and optimizing the node interaction polling mechanism according to the polling scenario network congestion data includes the following steps: Calculate the network congestion connection volume for the polling scenario network congestion data to obtain the network congestion connection volume data; Congestion traffic surge capture is performed on the polling scenario network congestion data according to the network congestion connection volume data to obtain congestion traffic surge capture data; Perform weighted linear growth regression analysis on the congestion traffic surge capture data to obtain traffic surge weighted linear regression data; According to the traffic surge weighted linear regression data, the congestion traffic surge capture data is segmented to obtain the congestion traffic surge segmentation data; The Binary Increase Congestion algorithm is used to perform sudden segment response control on the segment data of congestion traffic surge, and the segment response control data of congestion traffic is obtained; The node interactive polling mechanism is controlled and optimized according to the segmented response control data of the congested traffic, and a node interactive polling optimization mechanism is obtained.

[0017] The present invention obtains network congestion connection volume data by calculating the connection volume of network congestion data in a polling scenario. This step can accurately reflect the current network load and provide basic data for subsequent optimization control. According to the network congestion connection volume data, the situation of congestion traffic surge is captured. This includes identifying and recording the sudden increase in traffic in the network, that is, the congestion traffic surge. These data can help understand the causes and patterns of congestion. Weighted linear growth regression analysis is performed on the congestion traffic surge capture data to obtain weighted linear regression data of traffic surge. This analysis helps to predict and simulate the trend of congestion traffic, so as to better understand the changing law of network load. According to the weighted linear regression data, the congestion traffic surge data is segmented. This step decomposes the congestion situation into different trend segments so as to respond to and control the changes in congestion traffic in a more refined manner. The Binary Increase Congestion (BIC) algorithm is used to perform sudden segmented response control on the congestion traffic surge segmented data. The BIC algorithm dynamically adjusts the traffic control strategy to quickly and effectively respond to network congestion and avoid long-term unpredictable congestion. Based on the segmented response control data of congested traffic, the node interaction polling mechanism is optimized. This includes adjusting the communication frequency and data transmission strategy between nodes according to the real-time network congestion situation to ensure that the system can maintain stability and performance under high load and congestion conditions.

[0018] Preferably, step S3 comprises the following steps: Step S31: normalizing the multi-agent resource competition strategy to obtain a multi-agent resource competition normalization strategy; Step S32: Performing hierarchical deep learning processing according to the multi-agent resource competition normalization strategy to obtain resource competition hierarchical deep learning data; Step S33: Based on the resource competition hierarchical deep learning data, multi-agent hierarchical convergence behavior coupling is performed to obtain hierarchical convergence behavior coupling data.

[0019] The present invention can eliminate the unfairness or waste of resources caused by different resource requirements between different agents by normalizing the resource competition strategy between multiple agents. This step ensures that under competitive conditions, each agent can reasonably obtain and use resources according to its needs, thereby improving the efficiency and fairness of the overall system. Using the normalized competition strategy, hierarchical deep learning processing is performed. The main effect of this step is to analyze and predict the behavior and decision-making mode of multiple agents in the resource competition process through deep learning methods. This data can provide a strong basis for subsequent agent collaboration and decision-making, enabling the system to more intelligently adjust the resource allocation strategy to cope with the ever-changing competitive environment. Based on the hierarchical deep learning data of resource competition, the convergence behavior of multiple agents is analyzed and coupled. The main effect of this step is to promote collaboration and convergence between agents in the system, and optimize the collaborative behavior between agents by understanding the competitive impact at different levels. The coupling data reflects the mutual influence and adjustment process of agents in the resource competition in the system, thereby improving the stability and performance of the overall system.

[0020] Preferably, step S33 includes the following steps: Step S331: extracting the layered instability state among multiple agents from the resource competition layered deep learning data to obtain the learning behavior layered instability state data; Step S332: Calculate the instability coefficients between multiple levels of the learning behavior hierarchical instability state data to obtain the hierarchical instability coefficients; Step S333: performing multi-point instability initial value screening on the learning behavior hierarchical instability state data according to the hierarchical instability coefficient to obtain hierarchical instability multi-point screening data; Step S334: performing iterative calculation on the learning behavior stratified instability state data according to the stratified instability multi-point screening data to obtain stratified instability multi-point iterative data; Step S335: Perform multi-agent hierarchical convergence behavior coupling according to the hierarchical instability multi-point iteration data to obtain hierarchical convergence behavior coupling data.

[0021] The present invention identifies hierarchical instability states in a multi-agent system. These states may indicate unstable behaviors of certain agents or groups of agents in the system during resource competition or learning. By extracting these instability states, the system can identify and further analyze the causes of instability, so as to take appropriate measures to stabilize the system. The instability coefficients between different levels (which may be groups of agents, individuals, etc.) are calculated. The instability coefficient reflects the degree of instability within each level and between different levels. Through this step, the system can quantify the degree of instability and help optimize the collaboration and resource allocation strategies between agents. According to the hierarchical instability coefficient, appropriate initial instability points are selected, which may be the key points or starting conditions causing instability. This screening can help the system locate and analyze the root causes of instability more accurately, and provide an important basis for subsequent optimization and control. Based on the selected initial instability points, iterative calculations are performed to analyze the continuous development and evolution of the instability state. This process not only helps to understand the dynamic characteristics of the instability phenomenon, but also provides a more sophisticated and effective control strategy for system design. The final step is to analyze the hierarchical convergence behavior of the multi-agent system based on the iterative data. These data reveal the coupling relationship and influence between agents at different levels during the transition from unstable state to stable state. By analyzing the coupling data, the system can optimize the collaboration and regulation mechanism between agents to achieve more efficient system convergence and resource utilization.

[0022] Preferably, the present invention further provides a multi-agent based deep reinforcement learning system for executing the multi-agent based deep reinforcement learning method as described above, the multi-agent based deep reinforcement learning system comprising: The multi-agent interaction graph construction module is used to extract the operation logs of the multi-agents to obtain the multi-agent operation log data; perform multi-agent interaction structure analysis on the multi-agent operation log data to obtain the multi-agent interaction structure data; construct the multi-agent interaction graph according to the multi-agent interaction structure data to obtain the multi-agent interaction graph; The multi-agent resource competition strategy formulation module is used to label the multi-agent load nodes on the multi-agent interaction graph to obtain the multi-agent load node labeling data; calculate the resource release amount between different nodes on the multi-agent load node labeling data to obtain the load node resource release amount data; calculate the resource release limited interval between different load nodes on the load node resource release amount data to obtain the node resource release limited interval; formulate the multi-agent resource competition strategy according to the node resource release limited interval to obtain the multi-agent resource competition strategy; The multi-agent hierarchical convergence coupling module is used to perform hierarchical deep learning processing according to the multi-agent resource competition strategy to obtain resource competition hierarchical deep learning data; based on the resource competition hierarchical deep learning data, the multi-agent hierarchical convergence behavior coupling is performed to obtain hierarchical convergence behavior coupling data; The competition balance model construction module is used to construct a multi-agent resource competition balance model based on the multi-agent resource competition strategy and the hierarchical convergence behavior coupling data, obtain the multi-agent resource competition balance model, and send the multi-agent resource competition balance model to the cloud platform to perform multi-agent deep reinforcement learning.

[0023] The beneficial effect of the present invention is that by analyzing the operation log data of multiple agents, the interaction structure and mode between each agent can be deeply understood. This helps to identify key interaction nodes, frequently occurring interaction events, and the time and space distribution rules of interaction. The interaction map constructed based on the interaction structure data can help find bottlenecks and optimization points in the system. By identifying and optimizing the key interactions between multiple agents, the overall efficiency and response speed of the system can be improved. Analyzing the interaction map helps to identify potential system failure points or unstable factors. By better managing and regulating the interactions between multiple agents, the possibility of system crashes or failures can be reduced, and the stability and reliability of the system can be improved. According to the load node annotation data and resource release data, the resource usage between different nodes can be accurately calculated. This helps to optimize the allocation and utilization of resources, avoid waste and excessive consumption of resources, and improve the overall resource utilization efficiency of the system. Through the calculation of the limited interval of resource release and the formulation of the multi-agent resource competition strategy, the resource competition between different nodes can be effectively managed and mediated. This helps to reduce the conflict and efficiency loss caused by resource competition and maintain the stability and predictability of system operation. Formulating a resource competition strategy based on the limited interval of node resource release can make the system more flexible and adaptable to different operating environments and workloads. This helps the system to quickly adjust and respond to changes and challenges and maintain efficient operation. Therefore, the present invention provides a multi-agent based deep reinforcement learning method that optimizes a traditional multi-agent based deep reinforcement learning method, solves the problem that it is difficult to balance resource competition among multiple agents and that multiple agents are prone to falling into an unstable state in a traditional multi-agent based deep reinforcement learning method, improves the ability to balance resource competition among multiple agents, and reduces the problem that multiple agents are prone to falling into an unstable state. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the steps of a multi-agent deep reinforcement learning method; Figure 2 for Figure 1 Detailed implementation steps of step S2 in the flowchart; Figure 3 for Figure 1 Detailed implementation steps of step S3 in FIG.

[0025] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0028] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0029] To achieve this, please refer to Figures 1 to 3 , a multi-agent deep reinforcement learning method, the method comprising the following steps: Step S1: extracting the operation log of the multi-agent to obtain the multi-agent operation log data; performing multi-agent interaction structure analysis on the multi-agent operation log data to obtain the multi-agent interaction structure data; constructing a multi-agent interaction graph according to the multi-agent interaction structure data to obtain the multi-agent interaction graph; Step S2: perform multi-agent load node labeling on the multi-agent interaction graph to obtain multi-agent load node labeling data; perform resource release amount calculation between different nodes on the multi-agent load node labeling data to obtain load node resource release amount data; perform resource release limited interval calculation between different load nodes on the load node resource release amount data to obtain node resource release limited interval; formulate a multi-agent resource competition strategy according to the node resource release limited interval to obtain a multi-agent resource competition strategy; Step S3: Performing hierarchical deep learning processing according to the multi-agent resource competition strategy to obtain resource competition hierarchical deep learning data; performing multi-agent hierarchical convergence behavior coupling based on the resource competition hierarchical deep learning data to obtain hierarchical convergence behavior coupling data; Step S4: Construct a multi-agent resource competition balance model based on the multi-agent resource competition strategy and the hierarchical convergence behavior coupling data to obtain the multi-agent resource competition balance model, and send the multi-agent resource competition balance model to the cloud platform to perform multi-agent deep reinforcement learning.

[0030] In the embodiment of the present invention, reference Figure 1 The above is a schematic diagram of the steps of a multi-agent deep reinforcement learning method of the present invention. In this example, the multi-agent deep reinforcement learning method includes the following steps: Step S1: extracting the operation log of the multi-agent to obtain the multi-agent operation log data; performing multi-agent interaction structure analysis on the multi-agent operation log data to obtain the multi-agent interaction structure data; constructing a multi-agent interaction graph according to the multi-agent interaction structure data to obtain the multi-agent interaction graph; In the embodiment of the present invention, operation data is extracted from the log system of each intelligent agent. These data may include event logs, message logs, and error logs, and the log data format is unified. Common formats include JSON, XML, and CSV. Key events and interaction information are extracted from the logs, such as messages sent by the intelligent agent, messages received, and services called, etc., to ensure that all events are arranged in chronological order for subsequent analysis, to identify the interaction relationship between intelligent agents, to construct a message transmission graph between intelligent agents, to count the interaction frequency between various intelligent agents, to identify high-frequency interactions and low-frequency interactions, to organize the extracted interaction relationship and frequency information into structured data, and to store them in the form of a graph structure, wherein each intelligent agent is a node in the graph, and the interaction between intelligent agents is an edge in the graph, and the weight of the edge can be determined according to the interaction frequency, and an intelligent agent interaction graph is constructed using a graph algorithm, such as the Dijkstra algorithm, the Floyd-Warshall algorithm, etc., for path finding and optimization, and the interaction graph is visualized using Graphviz, Gephi, or other graph visualization tools, to analyze the centrality of each node in the graph, to identify key intelligent agents, to identify the community structure in the intelligent agent group, to find the functional clusters of intelligent agents, and to analyze the shortest path and the longest path between intelligent agents.

[0031] Step S2: perform multi-agent load node labeling on the multi-agent interaction graph to obtain multi-agent load node labeling data; perform resource release amount calculation between different nodes on the multi-agent load node labeling data to obtain load node resource release amount data; perform resource release limited interval calculation between different load nodes on the load node resource release amount data to obtain node resource release limited interval; formulate a multi-agent resource competition strategy according to the node resource release limited interval to obtain a multi-agent resource competition strategy; In an embodiment of the present invention, load indicators such as CPU usage, memory usage, bandwidth usage, etc. are defined, and the load data of each intelligent agent is collected from the operation log to record its CPU, memory and bandwidth usage. The load value of each intelligent agent is calculated. For example, the average CPU usage and the maximum memory usage are calculated. According to the load calculation results, nodes with high load values ​​are marked as high-load nodes, and nodes with low load values ​​are marked as low-load nodes. A resource usage model is established to describe the resource usage of nodes with different loads, and the resource release amount is defined, such as the amount of CPU, memory and other resources that can be released by a node within a certain period of time. The load node labeling data is analyzed to calculate the resource release amount between different nodes. For example, the amount of resources released by a high-load node when the load decreases is calculated. A limited interval of resource release is defined, such as the upper and lower limits of the resource release amount. The limited interval of resource release between nodes with different loads is calculated. For example, the interval of resource release amount of a high-load node within a certain load reduction range is calculated. The calculation results are organized into structured data, and the limited interval of resource release of each node is marked. A resource competition strategy model is established to describe the competition mechanism of resources between nodes with different loads. According to the actual operation situation, optimize the resource competition strategy to ensure the efficiency and fairness of resource allocation. According to the actual operation situation, optimize the resource competition strategy to ensure the efficiency and fairness of resource allocation. Deploy the resource competition strategy to the multi-agent system, monitor its operation effect, and adjust it as needed. (For example, there is a system composed of three agents (A, B, C), which needs to label the load nodes, calculate the resource release amount, calculate the limited interval of resource release, and formulate resource competition strategies. Collect the CPU usage and memory usage data of A, B, and C, calculate their average load value, mark A as a high-load node, A's CPU usage is 90%, and the memory usage is 80%. B and C are low-load nodes, with CPU usage of 30% and 40%, respectively, and memory usage of 20% and 30%, respectively. Analyze A's load data and calculate the amount of CPU resources (20%) and memory usage that can be released when the load drops to 70%. The amount of stored resources (10%), the limited interval of resource release of A is defined as the CPU release amount [20%-30%], the memory release amount [10%-20%], the limited interval of resource release of B and C is defined as the CPU release amount [10%-20%], the memory release amount [5%-15%], a resource competition strategy model is established, the competition mechanism of resources among A, B, and C is described, and specific resource competition strategy rules are formulated, such as allocating the resources released by A to B and C first, or allocating them according to the priority of load reduction, and deploying the resource competition strategy to the multi-agent system, monitoring its operation effect, and adjusting it as needed).

[0032] Step S3: Performing hierarchical deep learning processing according to the multi-agent resource competition strategy to obtain resource competition hierarchical deep learning data; performing multi-agent hierarchical convergence behavior coupling based on the resource competition hierarchical deep learning data to obtain hierarchical convergence behavior coupling data; In an embodiment of the present invention, features are extracted from resource competition strategy data as input of a deep learning model, and a hierarchical deep learning model is designed. Each layer of the model corresponds to a resource competition strategy at a different level. According to the complexity of the resource competition strategy, the model is divided into multiple layers, such as an input layer, a hidden layer (multiple layers), and an output layer. The resource competition strategy data is divided into a training set, a validation set, and a test set. The model is trained using the training set, the model parameters are adjusted, and the loss function is optimized. The model hyperparameters are adjusted using the validation set to avoid overfitting. The model performance is evaluated using the test set to ensure the generalization ability of the model. The model parameters are optimized using an optimization algorithm (such as Adam, SGD, etc.) to improve the accuracy and efficiency of the model. The model structure is adjusted, the number of layers is increased or decreased, and the number of neurons in each layer is adjusted to improve the model performance. The output data of the hierarchical deep learning model is collected and sorted as input data for hierarchical convergence behavior. Specific indicators of hierarchical convergence behavior coupling are defined, such as resource allocation efficiency, consistency of competition strategies, etc. A hierarchical coupling model is designed to describe the behavioral coupling relationship between different levels. The hierarchical deep learning data is analyzed to identify the behavioral patterns of each layer. Calculate the degree of behavioral coupling between layers, such as through similarity calculation, correlation analysis, etc. Analyze the convergence behavior between layers and evaluate their stability and consistency. Organize the results of behavioral coupling analysis into structured data, and record the coupling relationship and convergence behavior between layers. Develop optimization strategies to improve the degree of behavioral coupling between layers and enhance the stability and consistency of the system. Apply the optimization strategy to the multi-agent system, monitor its operation effect, and make adjustments as needed.

[0033] Step S4: Construct a multi-agent resource competition balance model based on the multi-agent resource competition strategy and the hierarchical convergence behavior coupling data to obtain the multi-agent resource competition balance model, and send the multi-agent resource competition balance model to the cloud platform to perform multi-agent deep reinforcement learning.

[0034] In an embodiment of the present invention, multi-agent resource competition strategy data is collected, and hierarchical convergence behavior coupling analysis results are collected. A multi-agent resource competition balance model is defined, taking into account indicators such as resource utilization efficiency, load balance, and response time. The structure of the balance model is designed, including an input layer, a hidden layer, and an output layer. The input layer contains resource competition strategy data and hierarchical convergence behavior coupling data, and the output layer is a resource allocation decision. Features are extracted from the resource competition strategy data and the hierarchical convergence behavior coupling data as the input of the model, and the features that have the greatest impact on resource competition balance are selected, such as CPU usage, memory usage, bandwidth usage, node coupling, etc. The data is divided into a training set, a validation set, and a test set. The training set is used to train the balance model and optimize the loss function so that the model can effectively balance resource competition. The validation set is used to adjust the model hyperparameters to avoid overfitting. The test set is used to evaluate the model performance to ensure the generalization ability of the model. The model parameters are optimized using optimization algorithms (such as Adam, SGD, etc.) to improve the accuracy and efficiency of the model. The model structure is adjusted, the number of layers is increased or decreased, and the number of neurons in each layer is adjusted to improve the model performance. Configure the required computing resources and environment on the cloud platform, including GPU / CPU, storage, operating system, etc. Package the trained multi-agent resource competition balance model and ensure that all dependent libraries and configuration files are complete. Upload the model to the cloud platform and perform necessary configuration and deployment. Install and configure reinforcement learning libraries (such as TensorFlow, PyTorch, etc.) to ensure that the cloud platform has the ability to perform deep reinforcement learning. Configure the training environment, including the simulation environment of the multi-agent system, the definition of the reward function, etc. (for example, select a suitable deep reinforcement learning algorithm (such as DQN, DDPG, PPO, etc.) for the training of the multi-agent resource competition balance model. Configure the parameters of the reinforcement learning algorithm, including the learning rate, discount factor, exploration strategy, etc. Initialize the multi-agent strategy and define the initial state and action space. Perform multi-agent interaction in the simulation environment and continuously optimize the strategy through exploration and utilization. Define the reward function to encourage balanced resource competition, improve resource utilization efficiency and system stability. Through multiple iterative training, continuously optimize the multi-agent strategy to achieve the goal of resource competition balance. Monitor the training process in real time, and record training logs and indicators. Use the validation set and test set to evaluate the trained strategy to ensure its effectiveness and robustness in the actual environment. Deploy the trained strategy to the actual multi-agent system and monitor its operation effect).

[0035] Preferably, step S1 comprises the following steps: Step S11: extracting the operation log of the multi-agent to obtain the multi-agent operation log data; Step S12: marking the task processing types between different agents on the multi-agent operation log data to obtain task processing type marking data; Step S13: performing multi-agent interaction structure analysis on the task processing type tag data according to the multi-agent operation log data to obtain multi-agent interaction structure data; Step S14: performing multi-scale characteristic analysis on the multi-agent interaction structure data to obtain multi-scale characteristic data of the interaction structure; Step S15: construct a multi-agent interaction graph based on the multi-agent interaction structure data and the multi-scale characteristic data of the interaction structure to obtain a multi-agent interaction graph.

[0036] In the embodiment of the present invention, operation data is extracted from the log system of each intelligent agent, including event logs, message logs, and error logs, to ensure that the log data format is unified. Common formats include JSON, XML, and CSV. Different types of task processing types are defined, such as computing tasks, data transmission tasks, storage tasks, etc. According to the keywords, event types, and other information in the operation log, a marking rule for the task processing type is formulated, and a script or tool is used to automatically identify and mark the task processing type of each intelligent agent. The task processing type marked data is stored in a database and associated with the original log data. Key events and interaction information are extracted from the log, such as messages sent by the intelligent agent, messages received, and calls. Use the services provided by the intelligent agent to ensure that all events are arranged in chronological order for subsequent analysis, identify the interaction relationship between intelligent agents, build a message passing diagram between intelligent agents, count the interaction frequency between intelligent agents, identify high-frequency interactions and low-frequency interactions, count the interaction frequency between intelligent agents, and identify high-frequency interactions and low-frequency interactions; analyze the interaction data according to different time windows (such as seconds, minutes, hours), identify short-term and long-term interaction patterns, analyze the interaction characteristics of intelligent agents under different network topologies, identify local and global interaction patterns, analyze local interaction characteristics, such as sub-group interactions in an intelligent agent group, analyze global interaction characteristics, such as the interaction network topology of the entire system, and integrate interaction characteristics at different scales. The data is organized into structured data to record the interaction characteristics of different time and space scales; each agent is a node in the graph, and the interaction between agents is the edge in the graph. The weight of the edge can be determined according to the interaction frequency. A graph algorithm is used to build an agent interaction graph, such as the Dijkstra algorithm, the Floyd-Warshall algorithm, etc., for path finding and optimization. Graphviz, Gephi or other graph visualization tools are used to visualize the interaction graph, analyze the centrality of each node in the graph, identify key agents, identify the community structure in the agent group, and find the functional clusters of agents, so as to construct a multi-agent interaction graph and obtain a multi-agent interaction graph. Graph (for example, there is a system composed of three agents A, B, and C. Extract running data from the log system of A, B, and C, delete redundant information, convert log data into JSON format, define computing tasks, data transmission tasks, and storage tasks, use scripts to automatically identify and mark the task types of A, B, and C, manually verify the marking results, extract events such as A sending messages, B receiving messages, and C calling services from the log, identify the interaction relationship between A and B, B and C, and C and A, count the interaction frequency, organize the interaction relationship and frequency information into structured data, analyze the interaction data by minute and hour, identify short-term and long-term interaction patterns, and analyze local characteristics, such as the interaction pattern between A and B;Analyze global characteristics, such as the interaction topology of the entire system, organize multi-scale characteristic data into structured data, define A, B, and C as nodes, establish edges AB, BC, and CA, use Graphviz to visualize the graph, display the interaction relationship between agents, analyze the centrality of A, and identify it as the key agent of the system; identify the community structure in the agent group). ;

[0037] Preferably, step S2 comprises the following steps: Step S21: annotating the multi-agent load nodes of the multi-agent interaction graph according to the multi-agent operation log data to obtain the multi-agent load node annotated data; Step S22: sorting the multi-agent load node labeling data according to resource scarcity among different load nodes to obtain load node resource scarcity sorting data; Step S23: Calculate the resource release amount between different nodes of the multi-agent load node annotation data according to the load node resource shortage ranking data and the multi-agent operation log data to obtain the load node resource release amount data; Step S24: Calculate the resource release limited intervals between different load nodes on the load node resource release amount data to obtain the node resource release limited intervals; Step S25: evaluating the performance difference between different load nodes according to the load node resource release amount data and the node resource release limited interval, and obtaining node performance difference evaluation data; Step S26: Formulate a multi-agent resource competition strategy based on the node performance difference evaluation data and the limited interval of node resource release to obtain a multi-agent resource competition strategy.

[0038] As an example of the present invention, refer to Figure 2 As shown, in this example, step S2 includes: Step S21: annotating the multi-agent load nodes of the multi-agent interaction graph according to the multi-agent operation log data to obtain the multi-agent load node annotated data; In an embodiment of the present invention, operation log data of multiple agents are collected, including resource usage such as CPU usage, memory usage, and bandwidth usage. The multi-agent interaction map obtained in step S1 is used as the basis for load node labeling, and the standards of load nodes are defined, such as high CPU usage and high memory usage. The resource usage of each agent is calculated based on the operation log data, and the load nodes are identified. According to the load calculation results, the agents are divided into high-load nodes, medium-load nodes, and low-load nodes. A load label is labeled for each node, such as "high load", "medium load", and "low load". The load node label data is stored in a database and associated with the multi-agent interaction map.

[0039] Step S22: sorting the multi-agent load node labeling data according to resource scarcity among different load nodes to obtain load node resource scarcity sorting data; In an embodiment of the present invention, the resource usage of each load node is analyzed, such as CPU usage, memory usage, and bandwidth usage, and resource scarcity indicators are defined, such as resource usage and resource remaining amount. A sorting algorithm (such as quick sort, heap sort, etc.) is used to sort the load nodes by resource scarcity, and load node resource scarcity sorting data is obtained, which is arranged from the most scarce to the least scarce. The load node resource scarcity sorting data is stored in a database for subsequent processing.

[0040] Step S23: Calculate the resource release amount between different nodes of the multi-agent load node annotation data according to the load node resource shortage ranking data and the multi-agent operation log data to obtain the load node resource release amount data; In an embodiment of the present invention, the resource shortage ranking data of the load node is extracted from the database, and the multi-agent operation log data is extracted for resource release calculation, and the indicators of resource release are defined, such as CPU release, memory release, bandwidth release, etc. According to the resource shortage ranking result, the conditions and thresholds for resource release are determined. The resource release amount of each load node is calculated according to the shortage ranking data and the operation log data. For each pair of load nodes, the resource release amount between them is calculated. (For example, the load node resource shortage ranking data and the operation log data are extracted from the database, and the CPU release amount is defined as the current CPU usage rate minus the target usage rate such as 70%, and the release condition is set: high-load nodes need to release resources to reduce the degree of shortage, and the CPU release amount of A is calculated such as the current usage rate is 90%, the target usage rate is 70%, and the release amount is 20%), and the calculated load node resource release amount data is stored in the database.

[0041] Step S24: Calculate the resource release limited intervals between different load nodes on the load node resource release amount data to obtain the node resource release limited intervals; In an embodiment of the present invention, load node resource release data is extracted from a database, multi-agent load node annotation data is extracted from a database, a calculation model for a limited interval of resource release is established, resource usage of each node and the total amount of system resources are considered, interval parameters such as a minimum release amount, a maximum release amount, and an average release amount are defined, and the resource release amount of each node is divided into different intervals (such as low, medium, and high) according to the release amount data. According to actual needs, the interval parameters are adjusted to ensure the rationality and operability of each interval, and the node resource release limited interval data is stored in the database and associated with the load node resource release amount data.

[0042] Step S25: evaluating the performance difference between different load nodes according to the load node resource release amount data and the node resource release limited interval, and obtaining node performance difference evaluation data; In an embodiment of the present invention, load node resource release data is extracted from a database. Node resource release limited interval data is extracted from a database. Indicators for performance difference evaluation are defined, such as response time, throughput, resource utilization, etc. A weight is set for each performance indicator to reflect its relative importance. Each node is evaluated using a performance difference evaluation model (such as a weighted scoring method, a hierarchical analysis method, etc.). Each node is scored based on the evaluation indicators and weights, and its performance difference score is calculated. The node performance difference evaluation data is stored in a database and associated with the node resource release data and the limited interval data.

[0043] Step S26: Formulate a multi-agent resource competition strategy based on the node performance difference evaluation data and the limited interval of node resource release to obtain a multi-agent resource competition strategy.

[0044] In an embodiment of the present invention, node performance difference evaluation data is extracted from a database, and node resource release limited interval data is extracted from a database. The objectives of the resource competition strategy are defined, such as maximizing system performance, minimizing resource conflicts, etc. A decision model for the resource competition strategy is established, taking into account the performance differences of each node and the limited interval of resource release. Specific resource competition strategy rules are formulated, such as resource priority allocation, dynamic adjustment mechanism, etc. Optimize the resource competition strategy using optimization algorithms (such as genetic algorithms, particle swarm algorithms, etc.) to ensure the effectiveness and stability of the strategy. The multi-agent resource competition strategy is stored in a database for subsequent execution and adjustment.

[0045] Preferably, step S24 comprises the following steps: Step S241: performing resource release amount fluctuation analysis between different nodes according to the load node resource release amount data to obtain node resource release amount fluctuation data; Step S242: calculating the resource release fluctuation extreme value difference between nodes with different loads on the node resource release amount fluctuation data to obtain the resource release fluctuation extreme value difference; Step S243: performing local similar fluctuation structure analysis between nodes with different loads on the node resource release amount fluctuation data according to the resource release fluctuation extreme value difference to obtain local similar fluctuation structure data; Step S244: performing nonlinear iterative numerical approximation on the node resource release amount fluctuation data based on the local similar fluctuation structure data to obtain release amount iterative numerical approximation data; Step S245: performing local convex optimization processing on the local similar fluctuation structure data according to the release amount iterative numerical approximation data to obtain local fluctuation convex optimization data; Step S246: Calculate the resource release finite interval between nodes with different loads based on the release amount iterative numerical approximation data and the local fluctuation convex optimization data for the node resource release amount fluctuation data to obtain the node resource release finite interval.

[0046] In an embodiment of the present invention, the resource release data of the load node is extracted from the database, an analysis time window (such as minutes, hours, and days) is defined for fluctuation analysis, the resource release data of each node is converted into time series data, the resource release fluctuation in each time window is calculated, including indicators such as mean, variance, and standard deviation, the calculated node resource release fluctuation data is stored in the database, associated with the original resource release data, the maximum and minimum values ​​of the resource release of each node are identified, the resource release fluctuation extreme value difference of each node, that is, the difference between the maximum and minimum values, is calculated, the extreme value differences between different nodes are compared, the nodes with the largest resource release fluctuation and the smallest node are identified, the calculated resource release fluctuation extreme value difference data is stored in the database, associated with the fluctuation data, a similarity measurement method for resource release fluctuation between nodes is defined, such as dynamic time warping (DTW) and Euclidean distance, the similarity measurement method is used to identify local similar fluctuation structures between different nodes, node pairs with similar fluctuation patterns are identified, their local fluctuation characteristics are analyzed, and the local similar fluctuation structure data is organized into structured data for subsequent analysis. (For example, there is a system composed of three agents A, B, and C. The resource release data of A, B, and C are extracted from the database. The analysis time window is defined as hours. The resource release data of A, B, and C are converted into time series data at the hour level. The mean, variance, and standard deviation of the resource release in each hour are calculated. The resource release fluctuation data of A, B, and C are extracted from the database. The maximum value of resource release of A is calculated to be 50%, the minimum value is 10%, and the extreme value difference is 40%. The extreme value difference of resource release between B and C is calculated. The extreme value difference of A, B, and C is compared to identify the node with the largest resource release fluctuation and the node with the smallest resource release fluctuation. The dynamic time warping (DTW) method is used to calculate the similarity of resource release fluctuation between A, B, and C, identify that A and B have similar fluctuation patterns, analyze the local fluctuation characteristics of A and B, identify their local similar fluctuation structures, and separate the local local similar fluctuation structure data is organized into structured data); select appropriate nonlinear iterative algorithm, such as Newton iteration method and conjugate gradient method, set initial conditions and parameters, including initial iteration value and convergence condition, iteratively calculate the node resource release fluctuation data, gradually approach the true value, verify the accuracy of the result after each iteration step, ensure that the iteration result gradually converges, store the calculated release iterative numerical approximation data in the database, associate it with the original fluctuation data, set the objective function of local fluctuation convex optimization, such as minimizing resource release fluctuation, define the constraints in the optimization process, such as total resource constraint and node performance constraint, select appropriate convex optimization algorithm, such as interior point method and gradient descent method, adjust the parameters of the optimization algorithm according to the data characteristics, ensure the stability and effectiveness of the optimization process, optimize the iterative approximation data, and obtain the convex optimization result of local fluctuation.During the optimization process, verify the results of each step to ensure that the optimization process converges to the local optimal solution. Define the finite interval of resource release based on the iterative approximation data and optimization data. Establish a calculation model for the finite interval of resource release, considering the resource usage between nodes and the total amount of system resources. According to the calculation model, divide the node resource release into different finite intervals (such as low, medium, and high). Adjust the interval parameters according to actual needs and optimization results to ensure the rationality and operability of each interval. The calculated node resource release finite interval data is stored in the database, and associated with the iterative approximation data and the optimization data (for example, the local similar fluctuation structure data and resource release fluctuation data of A, B, and C are extracted from the database, the Newton iteration method is selected as the nonlinear iterative algorithm, the initial iteration value is set to the current resource release, the convergence condition is that the error is less than 1%, the resource release fluctuation data of A, B, and C are iteratively calculated, and the true value is gradually approached. After each iteration step, the accuracy of the result is verified to ensure that the iteration result gradually converges, the iterative numerical approximation data and the local similar fluctuation structure data of the release of A, B, and C are extracted from the database, the optimization goal is set to minimize the resource release fluctuation, and the total resource constraint and node performance are defined. Constraints, select gradient descent as the convex optimization algorithm, tune the parameters of the optimization algorithm to ensure the stability and effectiveness of the optimization process, optimize the iterative approximation data of A, B, and C, and obtain the convex optimization results of local fluctuations. Verify the results of each step to ensure that the optimization process converges to the local optimal solution. Define the finite interval of resource release based on the iterative approximation data and optimization data, and establish a calculation model for the finite interval of resource release. Consider the resource usage between nodes and the total amount of system resources, and divide the resource release of A, B, and C into different finite intervals such as low 10%-20%, medium 20%-40%, and high 40%-50%. Adjust the interval parameters according to actual needs and optimization results to ensure the rationality and operability of each interval).

[0047] Preferably, step S26 comprises the following steps: Step S261: Calculate the processing efficiency of the same task volume between different nodes based on the node performance difference evaluation data to obtain node processing efficiency difference data; Step S262: Establish resource allocation priorities among different nodes according to the node processing efficiency difference data and the node resource release limited interval, and obtain node resource allocation priority data; Step S263: Designing a node interactive polling mechanism according to the node resource allocation priority data and the node resource release limited interval to obtain a node interactive polling mechanism; Step S264: simulating the operation scenario of the node interactive polling mechanism to obtain polling operation scenario simulation data; extracting network congestion features according to the polling operation scenario simulation data to obtain polling scenario network congestion data; Step S265: performing control optimization processing on the node interaction polling mechanism according to the polling scenario network congestion data to obtain a node interaction polling optimization mechanism; Step S266: Formulate a multi-agent resource competition strategy based on the node interaction polling optimization mechanism and the node resource allocation priority data to obtain a multi-agent resource competition strategy.

[0048] In the embodiment of the present invention, a calculation formula for processing efficiency of the same task volume is set, such as efficiency = task volume / (time multiplied by resource consumption), the efficiency of each node when processing the same task volume is calculated, and the node processing efficiency difference data is obtained. The algorithm for setting the resource allocation priority is set, such as setting the priority according to the efficiency and the limited interval of resource release, and calculating the resource allocation priority of each node according to the processing efficiency difference data and the limited interval of resource release. The priority calculation formula is: Multiply by processing efficiency + Multiply by (1- ),in, and is the weight parameter, and obtains the node resource allocation priority data, defines the interactive polling rules between nodes, usually designed according to the resource allocation priority and resource release situation, designs polling strategies, such as time slice polling, weighted polling, etc., to ensure that high-priority nodes obtain more resources, sets polling parameters, including time slice size, polling order, priority weight, and builds a node interactive polling mechanism model to ensure that the polling strategy can be adjusted dynamically (for example, there is a system composed of three intelligent agents A, B, and C. The time required for A, B, and C to process the same amount of tasks is counted, for example, A is 10 seconds, B is 12 seconds, and C is 8 seconds. The processing efficiency is calculated, A is 0.1 task / second, B is 0.083 task / second, and C is 0.125 task / second. Compare the processing efficiency and obtain the processing efficiency difference data of A, B, and C. Define the priority rules and set the weight parameters =0.6, =0.4, calculate the priority score, sort A, B, and C according to the score, define the polling rules, allocate time slices according to the priority, C is 50%, A is 30%, and B is 20%. Design the time slice polling strategy to ensure that C obtains more resources, set the polling parameters, such as the time slice size is 100 milliseconds, build the node interaction polling mechanism model, and ensure that C, A, and B are polled in order of priority. Implement the polling mechanism algorithm to ensure that high-priority nodes get resources first). Select and build a suitable simulation platform, such as MATLAB, Simulink, and OMNeT++. Select and build a suitable simulation platform, such as MATLAB, Simulink, and OMNeT++. Execute the node interactive polling mechanism on the simulation platform to simulate the actual operation of the multi-agent system. During the simulation process, collect various data of the polling operation scenario in real time, including task processing time, resource usage, and network transmission. Analyze the simulation data and extract the characteristic data of network congestion, such as transmission delay, packet loss rate, and bandwidth utilization. Define the control optimization goals, such as minimizing transmission delay, reducing packet loss rate, and improving bandwidth utilization. Set constraints in the optimization process, such as total resources and node priority. Control and optimize the node interactive polling mechanism, adjust polling parameters and strategies to achieve the optimization goals. During the optimization process, verify the results of each step to ensure that the optimization process converges to the optimal solution. Construct a multi-agent resource competition strategy model based on the optimized interactive polling mechanism and resource allocation priority. Define resource competition strategy rules, such as priority allocation rules and resource scheduling rules.

[0049] Preferably, designing a node polling mechanism according to the node resource allocation priority data and the node resource release limited interval includes the following steps: According to the node resource allocation priority data, the limited difference of resource release between nodes with different priorities is calculated for the limited interval of node resource release to obtain limited difference data of resource release; Based on the limited difference data of resource release, the resource release stability between nodes with different priorities is evaluated to obtain the resource release stability evaluation data; According to the resource release limited difference data and the resource release stability evaluation data, the node resource allocation priority data is compensated for the resource release correlation factor to obtain the node resource allocation priority compensation data; The node interactive polling mechanism is designed according to the node resource allocation priority compensation data to obtain the node interactive polling mechanism.

[0050] In an embodiment of the present invention, nodes are divided into different priority groups according to resource allocation priority data. The limited difference of resource release between different priority groups is calculated, for example: limited difference of resource release = |mean value of resource release interval of high priority group - mean value of resource release interval of low priority group |, and resource release stability evaluation data is obtained. According to the limited difference and stability of resource release, a compensation factor is set, and the node resource allocation priority is compensated according to the compensation factor to obtain node resource allocation priority compensation data. According to the compensated resource allocation priority, the node interactive polling rules are defined, and specific polling strategies are designed, such as time slice polling, weighted polling, etc., to ensure that high priority nodes obtain more resources, set polling parameters, including time slice size, polling order, priority weight, and construct a node interactive polling mechanism model to ensure that the polling strategy can be adjusted dynamically. Implement the polling mechanism algorithm to ensure the executability and effectiveness of the algorithm.

[0051] Preferably, controlling and optimizing the node interaction polling mechanism according to the polling scenario network congestion data includes the following steps: Calculate the network congestion connection volume for the polling scenario network congestion data to obtain the network congestion connection volume data; Congestion traffic surge capture is performed on the polling scenario network congestion data according to the network congestion connection volume data to obtain congestion traffic surge capture data; Perform weighted linear growth regression analysis on the congestion traffic surge capture data to obtain traffic surge weighted linear regression data; According to the traffic surge weighted linear regression data, the congestion traffic surge capture data is segmented to obtain the congestion traffic surge segmentation data; The Binary Increase Congestion algorithm is used to perform sudden segment response control on the segment data of congestion traffic surge, and the segment response control data of congestion traffic is obtained; The node interactive polling mechanism is controlled and optimized according to the segmented response control data of the congested traffic, and a node interactive polling optimization mechanism is obtained.

[0052] In an embodiment of the present invention, first, the network congestion data of the polling scenario is obtained and analyzed. The calculation index of the network congestion connection volume is defined, such as the number of connections, bandwidth occupancy rate, etc. The network congestion connection volume of the entire system is calculated by summarizing and counting the number of network connections and bandwidth usage at each time point. The specific method is to extract the number of connections and bandwidth occupancy data of each node, and then summarize these data to calculate the total network connection volume. Finally, the calculated network congestion connection volume data is stored in a database to provide basic data for subsequent analysis and optimization. According to the network congestion connection volume data obtained in the first step, it is further analyzed to capture the surge phenomenon of network traffic. Define the capture index of traffic surge, such as the instantaneous connection number growth rate or the traffic surge rate. In the data analysis process, a threshold is set, and when the network traffic or connection number growth exceeds the set threshold, it is marked as a traffic surge event. The specific method is to perform differential calculation on the time series data to identify significant growth events that occur in a short period of time. The data of these surge events are stored in the database for subsequent weighted linear growth regression analysis. Weighted linear growth regression analysis is performed on the traffic surge event data captured in the second step. First, assign a weight to each surge event, usually based on the importance or frequency of the event. Then, use a weighted linear regression model to fit the data, calculate the regression coefficient, and analyze the trend and characteristics of traffic growth. The specific method is to describe the changing law of traffic surges through a linear regression equation. The result data of the regression analysis is stored in the database to provide a basis for subsequent trend segmentation processing. According to the results of the weighted linear regression analysis, the congestion traffic surge capture data is trend segmented. First, define the trend segmentation rules, such as segmentation according to the time window or traffic growth rate. Then, the traffic surge data is segmented according to the set rules to identify the traffic growth trend in different time periods. The specific method is to divide the time series data into multiple continuous time periods through the sliding window technology, and analyze and describe the traffic changes in each time period. Finally, the congestion traffic surge segmentation data is obtained and stored in the database for subsequent sudden segmentation response control. The BinaryIncreaseCongestion (BIC) algorithm is used to perform sudden segmentation response control on the congestion traffic surge segmentation data obtained in the fourth step. The BIC algorithm is an algorithm for network traffic control that can effectively cope with sudden changes in network congestion. The specific method is to identify the key time points of traffic surges by performing binary search and optimization adjustments on the traffic surge segmentation data, and take corresponding control measures. The calculated congestion traffic segmentation response control data can provide precise control parameters and strategies for the optimization of the node interactive polling mechanism. According to the congestion traffic segmentation response control data obtained in the fifth step, the node interactive polling mechanism is optimized.First, analyze and understand the congestion response control data to identify potential bottlenecks and optimization space in the system. Then, adjust the parameters and strategies of the node interactive polling mechanism based on these data, such as adjusting the allocation of time slices, priority weights, etc., to improve the resource utilization efficiency and network performance of the overall system. Finally, apply the optimized node interactive polling mechanism to the actual system for testing and verification to ensure the effectiveness and stability of the optimization measures. Through the above steps, the control optimization of the node interactive polling mechanism is completed, and the operating efficiency and stability of the multi-agent system under high load and complex network environments are improved.

[0053] Preferably, step S3 comprises the following steps: Step S31: normalizing the multi-agent resource competition strategy to obtain a multi-agent resource competition normalization strategy; Step S32: Performing hierarchical deep learning processing according to the multi-agent resource competition normalization strategy to obtain resource competition hierarchical deep learning data; Step S33: Based on the resource competition hierarchical deep learning data, multi-agent hierarchical convergence behavior coupling is performed to obtain hierarchical convergence behavior coupling data.

[0054] As an example of the present invention, refer to Figure 3 As shown, in this example, step S3 includes: Step S31: normalizing the multi-agent resource competition strategy to obtain a multi-agent resource competition normalization strategy; In an embodiment of the present invention, multi-agent resource competition strategy data are obtained and organized. These data generally include information on resource allocation, usage efficiency, competition priority, etc. between different agents. The purpose of standardizing and normalizing these data is to eliminate the dimensional differences between different data dimensions and make the data more comparable. Specific methods include minimum-maximum normalization and Z-score normalization, normalizing each strategy parameter, and scaling it to the [0, 1] interval. Through normalization, a multi-agent resource competition normalization strategy is obtained to provide consistent and standard data input for subsequent hierarchical deep learning processing. The focus of this step is to ensure the integrity and consistency of the data, while retaining the characteristics and distribution of the original data as much as possible, so as to more accurately reflect the actual resource competition situation.

[0055] Step S32: Performing hierarchical deep learning processing according to the multi-agent resource competition normalization strategy to obtain resource competition hierarchical deep learning data; In an embodiment of the present invention, a hierarchical deep learning model is designed and trained based on the normalized multi-agent resource competition strategy. First, the structure of the model is determined, including the number of input layers, hidden layers, and output layers and the configuration of their neurons. For the problem of multi-agent resource competition, it is usually necessary to build a multi-layer neural network to capture complex nonlinear relationships. Then, training data is prepared, and the normalized strategy data is divided into a training set and a validation set. The model is trained by the back propagation algorithm and the gradient descent optimizer, and the network weights and biases are continuously adjusted to minimize the loss function. In the specific training process, a cross-validation method can be used to prevent overfitting. After the training is completed, the model is evaluated using the validation set data to verify its accuracy and robustness in predicting resource competition strategies. Through this process, resource competition hierarchical deep learning data is obtained, which provides basic data and model support for the next step of multi-agent hierarchical convergence behavior coupling.

[0056] Step S33: Based on the resource competition hierarchical deep learning data, multi-agent hierarchical convergence behavior coupling is performed to obtain hierarchical convergence behavior coupling data.

[0057] In an embodiment of the present invention, the hierarchical convergence behavior coupling analysis of multi-agents is performed using resource competition hierarchical deep learning data. First, the hierarchical structure and behavior model of the agent are defined, including the top decision layer, the middle coordination layer, and the bottom execution layer. Based on the data output by the deep learning model, the behavior coupling relationship between different levels is analyzed. For example, by analyzing the changes in the top-level decision, its impact on the middle-level coordination and the bottom-level execution is predicted. Using the multi-agent reinforcement learning algorithm, the behaviors of different levels are jointly optimized to ensure the collaborative work of the entire system in a resource competition environment. Specific methods include policy gradient methods, Q learning, etc. Through these algorithms, the collaboration strategies between the various levels are optimized to improve the resource utilization efficiency and stability of the overall system. In actual operation, simulation and experimental methods can be used to verify the effectiveness and performance of the coupling model. Finally, the hierarchical convergence behavior coupling data is obtained to provide support and guarantee for the efficient operation of the multi-agent system in a complex resource competition environment.

[0058] Preferably, step S33 includes the following steps: Step S331: extracting the layered instability state among multiple agents from the resource competition layered deep learning data to obtain the learning behavior layered instability state data; Step S332: Calculate the instability coefficients between multiple levels of the learning behavior hierarchical instability state data to obtain the hierarchical instability coefficients; Step S333: performing multi-point instability initial value screening on the learning behavior hierarchical instability state data according to the hierarchical instability coefficient to obtain hierarchical instability multi-point screening data; Step S334: performing iterative calculation on the learning behavior stratified instability state data according to the stratified instability multi-point screening data to obtain stratified instability multi-point iterative data; Step S335: Perform multi-agent hierarchical convergence behavior coupling according to the hierarchical instability multi-point iteration data to obtain hierarchical convergence behavior coupling data.

[0059] In an embodiment of the present invention, the hierarchical instability state extraction between multiple agents is performed using resource competition hierarchical deep learning data. The goal of this step is to identify and analyze the hierarchical instability phenomenon that may exist in the system, that is, the coordination and decision-making between different levels may cause the overall efficiency of the system to decrease or become unstable due to resource competition. Through the data output by the deep learning model, the resource competition between different levels can be observed, such as how the changes in the top decision-making layer affect the behavior of the middle-level coordination and the bottom-level execution. The specific operation includes time series analysis and feature extraction of the deep learning data to identify hierarchical instability events in the system. This may involve anomaly detection algorithms or pattern recognition techniques to capture and mark system instability signals that appear during resource competition. These signals can be change patterns of behaviors at each level, such as frequent changes in the decision-making layer, decreased execution efficiency of the coordination layer, etc. Through effective data processing and analysis, the hierarchical instability state data of the learning behavior is obtained, which lays the foundation for the subsequent calculation of the instability coefficient. The instability coefficient between multiple levels is calculated using the information extracted from the hierarchical instability state data. The instability coefficient is used to quantify the intensity and impact of the instability state between different levels, and is one of the important indicators for evaluating the stability and performance of the system. In the calculation process, statistical analysis methods and mathematical modeling techniques can be used to comprehensively consider the time series data of behaviors at each level and the frequency and duration of instability events. Specifically, the instability events at each level are first counted and classified to determine their frequency and intensity. Then, through appropriate mathematical models or algorithms, these data are converted into instability coefficients to reflect the degree of influence of resource competition between different levels. The calculation of the instability coefficient not only depends on the quantity and quality of the data, but also takes into account the complexity and nonlinear characteristics of the behavioral coupling between the levels. Through this step, the hierarchical instability coefficient is obtained, which provides a quantitative basis for subsequent screening and iterative calculations. According to the hierarchical instability coefficient, the multi-point instability initial value screening of the learning behavior layered instability state data is performed. The purpose of this step is to select the most representative and significant instability initial value points from all instability states for subsequent iterative calculations and in-depth analysis. The specific operation includes combining the instability coefficient and the characteristics of the time series data to identify those instability initial value points that have a significant influence in the resource competition process. This may involve pattern matching and cluster analysis techniques to ensure that the selected initial instability values ​​can effectively represent the response of the system under different resource competition conditions. The diversity and representativeness of the initial instability values ​​need to be taken into account during the screening process to ensure the comprehensiveness and accuracy of the subsequent analysis. The stratified instability state data of the learning behavior is iteratively calculated using the screened stratified instability initial value point data. In this step, the evolution process and impact of different instability initial value points under resource competition conditions are analyzed by applying numerical simulation and computational experimental techniques. Specific operations include establishing a mathematical model or simulation platform to simulate the system response and subsequent evolution caused by the instability initial value point.Through continuous iterative calculations, we observe the dynamic changes of the system under various competitive conditions and analyze its stability and behavioral characteristics. This process can not only deeply understand the resource competition mechanism between different levels in the system, but also reveal potential problems and optimization space. Finally, we obtain multi-point iterative data of hierarchical instability, which provides a scientific basis for the in-depth understanding and optimization of the coupling of multi-agent hierarchical convergence behavior.

[0060] Preferably, the present invention further provides a multi-agent based deep reinforcement learning system for executing the multi-agent based deep reinforcement learning method as described above, the multi-agent based deep reinforcement learning system comprising: The multi-agent interaction graph construction module is used to extract the operation logs of the multi-agents to obtain the multi-agent operation log data; perform multi-agent interaction structure analysis on the multi-agent operation log data to obtain the multi-agent interaction structure data; construct the multi-agent interaction graph according to the multi-agent interaction structure data to obtain the multi-agent interaction graph; The multi-agent resource competition strategy formulation module is used to label the multi-agent load nodes on the multi-agent interaction graph to obtain the multi-agent load node labeling data; calculate the resource release amount between different nodes on the multi-agent load node labeling data to obtain the load node resource release amount data; calculate the resource release limited interval between different load nodes on the load node resource release amount data to obtain the node resource release limited interval; formulate the multi-agent resource competition strategy according to the node resource release limited interval to obtain the multi-agent resource competition strategy; The multi-agent hierarchical convergence coupling module is used to perform hierarchical deep learning processing according to the multi-agent resource competition strategy to obtain resource competition hierarchical deep learning data; based on the resource competition hierarchical deep learning data, the multi-agent hierarchical convergence behavior coupling is performed to obtain hierarchical convergence behavior coupling data; The competition balance model construction module is used to construct a multi-agent resource competition balance model based on the multi-agent resource competition strategy and the hierarchical convergence behavior coupling data, obtain the multi-agent resource competition balance model, and send the multi-agent resource competition balance model to the cloud platform to perform multi-agent deep reinforcement learning.

[0061] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0062] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A multi-agent deep reinforcement learning method, characterized in that: The following steps are involved: Step S1: extracting the operation log of the multi-agent to obtain the multi-agent operation log data; Perform multi-agent interaction structure analysis on multi-agent operation log data to obtain multi-agent interaction structure data; Construct a multi-agent interaction graph based on the multi-agent interaction structure data to obtain a multi-agent interaction graph; Step S2: labeling the multi-agent load nodes on the multi-agent interaction graph to obtain multi-agent load node labeling data; The resource release amount between different nodes is calculated for the multi-agent load node labeling data to obtain the load node resource release amount data; The resource release finite intervals between different load nodes are calculated for the load node resource release data to obtain the node resource release finite intervals; the multi-agent resource competition strategy is formulated according to the node resource release finite intervals to obtain the multi-agent resource competition strategy; Step S3: Performing hierarchical deep learning processing according to the multi-agent resource competition strategy to obtain resource competition hierarchical deep learning data; Based on the resource competition hierarchical deep learning data, multi-agent hierarchical convergence behavior coupling is performed to obtain hierarchical convergence behavior coupling data; Step S4: Construct a multi-agent resource competition balance model based on the multi-agent resource competition strategy and the hierarchical convergence behavior coupling data to obtain the multi-agent resource competition balance model, and send the multi-agent resource competition balance model to the cloud platform to perform multi-agent deep reinforcement learning.

2. The multi-agent deep reinforcement learning method according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: extracting the operation log of the multi-agent to obtain the multi-agent operation log data; Step S12: marking the task processing types between different agents on the multi-agent operation log data to obtain task processing type marking data; Step S13: performing multi-agent interaction structure analysis on the task processing type tag data according to the multi-agent operation log data to obtain multi-agent interaction structure data; Step S14: performing multi-scale characteristic analysis on the multi-agent interaction structure data to obtain multi-scale characteristic data of the interaction structure; Step S15: construct a multi-agent interaction graph based on the multi-agent interaction structure data and the multi-scale characteristic data of the interaction structure to obtain a multi-agent interaction graph.

3. The multi-agent deep reinforcement learning method according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: labeling the multi-agent load nodes of the multi-agent interaction graph according to the multi-agent operation log data to obtain the multi-agent load node labeling data; Step S22: sorting the multi-agent load node labeling data according to resource scarcity among different load nodes to obtain load node resource scarcity sorting data; Step S23: Calculate the resource release amount between different nodes of the multi-agent load node annotation data according to the load node resource shortage ranking data and the multi-agent operation log data to obtain the load node resource release amount data; Step S24: Calculate the resource release limited intervals between different load nodes on the load node resource release amount data to obtain the node resource release limited intervals; Step S25: evaluating the performance difference between different load nodes according to the load node resource release amount data and the node resource release limited interval, and obtaining node performance difference evaluation data; Step S26: Formulate a multi-agent resource competition strategy based on the node performance difference evaluation data and the limited interval of node resource release to obtain a multi-agent resource competition strategy.

4. The multi-agent deep reinforcement learning method according to claim 3 is characterized in that: Step S24 includes the following steps: Step S241: performing resource release fluctuation analysis between different nodes according to the load node resource release data to obtain node resource release fluctuation data; Step S242: calculating the resource release fluctuation extreme value difference between nodes with different loads on the node resource release amount fluctuation data to obtain the resource release fluctuation extreme value difference; Step S243: performing local similar fluctuation structure analysis between nodes with different loads on the node resource release amount fluctuation data according to the resource release fluctuation extreme value difference to obtain local similar fluctuation structure data; Step S244: performing nonlinear iterative numerical approximation on the node resource release amount fluctuation data based on the local similar fluctuation structure data to obtain release amount iterative numerical approximation data; Step S245: performing local convex optimization processing on the local similar fluctuation structure data according to the release amount iterative numerical approximation data to obtain local fluctuation convex optimization data; Step S246: Calculate the resource release finite interval between nodes with different loads based on the release amount iterative numerical approximation data and the local fluctuation convex optimization data for the node resource release amount fluctuation data to obtain the node resource release finite interval.

5. The multi-agent deep reinforcement learning method according to claim 3, characterized in that: Step S26 includes the following steps: Step S261: Calculate the processing efficiency of the same task volume between different nodes based on the node performance difference evaluation data to obtain node processing efficiency difference data; Step S262: Establish resource allocation priorities among different nodes according to the node processing efficiency difference data and the node resource release limited interval, and obtain node resource allocation priority data; Step S263: Designing a node interactive polling mechanism according to the node resource allocation priority data and the node resource release limited interval to obtain a node interactive polling mechanism; Step S264: simulating the operation scenario of the node interactive polling mechanism to obtain polling operation scenario simulation data; extracting network congestion features according to the polling operation scenario simulation data to obtain polling scenario network congestion data; Step S265: performing control optimization processing on the node interaction polling mechanism according to the polling scenario network congestion data to obtain a node interaction polling optimization mechanism; Step S266: Formulate a multi-agent resource competition strategy based on the node interaction polling optimization mechanism and the node resource allocation priority data to obtain a multi-agent resource competition strategy.

6. The multi-agent deep reinforcement learning method according to claim 5, characterized in that: The node polling mechanism design based on the node resource allocation priority data and the node resource release limited interval includes the following steps: According to the node resource allocation priority data, the limited difference of resource release between nodes with different priorities is calculated for the limited interval of node resource release to obtain limited difference data of resource release; Based on the limited difference data of resource release, the resource release stability between nodes with different priorities is evaluated to obtain the resource release stability evaluation data; According to the resource release limited difference data and the resource release stability evaluation data, the node resource allocation priority data is compensated for the resource release correlation factor to obtain the node resource allocation priority compensation data; The node interactive polling mechanism is designed according to the node resource allocation priority compensation data to obtain the node interactive polling mechanism.

7. The multi-agent deep reinforcement learning method according to claim 5, characterized in that: The control optimization process of the node interaction polling mechanism based on the polling scenario network congestion data includes the following steps: Calculate the network congestion connection volume for the polling scenario network congestion data to obtain the network congestion connection volume data; Congestion traffic surge capture is performed on the polling scenario network congestion data according to the network congestion connection volume data to obtain congestion traffic surge capture data; Perform weighted linear growth regression analysis on the congestion traffic surge capture data to obtain traffic surge weighted linear regression data; According to the traffic surge weighted linear regression data, the congestion traffic surge capture data is segmented to obtain the congestion traffic surge segmentation data; The Binary Increase Congestion algorithm is used to perform sudden segment response control on the congestion traffic surge segment data to obtain the congestion traffic segment response control data; The node interactive polling mechanism is controlled and optimized according to the segmented response control data of the congested traffic, and a node interactive polling optimization mechanism is obtained.

8. The multi-agent deep reinforcement learning method according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: normalizing the multi-agent resource competition strategy to obtain a multi-agent resource competition normalization strategy; Step S32: Performing hierarchical deep learning processing according to the multi-agent resource competition normalization strategy to obtain resource competition hierarchical deep learning data; Step S33: Based on the resource competition hierarchical deep learning data, multi-agent hierarchical convergence behavior coupling is performed to obtain hierarchical convergence behavior coupling data.

9. The multi-agent deep reinforcement learning method according to claim 8, characterized in that: Step S33 includes the following steps: Step S331: extracting the layered instability state among multiple agents from the resource competition layered deep learning data to obtain the learning behavior layered instability state data; Step S332: Calculate the instability coefficients between multiple levels of the learning behavior hierarchical instability state data to obtain the hierarchical instability coefficients; Step S333: performing multi-point instability initial value screening on the learning behavior hierarchical instability state data according to the hierarchical instability coefficient to obtain hierarchical instability multi-point screening data; Step S334: performing iterative calculation on the learning behavior stratified instability state data according to the stratified instability multi-point screening data to obtain stratified instability multi-point iterative data; Step S335: Perform multi-agent hierarchical convergence behavior coupling according to the hierarchical instability multi-point iteration data to obtain hierarchical convergence behavior coupling data.

10. A multi-agent deep reinforcement learning system, characterized in that: For executing the multi-agent based deep reinforcement learning method as claimed in claim 1, the multi-agent based deep reinforcement learning system comprises: The multi-agent interaction graph construction module is used to extract the operation logs of the multi-agents to obtain the multi-agent operation log data; perform multi-agent interaction structure analysis on the multi-agent operation log data to obtain the multi-agent interaction structure data; construct the multi-agent interaction graph according to the multi-agent interaction structure data to obtain the multi-agent interaction graph; The multi-agent resource competition strategy formulation module is used to label the multi-agent load nodes on the multi-agent interaction graph to obtain the multi-agent load node labeling data; calculate the resource release amount between different nodes on the multi-agent load node labeling data to obtain the load node resource release amount data; calculate the resource release limited interval between different load nodes on the load node resource release amount data to obtain the node resource release limited interval; formulate the multi-agent resource competition strategy according to the node resource release limited interval to obtain the multi-agent resource competition strategy; The multi-agent hierarchical convergence coupling module is used to perform hierarchical deep learning processing according to the multi-agent resource competition strategy to obtain resource competition hierarchical deep learning data; based on the resource competition hierarchical deep learning data, the multi-agent hierarchical convergence behavior coupling is performed to obtain hierarchical convergence behavior coupling data; The competition balance model construction module is used to construct a multi-agent resource competition balance model based on the multi-agent resource competition strategy and the hierarchical convergence behavior coupling data, obtain the multi-agent resource competition balance model, and send the multi-agent resource competition balance model to the cloud platform to perform multi-agent deep reinforcement learning.

Images