Collaborative decision-making system based on multi-agent reinforcement learning

Abstract

The invention provides a collaborative decision-making system based on multi-agent reinforcement learning, and the system comprises a global decision-making coordination module, an agent local decision-making module, a state information sharing module, a conflict detection and solution module, and a security and privacy protection module. Each agent realizes autonomous decision making and real-time learning in a local environment, the dependence of the system on centralized control is reduced, the communication burden of the system during data interaction is effectively reduced in a distributed manner, the agents can directly respond based on local data, and the data interaction efficiency is improved. The state information of each agent is collected and analyzed in real time through a global decision coordination module, and the overall strategy of the system is adjusted based on a reinforcement learning algorithm, so that global target implementation in a complex environment is ensured, and the limitation that a traditional system cannot be adaptively adjusted in a dynamic environment is effectively avoided through global optimization; and the global optimization of the multi-agent cooperative system on the basis of each agent target is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent collaborative decision-making, and in particular to a collaborative decision-making system based on multi-agent reinforcement learning. Background Art

[0002] In modern complex systems, the coordinated control and efficient decision-making of multiple intelligent agents are the core of system optimization, and the application scenarios cover intelligent transportation, logistics management, distributed energy management and other fields. Traditional multi-agent collaboration methods usually rely on preset rules and centralized control modes. However, with the complexity of application scenarios and the increase in dynamic change requirements, traditional methods have significant shortcomings.

[0003] When faced with high-frequency information interactions among multiple agents, centralized control methods often become inefficient due to limitations in computing resources and communication bandwidth, resulting in a decrease in system response speed;

[0004] Existing methods usually have difficulty in dynamically adjusting decisions according to real-time situations in dynamic environments, making it difficult to ensure global optimality. In addition, traditional systems use preset rules, and it is difficult for agents to learn and optimize their own strategies in real time according to environmental changes, resulting in insufficient response of the system in dynamic or uncertain environments. To this end, a collaborative decision-making system based on multi-agent reinforcement learning is proposed. Summary of the invention

[0005] In view of this, the present invention provides a collaborative decision-making system based on multi-agent reinforcement learning to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial option.

[0006] The technical solution of the present invention is implemented as follows: a collaborative decision-making system based on multi-agent reinforcement learning, including a global decision-making coordination module, an agent local decision-making module, a state information sharing module, a conflict detection and resolution module, and a security and privacy protection module:

[0007] The two-way communication connection between the global decision coordination module and the agent local decision modules, the global decision coordination module sends the global strategy and goal to each agent local decision module, and receives feedback from the local decision modules to dynamically adjust the global strategy;

[0008] A one-way data sharing connection between the local decision-making module of the agent and the state information sharing module. Each local decision-making module of the agent transmits environmental data and decision status to the state information sharing module in real time. The state information sharing module filters, compresses and adjusts the priority of the information before sharing it with other agents.

[0009] The trigger connection between the state information sharing module and the conflict detection and resolution module, when the state information sharing module detects information that causes resource or strategy conflicts between agents, triggers the conflict detection and resolution module to perform conflict analysis and processing on the relevant information;

[0010] The asynchronous request connection between the conflict detection and resolution module and the global decision coordination module, when the conflict detection and resolution module detects a conflict that cannot be resolved by a local strategy, it sends a request to the global decision coordination module, and the global decision coordination module performs global intervention and adjustment;

[0011] The security and privacy protection module is connected to the security control of all modules, and the data transmission between modules needs to be encrypted and privacy controlled by the security and privacy protection module.

[0012] Further preferably, the global decision coordination module includes a global strategy generator, a global state sensor and a conflict resolution unit, wherein:

[0013] The global strategy generator generates the optimal global strategy based on the deep reinforcement learning algorithm by constructing a global reward function Optimize system performance, where R i represents the reward value of the i-th agent, and N is the total number of agents;

[0014] The global state sensor obtains the overall state based on the local state information of each agent, using the weight formula S gl =α·S gl1 +β·S gl2 +…+γ·S glN Aggregate data to ensure global consistency;

[0015] The conflict resolution unit generates a resolution strategy when a serious conflict is detected in the system by dynamically adjusting the task priority or resource allocation of each agent.

[0016] Further preferably, the agent local decision module includes an environment sensor, a local strategy optimizer and a behavior execution unit, wherein:

[0017] The environmental sensor is used for local environmental perception of the intelligent agent and to obtain local state information in real time;

[0018] The local policy optimizer updates the policy based on a reinforcement learning algorithm, where the policy is optimized by the following formula:

[0019]

[0020] Among them, Q(s,a) represents the value of taking action a in state s, r is the reward, η is the learning rate, and γ is the discount factor;

[0021] The behavior execution unit executes decision operations based on the optimization strategy to ensure dynamic adaptation of the decision to the local environment.

[0022] Further preferably, the state information sharing module includes a data filtering unit, a data compression unit and a priority control unit, wherein:

[0023] The data filtering unit sets the data satisfying the condition I based on the set predetermined importance threshold T. data Information with a value greater than or equal to T is marked as high-priority information transmission;

[0024] The data compression unit uses the LZ algorithm to compress the data, and the compression ratio is controlled between 20% and 50%, thereby reducing the communication burden;

[0025] The priority control unit sets the priority according to the urgency and importance of the task, and shares the information with high priority first through the priority formula P=f(D,U), where D represents urgency and U represents importance.

[0026] Further preferably, the conflict detection and resolution module includes a conflict detection unit, a priority allocation unit and a dynamic adjustment unit, wherein:

[0027] The conflict detection unit detects conflicts by position and resource occupancy. When the positions of agent i and agent j meet d ij ≤d th , and when resource requirements overlap, it is determined to be a conflict;

[0028] The priority allocation unit allocates resource priorities based on task importance and urgency according to the formula P i =w1·E i +w2·T i Calculate the priority, where E i Indicates the level of task importance, T i Indicates urgency, w1 and w2 are preset weights;

[0029] The dynamic adjustment unit adjusts the strategy in real time when a conflict occurs, and reallocates the priority and resources of each intelligent agent.

[0030] Further preferably, the security and privacy protection module includes an encryption communication unit, a differential privacy controller and a data access control unit, wherein:

[0031] The encryption communication unit uses AES and RSA encryption algorithms to protect the transmitted data, where AES is used for symmetric data encryption and RSA is used to encrypt and decrypt AES keys. The differential privacy controller protects data privacy by adding noise that conforms to the Laplace distribution, and its noise parameter ∈ ensures reasonable control of the data privacy budget. The data access control unit manages the access rights of the intelligent agent based on the access control matrix to limit the access rights of sensitive data.

[0032] Further preferably, the global decision coordination module communicates with the conflict detection and resolution module via an asynchronous request connection, and triggers global policy intervention when the conflict cannot be resolved by local policies.

[0033] Further preferably, the global state sensor obtains the overall state of the system in real time based on the agent feedback information, and uses the weight synthesis formula S gl =α·S gl1 +β·S gl2 +α·S glN The final system state is obtained for dynamic adjustment of multiple task loads and resource allocation.

[0034] Further preferably, the status information sharing module gives priority to transmitting emergency information when sharing information, and ensures that key information is transmitted first under resource-constrained conditions through a data compression unit and a priority control unit.

[0035] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0036] 1. The present invention adopts a distributed multi-agent structure, so that each agent can realize autonomous decision-making and real-time learning in the local environment, reduce the system's dependence on centralized control, and effectively reduce the communication burden of the system during data interaction through distribution, so that the agent can respond directly based on local data, avoiding the problem of decreased response speed caused by centralized communication congestion.

[0037] 2. The present invention collects and analyzes the status information of each intelligent agent in real time through the global decision-making coordination module, and adjusts the overall strategy of the system based on the reinforcement learning algorithm to ensure the realization of the global goal in a complex environment. Through global optimization, it effectively avoids the limitation of traditional systems that cannot be adaptively adjusted in a dynamic environment, and ensures the global optimization of the multi-agent collaborative system based on the goals of each intelligent agent.

[0038] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0041] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0042] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0043] Example

[0044] like Figure 1 As shown, this embodiment discloses a collaborative decision-making system based on multi-agent reinforcement learning, including a global decision-making coordination module, an agent local decision-making module, a state information sharing module, a conflict detection and resolution module, and a security and privacy protection module:

[0045] The global decision-making coordination module is responsible for the overall strategy and goal planning of the system, achieving global optimal performance through deep reinforcement learning, including the global strategy generator, global state sensor and conflict resolution unit;

[0046] Use deep reinforcement learning algorithms to generate optimal strategies by constructing a global reward function To achieve the global reward function R global is the weighted sum of all agent rewards, where R i is the reward of the ith agent, N is the total number of agents in the system, and the reward function reflects the overall goal of the entire system. Deep reinforcement learning optimizes the strategy through a continuous training process, so that the agent maximizes the total reward of the system when performing tasks, thereby improving the overall efficiency of the system. Secondly, the global state sensor is responsible for collecting and integrating the local state information of each agent, using formula S gl =α·S gl1 +β·S gl2 +…+γ·S glN The overall state of the computing system, where S glis the local state of the i-th agent, and the weights α, β, and γ are the weight coefficients of the corresponding agents. The role of the global state sensor is to obtain and process the information of each agent in real time;

[0047] The conflict resolution unit is used to provide a global solution when resource or task conflicts occur between agents. It dynamically adjusts the task priority or resource allocation plan of the agent to ensure that the overall performance of the system is not seriously affected. When a conflict occurs, resources are reallocated or task priorities are adjusted according to the task urgency, priority and resource requirements of each agent to achieve the global optimal resource configuration. If the tasks of two agents conflict and cannot be resolved through local adjustments, the conflict resolution unit will prioritize and schedule tasks according to the global strategy to ensure that key tasks are completed first and avoid system performance degradation caused by conflicts.

[0048] The agent local decision-making module relies on local environment perception and reinforcement learning algorithms to optimize the decision-making process. It consists of an environment sensor, a local strategy optimizer, and a behavior execution unit.

[0049] The environmental sensor is responsible for sensing the local environment of the agent and collecting data on the environment status, the actions of surrounding agents, resource status, and task progress in real time. The environmental sensor obtains real-time information through sensors, network interfaces, or other hardware devices, and transmits it to the local policy optimizer to ensure that the decision-making process is based on the latest environmental data; the local policy optimizer uses the reinforcement learning algorithm to continuously optimize the decision-making strategy of the agent. The local policy optimizer uses the following formula to update the strategy through interaction with the environment:

[0050]

[0051] in:

[0052] Q(s,a) represents the value of taking action a in state s; r is the reward obtained after taking action a; η is the learning rate, which is used to control the step size of the learning process; γ is the discount factor, which represents the discount rate of future rewards;

[0053] This formula gradually adjusts the decision-making strategy by comparing the current strategy with the expected future rewards at each step, so that the behavior of the agent tends to maximize the long-term rewards. The local strategy optimizer realizes the gradual optimization of the strategy by continuously updating the Q value, ensuring that the agent can make the best decision in the local environment. Finally, according to the optimal action calculated by the local strategy optimizer, the behavior execution unit converts the decision of the agent into actual action. The execution unit ensures that the agent can act effectively according to the optimized strategy to achieve the established task objectives.

[0054] Through the collaborative work of environmental sensors, local policy optimizers and behavior execution units, the agent's local decision-making module can perceive the environment in real time, optimize decision-making strategies, and execute decisions based on the optimized strategies.

[0055] The state information sharing module optimizes the information flow in the multi-agent system, and ensures that key information can be transmitted quickly and accurately between agents through an efficient data sharing mechanism. It consists of a data filtering unit, a data compression unit and a priority control unit.

[0056] According to the set importance threshold T, key information is identified and extracted from the local state data of the agent, and these high-priority data are transmitted to other agents. The importance threshold T is dynamically adjusted according to actual needs. For example, some data plays a decisive role in the realization of the global goal of the system, and will be marked as high-priority information and transmitted first; then, the data compression unit is responsible for compressing the state information to reduce bandwidth occupancy and improve transmission efficiency. The LZ algorithm is used to compress the data. The LZ algorithm reduces the redundancy of the data by finding repeated patterns in the data, thereby effectively reducing the data volume. In practical applications, the compression ratio is controlled between 20% and 50%, which can not only reduce the burden of data transmission, but also ensure the accuracy and integrity of the data;

[0057] The priority control unit adjusts the priority of information transmission according to the urgency and importance of the task, evaluates the urgency of different agent tasks and their impact on global decision-making, and adjusts the data transmission order based on this information. The higher the priority of the task, the higher the priority of the transmission, ensuring that key data can be delivered to the relevant agents as soon as possible.

[0058] The conflict detection and resolution module is the part of the multi-agent system used to manage and coordinate resources, ensuring that the agents avoid conflicts as much as possible during the collaboration process. It includes a conflict detection unit, a priority allocation unit, and a dynamic adjustment unit.

[0059] The conflict detection unit is used to analyze the behavior data of each agent in real time to identify potential conflict situations. It determines conflicts by monitoring the location and resource occupancy of the agents. Especially in an environment with limited resources, conflicts are more likely to occur when the behavior and location of the agents are close. ij Satisfy ij ≤d thWhen there is overlap in their resource demands, the system will determine that there is a risk of conflict between the two agents. This conflict judgment is used in various dynamic environments and can timely capture resource competition or behavioral interference between agents, providing an accurate basis for subsequent conflict resolution steps. After detecting a conflict, the priority allocation unit allocates resources according to the priority of the task to reduce the impact of the conflict on task execution. The priority allocation unit uses a pre-set weight formula P i =w1·E i +w2·T i Calculate the priority of each agent, where E i Indicates the importance level of the task, T i It represents the urgency of the task, w1 and w2 are the weight factors preset by the system, which are used to adjust the proportion of importance and urgency. Then the dynamic adjustment unit adjusts the strategy and behavior of the agent in real time based on the results of conflict detection to reduce the impact of the conflict on the system. After identifying the high-priority agent, it will adjust the strategy of other agents in the conflict so as to optimize the overall resource utilization without interfering with the high-priority task, thereby ensuring the coordination and response speed of the system.

[0060] The security and privacy protection module is used to ensure the security and privacy protection of system data transmission and sharing, including: encryption communication unit, which protects the security of data during transmission based on AES and RSA encryption algorithms. AES is used for symmetric data encryption, and RSA is used to encrypt and decrypt AES keys;

[0061] The differential privacy controller protects the privacy of sensitive data by adding noise that conforms to the Laplace distribution. The noise parameter ∈ is set reasonably to ensure the control of the privacy budget.

[0062] The data access control unit restricts data access according to the agent's access rights to prevent unauthorized access.

[0063] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A collaborative decision-making system based on multi-agent reinforcement learning, characterized in that: It includes global decision-making coordination module, agent local decision-making module, state information sharing module, conflict detection and resolution module and security and privacy protection module: The two-way communication connection between the global decision coordination module and the agent local decision modules, the global decision coordination module sends the global strategy and goal to each agent local decision module, and receives feedback from the local decision modules to dynamically adjust the global strategy; The one-way data sharing connection between the local decision-making module of the agent and the state information sharing module, each local decision-making module of the agent transmits environmental data and decision status to the state information sharing module in real time, and the state information sharing module filters, compresses and adjusts the priority of the information before sharing it with other agents; The trigger connection between the state information sharing module and the conflict detection and resolution module, when the state information sharing module detects information that causes resource or strategy conflicts between agents, triggers the conflict detection and resolution module to perform conflict analysis and processing on the relevant information; The asynchronous request connection between the conflict detection and resolution module and the global decision coordination module, when the conflict detection and resolution module detects a conflict that cannot be resolved by a local strategy, it sends a request to the global decision coordination module, and the global decision coordination module performs global intervention and adjustment; The security and privacy protection module is connected to the security control of all modules, and the data transmission between modules needs to be encrypted and privacy controlled by the security and privacy protection module.

2. The collaborative decision-making system based on multi-agent reinforcement learning according to claim 1, characterized in that: The global decision-making coordination module includes a global strategy generator, a global state sensor and a conflict resolution unit, wherein: The global strategy generator generates the optimal global strategy based on the deep reinforcement learning algorithm by constructing a global reward function Optimize system performance, where R i represents the reward value of the i-th agent, and N is the total number of agents; The global state sensor obtains the overall state based on the local state information of each agent, using the weight formula S gl =α·S gl1 +β·S gl2 +…+γ·S glN Aggregate data to ensure global consistency; The conflict resolution unit generates a resolution strategy when a serious conflict is detected in the system by dynamically adjusting the task priority or resource allocation of each agent.

3. The collaborative decision-making system based on multi-agent reinforcement learning according to claim 1, characterized in that: The agent local decision module includes an environment sensor, a local strategy optimizer and a behavior execution unit, wherein: The environmental sensor is used for local environmental perception of the intelligent agent and to obtain local state information in real time; The local policy optimizer updates the policy based on a reinforcement learning algorithm, where the policy is optimized by the following formula: Among them, Q(s,a) represents the value of taking action a in state s, r is the reward, η is the learning rate, and γ is the discount factor; The behavior execution unit executes decision operations based on the optimization strategy to ensure dynamic adaptation of the decision to the local environment.

4. The collaborative decision-making system based on multi-agent reinforcement learning according to claim 1, characterized in that: The state information sharing module includes a data filtering unit, a data compression unit and a priority control unit, wherein: The data filtering unit sets the data satisfying the condition I based on the set predetermined importance threshold T. data Information with a value greater than or equal to T is marked as high-priority information transmission; The data compression unit uses the LZ algorithm to compress the data, and the compression ratio is controlled between 20% and 50%, thereby reducing the communication burden; The priority control unit sets the priority according to the urgency and importance of the task, and shares the information with high priority first through the priority formula P=f(D,U), where D represents urgency and U represents importance.

5. The collaborative decision-making system based on multi-agent reinforcement learning according to claim 1, characterized in that: The conflict detection and resolution module includes a conflict detection unit, a priority allocation unit and a dynamic adjustment unit, wherein: The conflict detection unit detects conflicts by position and resource occupancy. When the positions of agent i and agent j meet d ij ≤d th , and when resource requirements overlap, it is determined to be a conflict; The priority allocation unit allocates resource priorities based on task importance and urgency, according to the formula P i =w1·E i +w2·T i Calculate the priority, where E i Indicates the level of task importance, T i Indicates urgency, w1 and w2 are preset weights; The dynamic adjustment unit adjusts the strategy in real time when a conflict occurs, and reallocates the priority and resources of each intelligent agent.

6. The collaborative decision-making system based on multi-agent reinforcement learning according to claim 1, characterized in that: The security and privacy protection module includes an encryption communication unit, a differential privacy controller and a data access control unit, wherein: The encryption communication unit uses AES and RSA encryption algorithms to protect the transmitted data, where AES is used for symmetric data encryption and RSA is used to encrypt and decrypt AES keys. The differential privacy controller protects data privacy by adding noise that conforms to the Laplace distribution, and its noise parameter ∈ ensures reasonable control of the data privacy budget. The data access control unit manages the access rights of the intelligent agent based on the access control matrix to limit the access rights of sensitive data.

7. The method for preparing a collaborative decision-making system based on multi-agent reinforcement learning according to claim 1, characterized in that: The global decision coordination module communicates with the conflict detection and resolution module through an asynchronous request connection, and triggers global policy intervention when the conflict cannot be resolved through local policies.

8. The method for preparing a collaborative decision-making system based on multi-agent reinforcement learning according to claim 1, characterized in that: The global state sensor obtains the overall state of the system in real time based on the agent feedback information, and uses the weighted synthesis formula S gl =α·S gl1 +β·S gl2 +γ·S glN The final system state is obtained for dynamic adjustment of multiple task loads and resource allocation.

9. The method for preparing a collaborative decision-making system based on multi-agent reinforcement learning according to claim 1, characterized in that: The state information sharing module gives priority to transmitting emergency information during information sharing, and ensures that key information is transmitted first under resource-constrained conditions through a data compression unit and a priority control unit.

Images