Distributed deep Q learning method for multi-agent system path search

Abstract

A distributed deep Q learning method for path searching of a multi-agent system comprises the following steps: 1) constructing a deep Q network of each agent, and forming a D-DMAQL model corresponding to the multi-agent system in combination with a UCB strategy for calculating the Q value of each agent at the current moment and the maximum Q value of each agent at the next moment; 2) establishing a D-DMAQL training module which is used for training a D-DMAQL model; 3) establishing a reasoning decision module used for calculating the decision value of each agent according to the Q value of each agent at the current moment and the corresponding environment state and action; 4) establishing a global optimizer for calculating a global loss function of the D-DMAQL model according to the Q value of each agent at each moment and the corresponding environment state, action, reward and decision value; and 5) performing path searching of the multi-agent system by utilizing the trained D-DMAQL model and combining a global loss function.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a distributed deep Q learning method for path finding in a multi-agent system. Background Art

[0002] A multi-agent system is a system composed of multiple interacting agents, each of which has a certain degree of autonomy and decision-making ability. It perceives the environment, interacts with other agents, makes decisions, and performs actions. In a multi-agent system, agents need to work together and take actions according to their own goals and strategies to complete specific tasks, such as path finding, resource allocation, etc. This multi-agent system is widely needed in many practical applications, such as unmanned vehicle fleets, drone swarms, smart grids, etc.

[0003] Currently, the technologies for path finding in multi-agent systems are mainly divided into two categories: one is methods based on traditional algorithms, such as graph search algorithms, heuristic search algorithms, etc.; the other is methods based on deep learning and reinforcement learning, such as establishing a deep learning model to extract spatial features to build an environment, using reinforcement learning algorithms to learn the decision-making strategies of agents to optimize global goals, setting up independent Q tables for each agent to learn and make decisions, and having a central decision maker coordinate the cooperation strategies between agents.

[0004] In current practical applications, it is found that the above two methods for path finding in multi-agent systems have the following shortcomings:

[0005] ① High computational complexity and poor adaptability to large-scale environments:

[0006] Traditional algorithm-based path finding methods, such as A* search, require a lot of preprocessing time and storage space, and as the scale of the environment increases, the scale of the graph will also grow exponentially, which greatly increases the computational complexity. This limits their efficiency in large-scale environments. After that, it is necessary to use iterative optimization methods to continuously improve the path, which also requires a lot of computing resources. In a multi-agent environment, this method needs to consider issues such as collaboration and stratification when designing the algorithm, which further increases the amount of computation;

[0007] ② The training samples are inefficient and difficult to adapt to complex environments:

[0008] The deep learning-based path finding method relies on a large number of sample training. This method encounters difficulties in collecting enough training samples, especially in complex dynamic environments. In addition, the method has low sample efficiency, which directly affects the time cost of model training and is not conducive to real-time decision-making and planning.

[0009] ③ It is easy to fall into local optimality and difficult to plan the global path:

[0010] Reinforcement learning-based methods are prone to fall into local optimal solutions in complex environments and fail to find the global optimal strategy. In a multi-agent system, due to the inconsistency of the goals of individual agents, it is difficult to collaboratively plan the global optimal path;

[0011] ④Difficulties in coordination of distributed multi-agent systems:

[0012] In current technologies, there is a lack of effective information exchange and coordination mechanisms between agents. Each agent makes independent decisions based on local information, which easily leads to conflicts. In addition, agents cannot use the information of other agents to help them learn, which limits their ability to improve sample efficiency.

[0013] ⑤ Weak ability to cope with complex dynamic environments:

[0014] Existing technologies lack the ability to quickly adapt to dynamic changes in the environment. They rely on pre-modeling and extensive training, and are not very effective in optimizing long-term planning in frequently changing environments. In addition, these technologies have poor robustness, are easily disturbed by environmental noise, and have poor real-time performance, making it difficult to respond quickly to frequent changes in complex dynamic environments.

[0015] Due to the existence of the above-mentioned deficiencies, the accuracy, reliability and efficiency of existing technologies in dealing with path finding problems in large-scale, complex and dynamic environments are far from meeting actual application requirements. Summary of the invention

[0016] The purpose of the present invention is to address the corresponding deficiencies in the prior art and to provide a distributed deep Q-learning method for path finding in a multi-agent system, combining the advantages of deep learning and reinforcement learning, allowing each agent to perform distributed computing and participate in global optimization, so that the accuracy, reliability and efficiency can meet the requirements of practical applications when dealing with path finding problems in large-scale, complex and dynamic environments.

[0017] The object of the present invention is to adopt the following scheme to achieve: a distributed deep Q learning method for path finding in a multi-agent system, comprising the following steps:

[0018] 1) Construct a deep Q network for each agent and combine it with the UCB strategy to form a D-DMAQL model corresponding to the multi-agent system, which is used to calculate the Q value of each agent at the current moment and the maximum Q value at the next moment;

[0019] 2) Establish a D-DMAQL training module for training the D-DMAQL model;

[0020] 3) Establish a reasoning decision module to calculate the decision value of each agent based on the Q value of each agent at the current moment, as well as the corresponding environmental state and action;

[0021] 4) Establish a global optimizer to calculate the global loss function of the D-DMAQL model based on the Q value of each agent at each moment, as well as the corresponding environment state, action, reward, and decision value;

[0022] 5) Use the trained D-DMAQL model and the global loss function to perform path finding in the multi-agent system.

[0023] Preferably, the specific establishment process of the D-DMAQL model also includes:

[0024] 1-1) Define the type of environment state in which the multi-agent system is located;

[0025] 1-2) Define the action type of each agent in the multi-agent system to form a legal action set of the multi-agent system;

[0026] 1-3) Setting a state transfer function to use the current state of the agent to determine the state at the next moment;

[0027] 1-4) Set a reward function to guide each agent to complete the task.

[0028] Preferably, the environmental state type includes the current position of each intelligent agent, the target position of each intelligent agent at the current moment, the position vector of the multi-intelligent system at the current moment, the actual speed of each intelligent agent at the current moment, the actual acceleration of each intelligent agent at the current moment, and the accumulated energy consumption of each intelligent agent.

[0029] Preferably, the action type includes a process in which each agent moves from a current vertex to an adjacent vertex under a given environmental state.

[0030] Preferably, the state transfer function is as follows:

[0031] s i,t+1 =f(s i,t , a i,t )

[0032] In the formula, s i,t+1 is the state of agent i at time t+1, s i,t is the state of agent i at time t, a i,t Agent i at time t, state s i,t The action taken when

[0033] Preferably, the reward function is as follows:

[0034] ri,t =g(s i,t , a i,t ,s i,t+1 )

[0035] In the formula, r i,t is the reward obtained by agent i at time t, s i,t+1 is the state of agent i at time t+1, s i,t is the state of agent i at time t, a i,t Agent i at time t, state s i,t The action taken when

[0036] Preferably, the deep Q network is a convolutional neural network comprising an input layer, multiple hidden layers, and an output layer.

[0037] Preferably, the global loss function of the D-DMAQL model is as follows:

[0038]

[0039] Where L(θ) is the global loss function of the D-DMAQL model, which is used to measure the overall error of all agents in the current training step. By minimizing this function, the network parameters of all agents can be optimized to make their predicted Q values ​​closer to the target Q values; θ is the set of Q network parameters of all agents; N is the number of agents; Σ is the sum of the losses of all agents, indicating that the errors of all agents need to be considered and minimized; i is the agent index, indicating that the loss of the i-th agent is currently being calculated; t is the time step index, indicating that the loss at time t is currently being calculated; r i,t is the reward obtained by agent i at time t, which means the immediate feedback obtained by the agent after taking an action in the current state; γ is the discount factor, which (0<γ≤1) is used to balance the importance of immediate rewards and future rewards. The closer γ is to 1, the more important future rewards are; s i,t is the state of agent i at time t; a i,t Agent i at time t, state s i,t Action taken when i,t+1 is the state of agent i at time t+1, that is, in state s i,t Take action a i,t The new state after i,t+1 Agent i at time t+1, state s i,t+1 List all possible actions to take; is the parameter of the Q network of agent i at time t, which is used to estimate the current state-action pair (s i,t , a i,t)’s Q value; t is the parameter of the target Q network at time t, which is used to estimate the target Q value, keep it relatively stable, and avoid fluctuations during training. All agents share the same target Q network; Q(s,a;θ) is the Q value estimated when taking action a in state s using a Q network with parameter θ.

[0040] The advantages of the present invention are as follows:

[0041] ① Richness of environment state and action definitions

[0042] Compared with the traditional definition of the agent's environment state and action, the present invention provides richer information by including multiple factors such as the agent's position, target position, speed, acceleration and energy consumption. This innovative definition gives the agent a high degree of flexibility in the map, which helps to plan the path more effectively, surpassing the previous single or limited parameter definition.

[0043] ② Path finding based on deep Q network

[0044] This invention proposes a path finding method of deep Q network, which has the ability to handle high-dimensional and continuous state problems, and improves learning efficiency and search accuracy compared to traditional algorithms that may be used in other patents. This method shows significant superiority in path finding tasks in large-scale, complex and dynamic environments.

[0045] ③Distributed agent decision-making mechanism

[0046] By combining the deep Q network and the Upper Confidence Bound (UCB) strategy, the present invention provides a new distributed agent decision-making mechanism. Compared with the centralized decision-making methods in other patents, this mechanism enhances the exploration space, balances exploration and utilization, improves decision-making efficiency and accuracy, and provides a more advanced solution for distributed agent decision-making.

[0047] ④Collaborative working mode of global optimization and distributed computing

[0048] Compared with traditional local optimization or centralized computing methods, the global optimization and distributed computing design of the present invention ensures computing efficiency while ensuring the effectiveness of decision-making. The innovation of this design lies in the collaborative mechanism between the intelligent agent and the global optimizer, which has important theoretical value and practical significance, and has demonstrated strong performance in practice.

[0049] ⑤ Efficiency and stability of training modules

[0050] The present invention significantly improves the efficiency and stability of training by introducing technologies such as experience replay, fixed Q target, gradient clipping and learning rate decay, and shows obvious advantages over other patents that may only use basic training algorithms.

[0051] ⑥ Intelligent pruning strategy and coordination of reasoning modules

[0052] Compared with other multi-agent reinforcement learning methods, this paper introduces a unique intelligent pruning strategy and a dynamic distributed coordination strategy to reduce the search space and improve computational efficiency. These original designs make the system efficient and robust in dealing with large-scale multi-agent reinforcement learning problems, providing a new direction and thinking model for related research fields.

[0053] In summary, the present invention demonstrates obvious advantages and innovations in many aspects compared with other patents and methods, and shows excellent performance and broad application prospects in terms of environment state and action definition, path finding, distributed decision-making, global and distributed collaboration, training stability and reasoning efficiency.

[0054] Glossary

[0055] Deep learning: By building deep neural networks, it can process high-dimensional and complex data and automatically learn useful features from the data.

[0056] Reinforcement learning: By allowing the agent to interact with the environment, it learns a strategy that maximizes a certain long-term reward and can handle sequential decision-making problems.

[0057] Graph search algorithm: In the method using traditional algorithms, the environment first needs to be converted into a graph model, where nodes represent possible locations and edges represent paths that the agent can walk. Graph search algorithms such as breadth-first search, depth-first search, and A* search are further applied to this graph model to find the path from the starting point to the target. For example, in the A* search algorithm, it combines heuristic functions and path costs to find the shortest path from the starting point to the target point. For each node, the F value calculated by the A* search algorithm is equal to the sum of the actual distance G from the starting point to the node and the estimated distance H from the node to the target point.

[0058] Heuristic search algorithm: After finding a preliminary feasible path, it is necessary to use heuristic search algorithms, such as best-first search, iterative deepening A* search, etc., to further optimize the path. Taking iterative deepening A* search as an example, this algorithm effectively combines the memory efficiency of depth-first search and the integrity of breadth-first search by limiting the search depth and adopting a depth-first search strategy. Whenever the search fails, the search depth limit is increased until a solution is found.

[0059] Q-value: In reinforcement learning, Q-value is a very core concept used to represent the total reward expected from taking a specific action in a given state. Q-value is basically a way to measure the "action value", that is, how much value can be brought by taking an action in the current state.

[0060] Q-table: The Q-table is a wisdom book for reinforcement learning. It records in a tabular form the expected long-term benefits (Q-values) of different actions taken by the agent when facing various states. Each row of this book represents a state that the agent may encounter, and each column represents an action that the agent can choose in that state. The Q-value stored in each cell guides the agent on which action to choose in order to obtain the greatest long-term reward when facing a specific state. By constantly exploring the environment, trying actions, and updating Q-values, the agent can eventually learn to make optimal decisions in various complex environments and maximize its goals under the guidance of this wisdom book. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a structural diagram of the path finding system of the multi-agent system of the present invention. DETAILED DESCRIPTION

[0062] A distributed deep Q-learning method for path finding in a multi-agent system, comprising the following steps:

[0063] 1) Construct a deep Q network for each agent and combine it with the UCB strategy to form a D-DMAQL model corresponding to the multi-agent system, which is used to calculate the Q value of each agent at the current moment and the maximum Q value at the next moment;

[0064] In this way, each intelligent agent can use the local server to make fast decisions through a lightweight deep Q network, and regularly synchronize parameters with the central server for global optimization, taking into account both efficiency and optimization effect.

[0065] Typically, the deep Q network is a convolutional neural network that includes an input layer, multiple hidden layers, and an output layer.

[0066] In fact, the deep Q network can also use a variety of neural network structures as the backbone network, such as convolutional neural network, recurrent neural network, fully connected neural network, etc. The specific choice of network structure can be determined based on factors such as the size of the environmental state and action space in the actual application scenario, the complexity of the features, and whether time series information needs to be processed.

[0067] Generally, the specific establishment process of the D-DMAQL model also includes:

[0068] 1-1) Define the type of environmental state in which the multi-intelligent system is located, which includes the current position of each intelligent agent, the target position of each intelligent agent at the current moment, the position vector of the multi-intelligent system at the current moment, the actual speed of each intelligent agent at the current moment, the actual acceleration of each intelligent agent at the current moment, and the accumulated energy consumption of each intelligent agent.

[0069] 1-2) Define the action type of each agent in the multi-intelligence system to form a legal action set of the multi-intelligence system. The action type includes the process of each agent moving from the current vertex to an adjacent vertex under a given environmental state.

[0070] 1-3) Set a state transfer function to use the current state of the agent to determine the state at the next moment. The state transfer function is as follows:

[0071] s i,t+1 =f(s i,t , a i,t )

[0072] In the formula, s i,t+1 is the state of agent i at time t+1, s i,t is the state of agent i at time t, a i,t Agent i at time t, state s i,t The action taken when

[0073] 1-4) Set a reward function to guide each agent to complete the task. The reward function is as follows:

[0074] r i,t =g(s i,t , a i,t ,s i,t+1 )

[0075] In the formula, r i,t is the reward obtained by agent i at time t, s i,t+1 is the state of agent i at time t+1, s i,t is the state of agent i at time t, a i,t Agent i at time t, state s i,t The action taken when

[0076] 2) Establish a D-DMAQL training module for training the D-DMAQL model;

[0077] Generally speaking, each agent should establish a D-DMAQL training module locally, use local data for training, and regularly interact with the global optimizer to achieve a combination of distributed training and global optimization.

[0078] 3) Establish a reasoning decision module to calculate the decision value of each agent based on the Q value of each agent at the current moment, as well as the corresponding environmental state and action;

[0079] It is recommended that the inference decision module be deployed in the same location as the deep Q network, which can reduce communication overhead and improve decision-making efficiency. That is, when local deployment or hybrid deployment is adopted, the inference decision module is on the local server, and when central deployment is adopted, the inference decision module is on the central server.

[0080] 4) Establish a global optimizer to calculate the global loss function of the D-DMAQL model according to the Q value of each agent at each moment, as well as the corresponding environment state, action, reward, and decision value. The global loss function of the D-DMAQL model is as follows:

[0081]

[0082] Where L(θ) is the global loss function of the D-DMAQL model; θ is the set of all agent Q network parameters; N is the number of agents; i is the agent index; t is the time step index; r i,t is the reward obtained by agent i at time t; γ is the discount factor; s i,t is the state of agent i at time t; a i,t Agent i at time t, state s i,t Action taken when i,t+1 is the state of agent i at time t+1; a′ i,t+1 Agent i at time t+1, state s i,t+1 List all possible actions to take; is the parameter of the Q network of agent i at time t; θ′ t is the parameter of the target Q network at time t.

[0083] However, the global optimizer is best deployed on a central server to efficiently collect information from all agents and perform global loss function calculations and parameter updates.

[0084] 5) Use the trained D-DMAQL model and the global loss function to perform path finding in the multi-agent system.

[0085] According to the above steps, a path finding system of a multi-agent system is established, and an example is as follows:

[0086] 1. Environment state and action definition

[0087] 1.1 Definition of environmental status

[0088] Suppose there are N agents in the environment, where the i-th agent is represented by ai , i=1,2,

[0089] ..., N. The environment can be abstracted as an undirected graph G = (V, E), where V represents a vertex set and E represents an edge set. The undirected graph G = (V, E) can represent the map information that the agent can travel.

[0090] At time t, agent a i Environmental status i,t It is composed of the following elements:

[0091] Agent a i At time t, there is a vertex v i,t ∈V;

[0092] Agent a i The target position is a vertex g i ∈V;

[0093] The positions of all agents at time t, represented by a vector L t =[v 1,t , v 2,t , ...,

[0094] v N,t ];

[0095] Agent a i The velocity vel at time t i,t ;

[0096] Agent a i The acceleration acc at time t i,t ;

[0097] Agent a i Cumulative energy consumption i,t , which is related to its travel distance;

[0098] Then agent a i Environmental status i,t It can be expressed as: i,t =(v i,t , g i ,

[0099] L t ,vel i,t ,acc i,t , e i,t )

[0100] Through such environmental state representation, the agent can obtain sufficient environmental information, which is conducive to its decision-making and path planning. In particular, the location information of other agents can help the agent predict possible conflicts and adjust the path in advance. Information such as speed and acceleration can be used to determine energy consumption and perform speed planning.

[0101] 1.2 Action Definition

[0102] In a given environment state s i,t Under the condition of i The set of actions that can be performed at time t is A i,t Each action of the agent is represented as moving from the current vertex to another adjacent vertex.

[0103] That is, for action a i,t ∈A i,t , there are: a i,t :v i,t →v′, where v i,t For agent a i At time t, v′ is the same as v i,t Adjacent vertices.

[0104] Through such action definitions, the agent can move flexibly in the graph to reach the target position. i,t The determination of needs to consider environmental constraints, such as two agents cannot occupy a vertex at the same time (to achieve conflict avoidance), agents can only move to adjacent vertices, etc.

[0105] 1.3 State Transition Function

[0106] Execute action a i,t Will change the environment from the current state s i,t Transition to next state s i,t+1 The state transfer function is defined as: i,t+1 =f(s i,t , a i,t ) Among them, the function f determines the state of the agent at the next moment based on the current state and the action performed.

[0107] Specifically, perform action a i,t After that, the position of the agent is determined by v i,t becomes v′, and the velocity and acceleration also change according to the action. The states of other agents change independently according to the actions they perform.

[0108] 1.4 Reward Function

[0109] Define the immediate reward r i,t Indicates execution of action a i,t Feedback after.

[0110] The reward function is defined as: i,t =g(s i,t , a i,t ,s i,t+1 )

[0111] The reward function g can be based on the current state s i,t , Action performed a i,t and the new state s reached i,t+1 To determine the immediate reward.

[0112] Typically, rewards are designed to guide the agent to complete a task.

[0113] For example, you can set the following rewards:

[0114] When the agent reaches the goal position, it receives a large positive reward.

[0115] Get a small positive reward for each step to encourage continuous movement towards the goal;

[0116] If a conflict occurs, a negative reward is obtained;

[0117] Consuming too much energy will also result in negative rewards;

[0118] By properly designing the reward function, the agent can learn strategies to avoid conflicts, save energy, and ultimately achieve its goals during reinforcement learning. The above definitions of environmental states and actions can provide the agent with complete environmental information, allowing it to make reasonable decisions.

[0119] 2. Deep Q Network Construction

[0120] This embodiment introduces a deep Q network (DQN) in the agent path finding task as an approximation of the agent Q function. For each agent a i (i=1,2,...,N) to establish an independent deep Q network, which uses a convolutional neural network (CNN) as its basic structure. The Q function is an important concept in reinforcement learning. It means the expected value of the total future reward that can be obtained by performing an action a in a certain state s. In this embodiment, agent a i In status i,t Next, perform action a i,t The Q value is expressed as Q(s i,t ,a i,t ). Since the state and action space may be very large, it is unrealistic to directly store the Q value and update it. Therefore, this embodiment uses a deep learning model to approximate the Q function.

[0121] Each agent a iThe deep Q network, denoted as DQNi, is a convolutional neural network consisting of an input layer, multiple hidden layers, and an output layer. The input of the network is the environment state s i,t and action a i,t , the output is the corresponding Q value. The environment state s i,t and action a i,t It is converted into a continuous vector form through the embedding layer as the input of the network. The structure of the network is described in detail below:

[0122] 2.1 Input Layer

[0123] Let the input state be s i,t =(v i,t ,g i ,L t ,vel i,t ,acc i,t ,e i,t ), the input action is a i,t , needs to be processed by an embedding layer. The core task of the embedding layer is to map discrete symbols into real vectors, providing input for subsequent networks to perform calculations.

[0124] For any vertex v, a d-dimensional vector can be obtained through the embedding layer, expressed as

[0125] emb(v). This means that agent a i At time t, the position v i,t 、Target position g i , and the positions L of all agents at time t t can be mapped into a continuous vector form. For example, the position v i,t Can be embedded as emb(v i,t ), which is a d-dimensional vector, where d is the default embedding dimension. Action a i,t It will also be embedded as a d-dimensional vector, that is, emb(a i,t ). Environmental status s i,t and action a i,t The embedding representations are concatenated together to generate the network’s input vector x i,t , whose dimension is n, where n is the sum of the dimensions of the embedded state and action vectors. This design provides the network input with rich information, laying the foundation for subsequent feature extraction and learning.

[0126] 2.2 Hidden Layer

[0127] The design of the hidden layer makes full use of the characteristics of convolutional neural networks, and extracts and selects features through multiple alternating convolutional layers and pooling layers. Each convolutional layer can be regarded as a feature detector, which performs convolution operations on the input feature map in a sliding window manner, so that the model can obtain the local features and spatial relationships of the input data.

[0128] For the lth convolutional layer, let the input feature map be X l-1 , whose size is c l-1 ×h l-1 × l-1 , where c l-1 is the number of feature channels, h l-1 is the height, w l-1 is the width. Convolution kernel W l The size of c l ×

[0129] c l-1 ×h×w, where c l is the number of output feature channels, c l-1 is the number of input feature channels, h is the height of the convolution kernel, and w is the width of the convolution kernel. The convolution kernel can be understood as a filter that extracts local features from the input feature map. Each convolution kernel is multiplied by the corresponding element of the area it covers, and then the bias term b is added. l , and finally a nonlinear transformation is performed through the ReLU activation function to obtain the value of the corresponding position of the output feature map.

[0130] The convolutional layer is followed by a 2x2 max pooling layer, which is used to reduce the dimensionality of the feature map. The max pooling operation selects the largest element in a 2x2 window as the representative of the window, thereby reducing the scale of the data while retaining important features.

[0131] 2.3 Output Layer

[0132] The output layer adopts the design of a fully connected layer. In this layer, the output of the last convolutional layer is flattened into a one-dimensional vector, and then transformed through the fully connected layer to obtain the output of the network. The operation of the fully connected layer can be expressed as y = Wflatten (X L )+b, where W and b are the parameters of the fully connected layer, the dimension of W is 1×d, b is a real number, flatten(X L ) represents the output feature map X of the last convolutional layer L Perform a flattening operation.

[0133] The goal of the output layer is to give the Q value of the agent performing a specific action in a specific state, denoted as Q(s i,t ,a i,t; DQNi), which can be understood as the agent in state s i,t Next, perform action a i,t With this Q value, the agent can evaluate different actions and choose the best action to perform in order to maximize the expected future reward.

[0134] It is worth noting that each agent has its own deep Q network, which can run independently on their own computing devices to achieve distributed computing. In addition, this network structure can handle high-dimensional and continuous state problems, and the convolutional neural network can automatically learn useful features from the data to improve the performance of the agent. These characteristics make the network architecture have good performance in path finding tasks in large-scale, complex, and dynamic environments.

[0135] In the process of constructing the deep Q network, the diversity and dynamics of the environment state are fully considered, so that the intelligent agent can effectively find paths in complex environments. In addition, the design of the deep Q network helps to improve the learning efficiency of the intelligent agent and the accuracy of path finding.

[0136] 2.4 Neural Network Pseudocode

[0137] Define the class DQNetwork:

[0138] Initialization method:

[0139] Accepts input parameters state_size, action_size, embedding_dim,

[0140] conv_dims,fc_dims

[0141] Initialize state size, action size, and embedding dimension

[0142] Creating state embedding and action embedding layers

[0143] Initialize an empty list of convolutional layers and a list of batch normalization layers

[0144] For each conv_dim in conv_dims:

[0145] Create a convolution layer with input_dim as input channel, conv_dim as output channel, 3x3 kernel size, 1 stride, and 1 padding.

[0146] Add a convolutional layer to the list of convolutional layers

[0147] Create a batch normalization layer with the number of input channels conv_dim

[0148] Add a batch normalization layer to the list of batch normalization layers Set conv_dim to the input dimension of the next convolutional layer

[0149] Create a fully connected layer with the input feature number input_dim*fc_dims[0]*

[0150] fc_dims[1], the output feature number is 1

[0151] Forward propagation method:

[0152] Accepts input parameters state, action

[0153] Compute embedding representations of states and actions

[0154] Concatenate the state embedding and the action embedding together and increase the dimensions by two for each pair of layers in the list of convolutional and batch normalization layers:

[0155] Apply convolution operation and ReLU activation function to the input

[0156] Apply batch normalization to the result after convolution

[0157] Apply a 2x2 max pooling operation to the normalized result to save the feature map

[0158] Flatten the feature map into a one-dimensional vector

[0159] Pass the flattened vector through the fully connected layer and output the Q value

[0160] 3. Agent Decision-making

[0161] In the D-DMAQL model, in order to ensure the learning efficiency and final learning outcomes of the agent, the present invention designs an action selection mechanism based on a deep Q network for the D-DMAQL model, and combines the exploration-utilization strategy of Upper ConfidenceBound (UCB) so that each agent can determine how to choose the optimal action under a given environmental state through the core link of the decision-making process. This combined decision-making framework provides the agent with a decision-making environment that can effectively utilize known information and actively explore new information.

[0162] Deep Q-network plays a key role in the decision-making process of intelligent agents. Deep Q-network is a function approximator based on neural networks. Its goal is to estimate the future rewards that may be brought by performing a specific action under a given environmental state. Therefore, the action selection mechanism based on deep Q-network can provide intelligent agents with a quantitative action value standard to help them choose the action with the maximum Q value. This process can be expressed by the following mathematical formula:

[0163] at =argmax*aQ(s t ,a;θ)

[0164] This formula means that at time step t, the agent chooses action a t , so that the action value Q(s) estimated by the deep Q network t ,a;θ) is the largest. Here s t is the state of the environment at time step t, and θ is the parameter of the deep Q network. In this way, the deep Q network provides the agent with the possibility of optimal action selection in a specific environment state.

[0165] The UCB strategy is an exploration-exploitation strategy widely used in multi-armed bandit problems. Its application in the agent decision-making process is mainly to promote the agent's effective exploration of the environment. The UCB strategy encourages the agent to try actions that have not been fully explored to discover better action strategies, thereby improving the overall performance of the agent. The mathematical expression of the UCB strategy can be expressed as:

[0166] a t =argmax*a(Q(s t ,a;θ)+βlogn t (a))

[0167] The goal of this formula is to select an action a t , so that Q(s t ,a;θ) and βlogn t (a) is the maximum sum. Here, β is a constant that controls the degree of exploration, t is the current time step, and n t (a) is the number of times action a was chosen before time step t. With this strategy, the agent is encouraged to explore actions that have been chosen less often.

[0168] By combining the deep Q network and the UCB strategy, the model forms a new agent decision-making mechanism. This mechanism takes into account the deep Q network's estimation of action value and the UCB strategy's emphasis on exploration, providing the agent with a decision-making environment that can effectively utilize known information and actively explore new information. The combination is:

[0169]

[0170] This formula shows that the agent chooses an action a at each time step t t , so that Q(s t ,a;θ) and This means that when the agent selects an action, it not only considers the action value estimated by the deep Q network, but also encourages trying actions that have not been fully explored.

[0171] In the agent decision-making process, parameter updating is a crucial link because it is directly related to the performance of the deep Q network and the results of the agent's decision. The goal of parameter updating is to minimize the difference between the predicted Q value and the actual Q value, so that the deep Q network can more accurately estimate the value of the action. The formula for parameter updating is as follows:

[0172] L(θ)=∑[(r+γmax a′ Q(s′,a′;θ′)-Q(s,a;θ)) 2 ]

[0173] In this formula, r is the immediate reward obtained by the agent after performing action a, s′ is the new environment state after performing action a, a′ is the possible action to be selected under the new environment state s′, γ is the discount factor, indicating the importance of future rewards, θ′ is the network parameter before parameter update, Q(s,a;θ) and Q(s′,a′;θ′) are the estimates of the current action value and future action value before parameter update, respectively. By minimizing this loss function, the deep Q network can continuously improve its estimate of the action value, thereby improving the decision-making effect of the agent.

[0174] The D-DMAQL model organically combines the deep Q network and the UCB exploration-exploitation strategy, and proposes a new agent decision-making mechanism, thus injecting novel ideas and concepts into the field of reinforcement learning.

[0175] First, the unique role of the deep Q network in action value estimation is the basis of this new decision-making mechanism. Traditional reinforcement learning algorithms often ignore the direct estimation of action value, but learn the best strategy through continuous trial and error. This method has low learning efficiency in complex environments. The deep Q network directly links the environmental state and action, and can provide a specific expected reward estimate for each possible action, allowing the intelligent agent to directly select the action with the maximum expected reward, thereby greatly improving decision-making efficiency.

[0176] Secondly, the introduction of the UCB strategy gives this decision-making mechanism a larger exploration space. In many cases, over-reliance on known information and neglecting the exploration of new information will cause the agent to fall into the local optimum and fail to reach the global optimum. The UCB strategy encourages the agent to try actions that have not been fully explored, thereby discovering better strategies and improving the global optimization ability of decision-making. The combination of these two strategies constitutes the innovation of the D-DMAQL model. This new decision-making mechanism can not only effectively utilize known information and make efficient decisions, but also actively explore new information to prevent falling into the local optimum. This decision-making mechanism that balances exploration and utilization provides a new and effective solution for reinforcement learning.

[0177] In addition, the D-DMAQL model continuously improves the estimation of action value by optimizing network parameters, which is also the innovation of the model. By minimizing the difference between the predicted Q value and the actual Q value, the model can self-optimize after each decision and improve the accuracy of future decisions.

[0178] In general, the D-DMAQL model effectively integrates the deep Q network with the UCB strategy to form a novel intelligent agent decision-making mechanism, and improves the accuracy of decision-making by dynamically optimizing network parameters, so that the model can better balance exploration and utilization when dealing with complex tasks, improve decision-making efficiency and accuracy, and further promote the development of the field of reinforcement learning.

[0179] 4. Global Optimization and Distributed Computing

[0180] 4.1 Design of Distributed Learning

[0181] In this embodiment, a distributed learning design is adopted, so that each agent can learn and make decisions in parallel on its own computing device. Specifically, each agent performs local path finding and decision making through its own deep Q network. Such a design effectively reduces the computing burden of the central computing device and improves the efficiency and scalability of the entire system.

[0182] At each time step t, each agent i will be based on its own state s i,t , perform action a i,t , and then the environment returns to the new state s i,t+1 and instant rewards i,t On this basis, agent i calculates the current state-action pair (s i,t ,a i,t ) is denoted by in Represents the network parameters of agent i. At the same time, agent i will also calculate the new state s i,t+1 The maximum Q value under These two Q values ​​will be used for subsequent parameter updates and decisions.

[0183] 4.2 Implementation of global optimization

[0184] Based on distributed learning, this embodiment further designs a global optimizer to achieve global learning and optimization. At each time step, the global optimizer collects the states, actions, rewards, and new states of all agents, as well as their respective Q values, and then calculates the global loss function based on this information, denoted as L(θ).

[0185] This global loss function is the sum of the loss functions of all agents and can be expressed as follows:

[0186]

[0187] Among them, γ is a discount factor, which is used to balance the importance of immediate rewards and future rewards (0<γ≤1). The closer γ is to 1, the more important future rewards are. θ represents the parameters of the global optimizer, which is the set of network parameters of all agents, that is, θ=θ1,θ2,...,θ n .

[0188] After calculating the global loss function, the global optimizer can update the global parameter θ through the Adam optimization algorithm. The Adam optimization algorithm adjusts the learning rate in an adaptive manner and can automatically find a suitable learning rate during the continuous optimization process, thereby improving the efficiency and stability of the optimization. The update rule of the global optimizer can be expressed by the following formula:

[0189]

[0190] Among them, α t is the learning rate at step t, is the loss function L(θ) at θ t The gradient at .

[0191] 4.3 Synergy between Distributed Learning and Global Optimization

[0192] In order to further improve the system performance, this embodiment also proposes a synergistic mechanism of distributed learning and global optimization. At each time step, in addition to making local decisions and learning, each agent will send its own state, action, reward and new state, as well as the current Q value and the maximum Q value in the new state to the global optimizer. After the global optimizer collects this information, it will calculate the global loss function, and then update the global parameters through the Adam optimization algorithm to obtain the new global parameter θ t+1 .

[0193] Next, the global optimizer will periodically use the data collected by all agents to update the parameters θ′ of the target Q network through optimization algorithms such as gradient descent. All agents will use the latest target Q network parameters θ′ to calculate the target Q value for learning and decision making. In other words, the global optimizer will update the new global parameters θ t+1The information is sent to all agents, and each agent updates its own network parameters according to its own network structure and global parameters, and then makes the next decision and learns under the new network parameters. In this way, the global optimizer and each agent effectively collaborate in distributed learning and global optimization, so that the agent can make quick decisions using local information and learn and optimize effectively with the help of global information.

[0194] In addition, in order to further improve learning efficiency, this embodiment also designs a method for dynamically optimizing network parameters. Specifically, at the end of each time step, each agent will dynamically adjust its own network parameters based on its own learning rate, reward, and maximum Q value in the new state. This method not only enables the agent to adapt to changes in the environment more quickly, but also provides strong support for the coordination of distributed learning and global optimization.

[0195] Specifically, each agent i maintains a learning rate variable a i,t After each step of learning, according to the current immediate reward r i,t and maximum Q value To dynamically adjust the learning rate:

[0196]

[0197] Among them, κ is a preset learning rate adjustment factor.

[0198] Finally, the global optimizer will periodically evaluate the network parameters of all agents. If the network parameters of an agent perform poorly, the global optimizer will generate a new network parameter for the agent based on the network parameters of other agents, thereby achieving global optimization and upgrading of the network structure.

[0199] This design not only improves the scalability and efficiency of the system, but also provides strong support for further research on the synergy mechanism of distributed learning and global optimization. In practice, this design has been proven to effectively improve the learning efficiency and decision-making quality of intelligent agents, and has broad application prospects.

[0200] 5. Training Module and Distributed Computing

[0201] In this embodiment, the training module and distributed computing are key parts. The training module is responsible for learning and updating the deep Q network of each agent so that the agent can extract valuable information from the environment state and make reasonable decisions. The use of distributed computing enables each agent to use its own computing resources for parallel learning and decision-making in a large-scale, complex, and dynamic environment, greatly improving the efficiency and scalability of the system:

[0202] 5.1 Training Module

[0203] The training module of this embodiment is based on the concepts of experience replay and fixed Q-target to effectively improve the stability and efficiency of training. In each decision-making process, the agent will generate a series of states, actions and rewards, and these experience data are stored in the experience replay pool. Then, the training module randomly extracts a portion of data from the experience replay pool to update the parameters of the deep Q network. This method avoids the impact of the correlation between data on network training. For the fixed Q target method, the training module maintains two deep Q networks: one is a network for calculating the current Q value, and the other is a network for calculating the target Q value. In each update process, only the parameters of the current Q network are updated, and the parameters of the target Q network are updated after a certain time interval. This method can effectively prevent the problem of target movement during training.

[0204] 5.2 Distributed Computing

[0205] In order to meet the challenges brought by big data and complex environments, this embodiment also proposes a unique distributed computing mechanism. In this mechanism, each agent is equipped with its own computing device and can perform parallel computing on its own device. This design not only greatly improves the computing efficiency, but also ensures the scalability of the system. On this basis, the global optimizer will also perform global learning and optimization based on the information of all agents to achieve the global optimal decision. This combination of global optimization and distributed computing not only ensures computing efficiency, but also ensures the optimization of decision-making.

[0206] 5.3 Mathematical Formulas for Quantifying Agent Learning and Optimization

[0207] In order to quantify the learning and optimization process of the agent, this embodiment also designs a mathematical formula to describe how the agent updates the parameters of its deep Q network in a quantitative way:

[0208] Let S t is the current state, A t For state S t Next, select the action, R t To perform action A t The immediate reward obtained after S t+1 is the new state, Q(S t ,A t ) is in state S t Next select Action A t The Q value, Q(S t+1 ,A t+1 ) is in the new state S t+1 Next select Action A t+1Then, updating the parameters of the deep Q network is to minimize the loss function L(θ), where θ is the parameter of the deep Q network, and the loss function L(θ) is defined as:

[0209]

[0210] Where γ is a discount factor that balances immediate rewards and future rewards. During each update, the training module optimizes the loss function to make the predicted Q value as close to the target Q value as possible.

[0211] In summary, the training module and distributed computing design of this embodiment not only realizes the distributed computing and global optimization of intelligent agents, but also takes into account the stability and efficiency of training, so that the performance of the multi-agent system in path finding and efficiency is greatly improved.

[0212] 6. Reasoning Decision Module and Dynamic Distributed Computing

[0213] 6.1 Reasoning and Decision-making Module

[0214] 6.1.1 Reasoning Phase

[0215] In the reasoning stage, the agent needs to select the optimal action a in the huge action space A. * , which greatly increases the computational complexity. Therefore, a unique intelligent pruning strategy is introduced to reduce the search space and improve computational efficiency. In this strategy, a quantitative index I is first defined, which is jointly determined by the Q-value function Q(s,a) and the environmental state information s. Let Q(s,a) be the Q-value of executing action a in state s, and env(s) be the information of the environmental state s, then the quantitative index can be defined as follows:

[0216] I=f(Q(s,a),env(s))

[0217] By sorting the quantitative index I, we can determine which actions are retained in the pruned action set A′ and which are pruned. For example, if T is a threshold, we can select actions with I greater than T for further decision making, which can greatly reduce the search space and improve the reasoning speed. By reasonably designing the f function and the threshold T, we can keep as many important actions as possible while ensuring the reasoning efficiency, and achieve a balance between reasoning speed and reasoning accuracy.

[0218] 6.1.2 Decision-making stage

[0219] In order to coordinate the decisions of each agent in a multi-agent system to ensure the overall optimality, an innovative quantitative decision-making mechanism is designed.

[0220] In this mechanism, each agent generates a decision value D when making a decision.i , this value is determined by the Q value function Q(s,a), the environment state information s and the action information a. Specifically, let Q(s,a) be the Q value of executing action a in state s, env(s) be the information of the environment state s, and act(a) be the information of action a. Then define the decision value D i =g(Q(s,a i ),env(s),act(a i )). Each agent will take its decision value D i Send it to the global optimizer, which will optimize all D i The information is summarized and analyzed, and then fed back to each agent, which then modifies its decision based on the feedback information. This mechanism ensures that each agent can participate in global optimization while maintaining its own decision-making, thus achieving good coordination between the local and global aspects.

[0221] 6.2 Dynamic Distributed Computing

[0222] In addition, this embodiment also designs a dynamic distributed coordination strategy, which uses a global optimizer to optimize the decision value D of each agent. i The information is summarized and analyzed, and then fed back to each agent, allowing the agent to adjust its decision function g(x) based on the feedback information to adapt to changes in the environment. This strategy can dynamically coordinate the agents to make their behavior more coordinated and efficient.

[0223] The above algorithm is implemented using the distributed computing framework Spark. Each agent performs local calculations on its own computing node, including executing pruning, generating decision values, etc., and then summarizes them in the global optimizer. This framework uses parallel computing to accelerate this process. At the same time, fault tolerance and load balancing mechanisms are designed to improve the robustness of the system.

[0224] In summary, by introducing unique mechanisms such as intelligent pruning, quantitative decision-making, dynamic distributed coordination, and a distributed computing framework, the entire system can efficiently handle large-scale multi-agent reinforcement learning problems and has good scalability and robustness.

[0225] VII. Experimental methods, results and comparison

[0226] In order to verify the effectiveness of the present invention, simulation experiments were conducted in multiple complex environments, and the performance was compared with several existing multi-agent path planning algorithms, as follows:

[0227] 7.1 Experimental Environment Setup

[0228] 7.1.1 Environment:

[0229] The experiments were conducted on three 2D grid environments of different sizes and complexity:

[0230] Small environment: 20x20 grid with 5 randomly distributed obstacles and 5 agents.

[0231] Medium environment: 50x50 grid with 20 randomly distributed obstacles and 10 agents.

[0232] Large environment: 100x100 grid with 50 randomly distributed obstacles and 20 agents.

[0233] 7.1.2 Comparison Algorithms:

[0234] This example selects the following three representative multi-agent path planning algorithms for comparison:

[0235] Independent Deep Q-Learning (IQL): Each agent independently trains a deep Q-network without information sharing and global optimization.

[0236] Centralized A* search: All agents are considered as a whole and the A* algorithm is used to search for the global optimal path.

[0237] Distributed A* Search: Each agent uses the A* algorithm to search for the local optimal path and coordinate actions through information exchange.

[0238] 7.1.3 Evaluation indicators:

[0239] We use the following metrics to evaluate the performance of the algorithm:

[0240] Average Completion Time: The average number of time steps required for all agents to complete the task.

[0241] Success Rate: The proportion of agents that successfully reach the target point within a specified number of time steps.

[0242] Collision Number: The total number of collisions that occurred among all agents while completing the task.

[0243] Average Computation Time: The average computation time required for the algorithm to complete a path planning.

[0244] 7.2 Experimental Results and Analysis

[0245] 7.2.1 Experimental results:

[0246] Small Environment: In a small environment, the present invention achieved an average completion time of 45.2 steps, a success rate of 98.5%, 1.8 collisions, and an average computation time of 12.3 milliseconds.

[0247] Medium environment: In the medium environment, the average completion time of the present invention is 165.4 steps, the success rate is 96.3%, the number of collisions is 4.2 times, and the average calculation time is 48.6 milliseconds.

[0248] Large environment: In a large environment, the average completion time of the present invention is 384.6 steps, the success rate is 93.7%, the number of collisions is 9.5 times, and the average calculation time is 102.1 milliseconds.

[0249] 7.2.2 Comparative analysis:

[0250] Under all environmental scales, the average completion time of the proposed method is significantly lower than that of the other three algorithms, which reflects its efficiency advantage.

[0251] The success rate of the present invention is always maintained at a high level, and the number of collisions is relatively low, which shows that while ensuring efficiency, it can also effectively coordinate the actions between intelligent agents, avoid collisions, and improve the success rate of task completion.

[0252] As the scale of the environment increases, the performance advantage of the present invention becomes more obvious. For example, in a large environment, the average completion time of the present invention is reduced by 107.7 steps compared with the IQL algorithm, 37.2 steps compared with the centralized A search algorithm, and 61.6 steps compared with the distributed A search algorithm. This reflects the good scalability of the present invention, which is more suitable for processing large-scale multi-agent path planning problems.

[0253] 7.2.3 Results and Discussion

[0254] Experimental results show that the proposed method is superior to the existing technology in terms of efficiency, accuracy and scalability, which is mainly attributed to the following innovations:

[0255] Distributed learning framework: Each agent can learn and make decisions independently, which improves learning efficiency and system scalability.

[0256] Global optimization mechanism: The global optimizer can integrate the information of all intelligent agents, perform global learning and optimization, avoid local optimal solutions, and improve the accuracy of path planning.

[0257] Intelligent pruning strategy: effectively reduces the search space and improves the efficiency of path planning.

[0258] Quantitative decision-making mechanism: coordinates the actions between agents, reduces collisions, and improves the success rate of task completion.

[0259] In summary, compared with the prior art, the present invention provides an efficient, accurate and scalable solution to the multi-agent path planning problem in complex environments, which greatly improves the precision, speed and accuracy of the path planning of the multi-agent system.

[0260] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification made to the present invention by those skilled in the art without departing from the spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A distributed deep Q-learning method for path finding in a multi-agent system, characterized in that: The following steps are involved: 1) Construct a deep Q network for each agent and combine it with the UCB strategy to form a D-DMAQL model corresponding to the multi-agent system, which is used to calculate the Q value of each agent at the current moment and the maximum Q value at the next moment; 2) Establish a D-DMAQL training module for training the D-DMAQL model; 3) Establish a reasoning decision module to calculate the decision value of each agent based on the Q value of each agent at the current moment, as well as the corresponding environmental state and action; 4) Establish a global optimizer to calculate the global loss function of the D-DMAQL model based on the Q value of each agent at each moment, as well as the corresponding environment state, action, reward, and decision value; 5) Use the trained D-DMAQL model and the global loss function to perform path finding in the multi-agent system.

2. The distributed deep Q learning method according to claim 1, characterized in that: The specific establishment process of the D-DMAQL model also includes: 1-1) Define the type of environment state in which the multi-agent system is located; 1-2) Define the action type of each agent in the multi-agent system to form a legal action set of the multi-agent system; 1-3) Setting a state transfer function to use the current state of the agent to determine the state at the next moment; 1-4) Set a reward function to guide each agent to complete the task.

3. The distributed deep Q learning method according to claim 2, characterized in that: The environmental state type includes the current position of each intelligent agent, the target position of each intelligent agent at the current moment, the position vector of the multi-intelligent system at the current moment, the actual speed of each intelligent agent at the current moment, the actual acceleration of each intelligent agent at the current moment, and the accumulated energy consumption of each intelligent agent.

4. The distributed deep Q learning method according to claim 2, characterized in that: The action type includes the process of each agent moving from a current vertex to an adjacent vertex in a given environmental state.

5. The distributed deep Q learning method according to claim 2, characterized in that: The state transfer function is as follows: s i,t+1 =f(s i,t ,a i,t ) In the formula, s i,t+1 is the state of agent i at time t+1, s i,t is the state of agent i at time t, a i,t Agent i at time t, state s i,t The action taken when 6. The distributed deep Q learning method according to claim 2, characterized in that: The reward function is as follows: r i,t =g(s i,t ,a i,t ,s i,t+1 ) In the formula, r i,t is the reward obtained by agent i at time t, s i,t+1 is the state of agent i at time t+1, s i,t is the state of agent i at time t, a i,t Agent i at time t, state s i,t The action taken when 7. The distributed deep Q learning method according to claim 2, characterized in that: The deep Q network is a convolutional neural network, which includes an input layer, multiple hidden layers, and an output layer.

8. The distributed deep Q learning method according to claim 2, characterized in that: The global loss function of the D-DMAQL model is as follows: Where L(θ) is the global loss function of the D-DMAQL model; θ is the set of all agent Q network parameters; N is the number of agents; i is the agent index; t is the time step index; r i,t is the reward obtained by agent i at time t; γ is the discount factor; s i,t is the state of agent i at time t; a i,t Agent i at time t, state s i,t Action taken when i,t+1 is the state of agent i at time t+1; a i ′ ,t+1 Agent i at time t+1, state s i,t+1 List all possible actions to take; is the parameter of the Q network of agent i at time t; θ t ′ is the parameter of the target Q network at time t.

Images