Multi-objective navigation method for agent based on deep reinforcement learning in dynamic environment

Abstract

The application discloses an intelligent agent multi-target navigation method based on deep reinforcement learning in a dynamic environment, and aims at the problems of separation of pedestrian behavior prediction and decision, insufficient modeling of behavior difference and insufficient multi-target trade-off in the prior art, and constructs a decision framework integrating pedestrian impatience prediction and Nash game. First, a simulation environment containing intelligent agents and pedestrians is established, and state and action spaces are designed; second, a multi-target reward function of safety, sociality and efficiency is constructed, and a meeting angle is introduced to depict the interaction relationship; further, a pedestrian impatience evolution model is established to realize dynamic prediction of the behavior trend of pedestrians; finally, the prediction result is embedded into a Nash equilibrium game model to guide the optimal decision; a safety penalty and a cost Critic network are introduced on the basis of a Soft Actor-Critic algorithm to realize parameter collaborative optimization.

Description

Technical Field

[0001] This invention relates to the fields of deep reinforcement learning and intelligent agent navigation, specifically to an intelligent agent autonomous navigation method based on deep reinforcement learning in dynamic environments. Background Technology

[0002] As the application of autonomous mobile intelligent agents in public places continues to expand, their autonomous navigation capabilities in densely populated environments such as shopping malls, hospitals, and transportation hubs are receiving widespread attention. In such dynamic environments, intelligent agents not only need to complete the basic task of reaching the target location, but also need to ensure movement safety, traffic efficiency, and behavioral norms that conform to human social habits while frequently interacting with multiple pedestrians.

[0003] Existing intelligent agent navigation methods can be broadly categorized into rule-based methods, traditional path planning methods, and learning-based methods. Rule-based methods typically rely on manually set obstacle avoidance and yielding strategies, which are easy to implement in simple scenarios but struggle to cope with dynamic environments with dense pedestrian traffic and complex movement patterns. Traditional path planning methods often assume a static environment or only make local dynamic corrections, lacking proactive consideration of changes in pedestrian behavior, and are prone to frequent pauses or local oscillations in high-density crowds.

[0004] In recent years, with the development of reinforcement learning technology, autonomous navigation methods for intelligent agents based on deep reinforcement learning have gradually become a research hotspot. These methods learn navigation strategies through interaction with the environment, enabling them to adapt to complex dynamic scenarios to a certain extent. However, most existing reinforcement learning navigation methods typically treat pedestrians as simple dynamic obstacles when modeling the environment, primarily relying on instantaneous information such as the pedestrian's current position and speed for decision-making, lacking effective prediction of the pedestrian's future behavioral trends. This passive, reactive decision-making approach can easily lead to insufficient safety margins or decreased navigation efficiency in scenarios where pedestrian movement changes rapidly or where interactive conflicts are frequent.

[0005] In the field of pedestrian modeling and simulation, various pedestrian motion models have been proposed, including social force models, rule-based individual models, and learning-based pedestrian trajectory prediction models. These methods can characterize the interaction relationships between pedestrians to a certain extent and predict their future positions. However, existing pedestrian simulation or prediction technologies are mostly used as independent modules, and their outputs are often only used for offline analysis or environmental simulation. They lack deep integration with the real-time decision-making process of intelligent agents, making it difficult to fully leverage the value of predictive information at the decision-making level.

[0006] Furthermore, in multi-pedestrian interaction scenarios, pedestrian movement decisions are typically influenced by a combination of factors, including target distance, surrounding density, and relative movement direction. Different pedestrians may exhibit significantly different movement intentions and behavioral intensities under the same spatial constraints. However, existing intelligent agent navigation methods often employ uniform obstacle avoidance or yielding strategies in their decision-making mechanisms, failing to distinguish the differences in pedestrian behavior states. This results in the agent lacking targeted adjustments when facing different interaction scenarios, impacting the overall pedestrian experience and system robustness.

[0007] On the other hand, in the design of reward functions or decision objectives, existing reinforcement learning navigation methods often focus on a single objective or a simple weighted objective, such as the shortest path or the lowest navigation time. They do not give enough consideration to the dynamic trade-offs between safety, social coordination and efficiency, and especially lack systematic modeling of the impact of changes in pedestrian behavior on decision-making, making it difficult to achieve a stable and reasonable multi-objective balance in complex crowd environments.

[0008] In summary, existing technologies for autonomous navigation in complex and dynamic crowd environments generally suffer from problems such as the disconnect between pedestrian behavior prediction and agent decision-making, insufficient modeling of pedestrian behavior differences, and imperfect multi-objective decision-making mechanisms. Therefore, a technical solution is needed that can effectively incorporate pedestrian behavior prediction information within a reinforcement learning framework and use it to guide real-time agent decision-making, thereby improving the safety, efficiency, and social acceptability of agents in complex crowd environments. Summary of the Invention

[0009] This invention aims to address the problems of poor overall navigation performance, insufficient robustness, and inadequate social acceptability in dynamic crowd environments caused by the lack of forward-looking prediction and collaborative decision-making regarding pedestrian behavior during navigation in multi-objective deep reinforcement learning processes in complex environments. To this end, this invention proposes a reinforcement learning-based method and system for pedestrian movement decision-making based on behavioral urgency prediction. By introducing a pedestrian behavior prediction module into the reinforcement learning framework, the evolution trend of individual behavioral urgency at future moments is predicted. The prediction results are then embedded as state input into game theory and finally integrated into the Soft Actor-Critic reinforcement learning algorithm with an added-cost Critic network. This achieves forward-looking optimization of individual movement decisions, improving overall traffic efficiency and social acceptability while ensuring safety.

[0010] The specific steps are as follows:

[0011] Step 1: Establish a simulation environment. The algorithm in this invention needs to be trained and tested in a crowded and dynamic human-machine collaborative environment. Webots simulation software is used to customize the environment and add elements such as intelligent agents and pedestrians.

[0012] Step 2: In the design of the state space and action space, the environmental state is divided into two parts: agent state and pedestrian state. The agent state describes the agent's own kinematic information, including its position, velocity, and orientation; the pedestrian state characterizes the pedestrian's motion characteristics, encompassing information such as the pedestrian's position, velocity, and direction of movement. Based on this, pedestrian psychological prediction variables are introduced to help the agent make more flexible navigation decisions when interacting with pedestrians. The action space defines the control behaviors that the agent can perform, including acceleration, deceleration, and turning. To ensure the agent's safety and stability in the dynamic environment, constraints are imposed on its maximum velocity and maximum acceleration, and the decision-making process comprehensively considers the pedestrian's motion state and behavioral changes to achieve a safe, efficient, and socially acceptable navigation strategy.

[0013] set up Indicates time The state of the intelligent agent at all times. Indicates the agent's time The actions taken pedestrian In time The state of the agent. Including observable state portion With unobservable state part The observable portion The two-dimensional position coordinates of the agent Velocity vector and acceleration Composition, the agent's velocity vector is It also includes the distance between the agent and the target point. Distance between the agent and the pedestrian Furthermore, the observable state also includes the state information of neighboring pedestrians within the agent's perception range, the pedestrian state information including pedestrians Position coordinates Current velocity vector The unobservable state portion Including target location Optimal speed Furthermore, a behavioral urgency variable is introduced. Used to identify pedestrians In time The urgency of behavioral decision-making Indicates the time step. pedestrian Allowing the maximum desired speed, the agent's action space This includes, but is not limited to, continuous or discrete control commands such as going straight, turning left, turning right, accelerating, and decelerating.

[0014] Step 3: Design a multi-objective reward function for the agent's autonomous navigation in crowded environments. This involves constructing a safety reward function, a social reward function, and an efficiency reward function, and introducing a meeting angle to guide the agent to adhere to pedestrian social norms and optimize task completion efficiency while ensuring operational safety. The meeting angle is defined by the angle between the agent's velocity vector and the pedestrian's velocity vector.

[0015] ,

[0016] Wherein, the agent's velocity vector is The pedestrian's velocity vector is , The dot product of velocity vectors. and This indicates the speed of the agent and the pedestrian. Representing intelligent agents and pedestrians The meeting angle between opposing directions.

[0017] Step 3.1: Define the meeting relationship induction factor based on the meeting angle relationship:

[0018] ,

[0019] in It is intelligent agents and pedestrians The meeting angle.

[0020] Based on inducing factors, a pedestrian social stress index can be defined:

[0021] ,

[0022] in As an inducing factor for encounter relationship, The level of impatience among pedestrians.

[0023] Step 3.2: Construct a safety reward function to constrain the safe distance relationship between the agent and pedestrians during navigation. No penalty is incurred when the agent and pedestrian are within a safe social range; a safety penalty is triggered when the agent enters a collision or psychological pressure zone.

[0024] The security reward function is defined as follows:

[0025] ,

[0026] in For pedestrians The degree of impatience, For intelligent agents and pedestrians European distance, This represents the minimum distance between the agent and the pedestrian.

[0027] Step 3.3: Construct a social reward function to measure the degree of respect the agent shows for the pedestrian's movement intentions and behavioral states during navigation. This reward function categorizes the intensity of social conflict between pedestrians and the agent under different encounter scenarios by integrating the geometric relationship between the pedestrian's psychological state and movement, enabling the agent to make avoidance behaviors that conform to social norms while ensuring movement efficiency.

[0028] The social reward function is defined as:

[0029] ,

[0030] in For pedestrians The degree of impatience, As an indicator of social stress, To achieve a low conflict threshold, This is a high conflict threshold.

[0031] Step 3.4: Construct an efficiency reward function to encourage the agent to complete the navigation task with high efficiency while ensuring safety and social constraints, and to avoid long periods of stagnation or unnecessary detours.

[0032] The efficiency reward function is defined as:

[0033] ,

[0034] in This represents the real-time distance from the agent to the target point. This is the efficiency threshold; anything exceeding this threshold is considered the worst efficiency.

[0035] Step 4: Based on a large amount of simulation data or historical interaction data, a pedestrian patience database can be established to predict pedestrian behavior trends under different interaction scenarios. During navigation, the agent estimates the patience level of surrounding pedestrians in real time and adjusts its own behavior strategy accordingly. It proactively engages in negotiated interactions with pedestrians with higher patience levels while actively moving away from pedestrians with lower patience levels, thereby reducing the risk of collisions and the probability of entering uncomfortable social spaces for pedestrians. Pedestrians' patience levels can be monitored at any given time. Patience is manifested in the following ways:

[0036] Assume that the rate of increase in pedestrian impatience is positively correlated with the degree of speed obstruction:

[0037] ,

[0038] in For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed.

[0039] Pedestrians will psychologically adapt after a long wait, and their impatience will diminish over time, exhibiting an exponential decay pattern.

[0040] ,

[0041] in For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed, This is the time decay coefficient.

[0042] The degree of impatience satisfies the differential equation of continuous-time evolution:

[0043] ,

[0044] Multiplying the time decay term by the velocity resistance intensity yields:

[0045] ,

[0046] in For pedestrians The degree of impatience, For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed, This is the time decay coefficient.

[0047] It is also discretized to adapt to the discrete time steps used in simulation and reinforcement learning:

[0048] ,

[0049] in For pedestrians The degree of impatience, For time step.

[0050] because This represents a psychological quantity, and its value needs to be restricted to a certain range. Therefore, a cutoff factor is introduced:

[0051] ,

[0052] in For pedestrians The degree of impatience, For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed, The time decay coefficient, For time step, This represents the interval clipping function.

[0053] Step 5: Integrate pedestrian patience data into the Nash equilibrium game module to improve the accuracy of the pedestrian prediction module. Using game theory equilibrium solutions, guide the agent to make optimal decisions flexibly in complex social environments. The pedestrian payoff function is:

[0054] ,

[0055] ,

[0056] ,

[0057] in This is the pedestrian efficiency weighting coefficient. This is the pedestrian conflict risk weighting coefficient. This represents the pedestrian social norm weighting coefficient. For the social comfort cost function, For patience factor, For pedestrians The degree of impatience, The minimum psychological safety threshold that pedestrians can accept. As a social distancing variable, This refers to the social comfort scale parameter.

[0058] Agent reward function:

[0059] ,

[0060] in For efficiency weighting coefficients, For safety weighting coefficients, Social weight coefficient, For efficiency reward function, For the safety reward function, This is a social reward function.

[0061] Solving for the Nash equilibrium solution based on the payoff function:

[0062] ,

[0063] ,

[0064] ,

[0065] ,

[0066] ,

[0067] in For agent actions, Accelerate the intelligent agent, The maximum threshold for agent acceleration. The maximum speed threshold for pedestrians, For the action space of the intelligent agent, Provide space for pedestrian movement. For the actions of pedestrians, Let be the agent's reward function. Let be the pedestrian benefit function. The optimal action for the agent. The predicted pedestrian actions are used to determine the optimal action for the agent through Nash game theory. The function that maximizes the agent's reward.

[0068] Step 6: Constructing the SAC Network. To address the shortcomings of ordinary AC (Acotr-Critic) networks in terms of insufficient environment exploration and unstable training, this invention employs the Soft Actor-Critic (SAC) algorithm for policy learning. The core idea of ​​SAC is to maximize cumulative reward while introducing a maximum entropy mechanism to enhance the policy's exploration ability and robustness, and to achieve this through embedded cost... The network enables adaptive security requirements, ensuring the safety of intelligent agents navigating in highly dynamic crowds.

[0069] Step 6.1: The optimization objective of the SAC network includes maximizing the cumulative reward and entropy. First, calculate the immediate reward at the current time, then calculate the policy entropy, and weight them according to their respective weights. The formula is as follows:

[0070] ,

[0071] in To optimize the objective function of the strategy, For instant rewards, For temperature parameters, For policy entropy, This is the current state.

[0072] To encourage policies to remain random, SAC introduces policy entropy:

[0073] ,

[0074] in For policy entropy, This is the current state. This represents the current action of the agent.

[0075] Step 6.2: Construct the Actor network. Based on the existing network, introduce a safety penalty term and integrate game theory into the Actor network. Therefore, the Actor optimization objective function is:

[0076] ,

[0077] in For policy network parameters, For the current reward Network parameters, For temperature parameters, This is the current state. For the agent's current action, This represents the policy probability distribution of the current Actor network. The cost Critic network parameters are used to measure the security benefits of the agent. It is a dynamic Lagrange multiplier.

[0078] Step 6.3: Construct the Critic network. The SAC network uses a dual-channel design. Structure can effectively solve To address the issue of overestimation and avoid policy collapse during training, a cost critic network is added to the existing system. Specifically assess the security benefits of intelligent agents.

[0079] It consists of the current reward and the expected future value, minus the entropy penalty term, and is calculated using the following formula:

[0080] ,

[0081] in For the target Q value, For instant rewards, As a discount factor, Rewards for Targets network, Rewards for Targets Network parameters, For the next action, For the state at the next moment, Entropy weights, usually .

[0082] We specifically evaluate safety benefits because risks are objectively present and cannot be offset by random exploration; therefore, we do not subtract the entropy penalty term here.

[0083] ,

[0084] in As a discount factor, For the sake of safety and benefit at the present moment, For cost network, The network parameters are the target cost.

[0085] Step 6.4: Update the target reward Critic network by minimizing the current... Estimated value and target The mean squared error between values ​​is used to update the value network parameters, and the loss function is applied to each value. Updated, the calculation formula is as follows:

[0086] ,

[0087] in For the goal Network parameters, Current goal Estimated value For the goal value.

[0088] Update cost Critic network:

[0089] ,

[0090] in For cost Network parameters For natural costs Estimated value For cost value.

[0091] Step 6.5: Update the Actor network parameters. The Actor network samples actions from the current policy, calculates the log probability of that action, and calculates two... Find the minimum value of the network, and for Find the gradient descent.

[0092] The loss function of the Actor network is:

[0093] ,

[0094] in The loss function of the Actor network. For policy network parameters, For the current goal Network parameters, For temperature parameters, For cost Critic network parameters, It is a dynamic Lagrange multiplier.

[0095] Step 6.6: Temperature parameters are updated collaboratively with Lagrange multipliers. The temperature parameters are used to adaptively adjust the exploration intensity. If the policy entropy is lower than the target entropy, the temperature is increased; otherwise, it is decreased. The formula is as follows:

[0096] ,

[0097] in For entropy weights, For the target entropy value, For the current state strategy, the role of the temperature parameter is to balance the agent's trade-off between exploration and reward when making decisions.

[0098] The Lagrange multiplier update formula is:

[0099] ,

[0100] in For the Lagrange learning rate, The expected cumulative cost of the current strategy. For safety tolerance threshold, For Lagrange multipliers, It automatically adjusts the penalty intensity based on the difference between the actual security gains and the security tolerance threshold, thereby controlling the action generation of the Actor network by game theory and achieving the goal of optimizing security gains.

[0101] Step 6.7: Target network soft update, to prevent the target Network and cost The network experiences severe oscillations. An exponential moving average is used to update the target network, and the calculation formulas are as follows:

[0102] ,

[0103] ,

[0104] in This is the soft update coefficient. For the target network parameters, For the current network parameters, For the target cost network parameters, These are the current cost network parameters.

[0105] To enable the agent to achieve safe, social, and efficient navigation in complex environments, this invention constructs a reinforcement learning training model that integrates pedestrian prediction, patience modeling, Nash game prediction, and the Soft Actor-Critic decision-making framework. The specific training steps are as follows:

[0106] Step 1: Initialize the Actor and Critic networks;

[0107] Step 2: Build a simulation environment and connect it to the pedestrian prediction model;

[0108] Step 3: Calculate pedestrian patience level;

[0109] Step 4: The Actor network generates agent actions based on the game theory equilibrium solution;

[0110] Step 5: The Critic network conducts a value assessment;

[0111] Step 6: Update network parameters;

[0112] Step 7: End training if the training requirements are met; if the training requirements are not met, repeat the above steps in sequence.

[0113] This invention provides a multi-objective navigation method for intelligent agents in dynamic environments based on deep reinforcement learning. By introducing Nash equilibrium game theory and a safety penalty mechanism into the decision network, the agent prioritizes safety when facing multi-objective conflicts, and on this basis, collaboratively optimizes social comfort and navigation efficiency. This method not only improves the robustness of autonomous navigation in crowded dynamic environments, but also achieves a deep integration of pedestrian psychology prediction, game-theoretic decision-making, and reinforcement learning strategies. This method has certain engineering application value and can be used for autonomous navigation of intelligent agents in complex scenarios with high pedestrian density, such as shopping malls, hospitals, and transportation hubs. The technical solution of this invention has the following beneficial technical effects:

[0114] 1. The reinforcement learning decision-making framework for predicting pedestrian behavior urgency constructed in this invention uses a pedestrian patience evolution model to dynamically predict pedestrian behavior trends and adapt to individual behavioral state differences; the prediction results are deeply embedded into the real-time decision-making process of the agent, breaking down the technical barrier of the separation between pedestrian prediction and decision-making; through quantitative modeling of pedestrian impatience, the agent can make forward-looking judgments on changes in pedestrian behavior, effectively improving the decision-making pertinence and response timeliness of the model in dynamic crowd environments.

[0115] 2. This invention proposes a SAC algorithm decision-making system integrating a Nash equilibrium module. By transforming the interaction behavior between pedestrians and agents into a payoff problem in game theory, it achieves optimization of interaction decisions between agents and pedestrians. The main decision layer guides the agent's action generation with the game equilibrium solution, while the subordinate decision layers complete the optimization of multi-objective rewards and policy learning through the SAC network, thus resolving behavioral conflicts in multi-pedestrian interactions in a hierarchical manner. This not only improves the agent's social coordination in dense crowds but also enhances the robustness and acceptability of navigation strategies in complex interaction scenarios.

[0116] 3. This invention proposes a cost-critic network based on the SAC network, which introduces a safety penalty mechanism. This network is used to evaluate the navigation safety benefits of the agent and constrain dangerous behaviors. It focuses on the quantitative assessment of safety risks and can identify unsafe behaviors such as the agent entering the collision zone or causing psychological pressure on pedestrians. It outputs corresponding safety penalty signals, which directly affect the Actor network to constrain action generation, ensuring navigation safety and improving the agent's decision-making flexibility in dynamic environments. This cost-critic network transforms pedestrian behavior prediction, social pressure, and other information into calculable safety costs, enabling the agent to have a forward-looking obstacle avoidance capability.

[0117] The invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0118] Figure 1 This is an overall flowchart of the present invention;

[0119] Figure 2 This is the algorithm flowchart. Detailed Implementation

[0120] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0121] Appendix Figure 1 This implementation case provides an overall flowchart, demonstrating a multi-objective navigation method for intelligent agents based on deep reinforcement learning in dynamic environments. It utilizes Webots simulation software to establish the simulation environment, design the state and action space, construct the multi-objective reward function, model and predict pedestrian patience, construct a Nash game decision model, construct the SAC reinforcement learning network, and train the model.

[0122] Appendix Figure 2 This implementation case provides an algorithm flowchart. The process involves the agent acquiring the current state from the simulation environment. These states are input into the policy network, which then generates an action. After executing the action, the environment provides immediate rewards, immediate costs, and a new state. All this interaction data is stored in an experience replay pool for subsequent network training. The core innovation of the algorithm lies in its unique training mechanism. The Lagrange multiplier automatically adjusts the emphasis on safety in the optimization objective based on whether the cumulative cost of the current policy exceeds a preset safety threshold. If the policy is too risky and the cost is too high, this parameter increases, forcing the policy network to be more cautious in subsequent decisions.

[0123] To facilitate understanding, we will use a specific example to illustrate complex environments and intelligent agents. Taking a shopping mall environment as an example, a complex environment consists of intelligent agents, staff, pedestrians, and fixed obstacles within the mall. The intelligent agents are service agents, which mainly complete tasks such as movement, guidance, and item delivery.

[0124] The specific implementation steps are as follows:

[0125] Implementation Step 1: Establish a simulation environment. The algorithm in this invention needs to be trained and tested in a crowded and dynamic human-machine collaborative environment. Webots simulation software is used to customize the environment and add elements such as intelligent agents and pedestrians.

[0126] Implementation Step 2: In the design of the state space and action space, the environmental state is divided into two parts: agent state and pedestrian state. The agent state describes the agent's own kinematic information, including its position, velocity, and orientation; the pedestrian state characterizes the pedestrian's motion characteristics, encompassing information such as the pedestrian's position, velocity, and direction of movement. Based on this, pedestrian psychological prediction variables are introduced to help the agent make more flexible navigation decisions when interacting with pedestrians. The action space defines the control behaviors that the agent can perform, including acceleration, deceleration, and turning. To ensure the agent's safety and stability in the dynamic environment, constraints are imposed on its maximum velocity and maximum acceleration, and the decision-making process comprehensively considers the pedestrian's motion state and behavioral changes to achieve a safe, efficient, and socially acceptable navigation strategy.

[0127] set up Indicates time The state of the intelligent agent at all times. Indicates the agent's time The actions taken pedestrian In time The state of the agent. Including observable state portion With unobservable state part The observable portion The two-dimensional position coordinates of the agent Velocity vector and acceleration Composition, the agent's velocity vector is It also includes the distance between the agent and the target point. Distance between the agent and the pedestrian Furthermore, the observable state also includes the state information of neighboring pedestrians within the agent's perception range, the pedestrian state information including pedestrians Position coordinates Current velocity vector The unobservable state portion Including target location Optimal speed Furthermore, a behavioral urgency variable is introduced. Used to identify pedestrians In time The urgency of behavioral decision-making Indicates the time step. pedestrian Allowing the maximum desired speed, the agent's action space This includes, but is not limited to, continuous or discrete control commands such as going straight, turning left, turning right, accelerating, and decelerating.

[0128] Implementation Step 3: Design a multi-objective reward function for the agent's autonomous navigation in crowded environments. This involves constructing a safety reward function, a social reward function, and an efficiency reward function, and introducing a meeting angle to guide the agent in adhering to pedestrian social norms and optimizing task completion efficiency while ensuring operational safety. The meeting angle is defined by the angle between the agent's velocity vector and that of the pedestrians.

[0129] ,

[0130] Wherein, the agent's velocity vector is The pedestrian's velocity vector is , The dot product of velocity vectors. and This indicates the speed of the agent and the pedestrian. Representing intelligent agents and pedestrians The meeting angle between opposing directions.

[0131] Step 3.1: Define the meeting relationship induction factor based on the meeting angle relationship:

[0132] ,

[0133] in It is intelligent agents and pedestrians The meeting angle.

[0134] Based on inducing factors, a pedestrian social stress index can be defined:

[0135] ,

[0136] in As an inducing factor for encounter relationship, For pedestrians The level of impatience.

[0137] Step 3.2: Construct a safety reward function to constrain the safe distance relationship between the agent and pedestrians during navigation. No penalty is incurred when the agent and pedestrian are within a safe social range; a safety penalty is triggered when the agent enters a collision or psychological pressure zone.

[0138] The security reward function is defined as follows:

[0139] ,

[0140] in For pedestrians The degree of impatience, For intelligent agents and pedestrians European distance, This represents the minimum distance between the agent and the pedestrian.

[0141] Step 3.3: Construct a social reward function to measure the degree of respect the agent shows for the pedestrian's movement intentions and behavioral states during navigation. This reward function categorizes the intensity of social conflict between pedestrians and the agent under different encounter scenarios by integrating the geometric relationship between the pedestrian's psychological state and movement, enabling the agent to make avoidance behaviors that conform to social norms while ensuring movement efficiency.

[0142] The social reward function is defined as:

[0143] ,

[0144] in For pedestrians The degree of impatience, As an indicator of social stress, To achieve a low conflict threshold, This is a high conflict threshold.

[0145] Step 3.4: Construct an efficiency reward function to encourage the agent to complete the navigation task with high efficiency while ensuring safety and social constraints, and to avoid long periods of stagnation or unnecessary detours.

[0146] The efficiency reward function is defined as:

[0147] ,

[0148] in This represents the real-time distance from the agent to the target point. This is the efficiency threshold; anything exceeding this threshold is considered the worst efficiency.

[0149] Implementation Step 4: Based on a large amount of simulation data or historical interaction data, a pedestrian patience database can be established to predict pedestrian behavior trends under different interaction scenarios. During navigation, the agent estimates the patience level of surrounding pedestrians in real time and adjusts its own behavior strategy accordingly. It proactively engages in negotiated interactions with pedestrians with higher patience levels while actively moving away from pedestrians with lower patience levels, thereby reducing the risk of collisions and the probability of entering uncomfortable social spaces for pedestrians. Pedestrians' patience levels can be monitored at any time. Impatience manifests itself in the following ways:

[0150] Assume that the rate of increase in pedestrian impatience is positively correlated with the degree of speed obstruction:

[0151] ,

[0152] in For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed.

[0153] Pedestrians will psychologically adapt after a long wait, and their impatience will diminish over time, exhibiting an exponential decay pattern.

[0154] ,

[0155] in For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed, This is the time decay coefficient.

[0156] The degree of impatience satisfies the differential equation of continuous-time evolution:

[0157] ,

[0158] Multiplying the time decay term by the velocity resistance intensity yields:

[0159] ,

[0160] in For pedestrians The degree of impatience, For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed, This is the time decay coefficient.

[0161] It is also discretized to adapt to the discrete time steps used in simulation and reinforcement learning:

[0162] ,

[0163] in For pedestrians The degree of impatience, For time step.

[0164] because This represents a psychological quantity, and its value needs to be restricted to a certain range. Therefore, a cutoff factor is introduced:

[0165] ,

[0166] in For pedestrians The degree of impatience, For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed, The time decay coefficient, For time step, This represents the interval clipping function.

[0167] Implementation Step 5: Integrate pedestrian patience data into the Nash equilibrium game module to improve the accuracy of the pedestrian prediction module. Using game theory equilibrium solutions, guide the agent to make optimal decisions flexibly in complex social environments. The pedestrian payoff function is:

[0168] ,

[0169] ,

[0170] ,

[0171] in This is the pedestrian efficiency weighting coefficient. This is the pedestrian conflict risk weighting coefficient. This represents the pedestrian social norm weighting coefficient. For the social comfort cost function, For patience factor, For pedestrians The degree of impatience, The minimum psychological safety threshold that pedestrians can accept. As a social distancing variable, This refers to the social comfort scale parameter.

[0172] Agent reward function:

[0173] ,

[0174] in For efficiency weighting coefficients, For safety weighting coefficients, Social weight coefficient, For efficiency reward function, For the safety reward function, This is a social reward function.

[0175] Solving for the Nash equilibrium solution based on the payoff function:

[0176] ,

[0177] ,

[0178] ,

[0179] ,

[0180] ,

[0181] in For agent actions, Accelerate the intelligent agent, The maximum threshold for agent acceleration. The maximum speed threshold for pedestrians, For the action space of the intelligent agent, Provide space for pedestrian movement. For the actions of pedestrians, Let be the agent's reward function. Let be the pedestrian benefit function. The optimal action for the agent. The predicted pedestrian actions are used to determine the optimal action for the agent through Nash game theory. The function that maximizes the agent's reward.

[0182] Implementation Step 6: Constructing the SAC Network. To address the shortcomings of ordinary AC (Acotr-Critic) networks in terms of insufficient environment exploration and unstable training, this invention employs the Soft Actor-Critic (SAC) algorithm for policy learning. The core idea of ​​SAC is to maximize cumulative reward while introducing a maximum entropy mechanism to enhance the policy's exploration ability and robustness, and to achieve this through embedded cost... The network enables adaptive security requirements, ensuring the safety of intelligent agents navigating in highly dynamic crowds.

[0183] Step 6.1: The optimization objective of the SAC network includes maximizing the cumulative reward and entropy. First, calculate the immediate reward at the current time, then calculate the policy entropy, and weight them according to their respective weights. The formula is as follows:

[0184] ,

[0185] in To optimize the objective function of the strategy, For instant rewards, For temperature parameters, For policy entropy, This is the current state.

[0186] To encourage policies to remain random, SAC introduces policy entropy:

[0187] ,

[0188] in For policy entropy, This is the current state. This represents the current action of the agent.

[0189] Step 6.2: Construct the Actor network. Based on the existing network, introduce a safety penalty term and integrate game theory into the Actor network. Therefore, the Actor optimization objective function is:

[0190] ,

[0191] in For policy network parameters, For the current reward Network parameters, For temperature parameters, This is the current state. For the agent's current action, This represents the policy probability distribution of the current Actor network. The cost Critic network parameters are used to measure the security benefits of the agent. It is a dynamic Lagrange multiplier.

[0192] Step 6.3: Construct the Critic network. The SAC network uses a dual-channel design. Structure can effectively solve To address the issue of overestimation and avoid policy collapse during training, a cost critic network is added to the existing system. Specifically assess the security benefits of intelligent agents.

[0193] It consists of the current reward and the expected future value, minus the entropy penalty term, and is calculated using the following formula:

[0194] ,

[0195] in For the target Q value, For instant rewards, As a discount factor, Rewards for Targets network, Rewards for Targets Network parameters, For the next action, For the state at the next moment, Entropy weights, usually .

[0196] We specifically evaluate safety benefits because risks are objectively present and cannot be offset by random exploration; therefore, we do not subtract the entropy penalty term here.

[0197] ,

[0198] in As a discount factor, For the sake of safety and benefit at the present moment, For cost network, The network parameters are the target cost.

[0199] Step 6.4: Update the target reward Critic network by minimizing the current... Estimated value and target The mean squared error between values ​​is used to update the value network parameters, and the loss function is applied to each value. Updated, the calculation formula is as follows:

[0200] ,

[0201] in For the goal Network parameters, Current goal Estimated value For the goal value.

[0202] Update cost Critic network:

[0203] ,

[0204] in For cost Network parameters For natural costs Estimated value For cost value.

[0205] Step 6.5: Update the Actor network parameters. The Actor network samples actions from the current policy, calculates the log probability of that action, and calculates two... Find the minimum value of the network, and for Find the gradient descent.

[0206] The loss function of the Actor network is:

[0207] ,

[0208] in The loss function of the Actor network. For policy network parameters, For the current goal Network parameters, For temperature parameters, For cost Critic network parameters, It is a dynamic Lagrange multiplier.

[0209] Step 6.6: Temperature parameters are updated collaboratively with Lagrange multipliers. The temperature parameters are used to adaptively adjust the exploration intensity. If the policy entropy is lower than the target entropy, the temperature is increased; otherwise, it is decreased. The formula is as follows:

[0210] ,

[0211] in For entropy weights, For the target entropy value, For the current state strategy, the role of the temperature parameter is to balance the agent's trade-off between exploration and reward when making decisions.

[0212] The Lagrange multiplier update formula is:

[0213] ,

[0214] in For the Lagrange learning rate, The expected cumulative cost of the current strategy. For safety tolerance threshold, For Lagrange multipliers, It automatically adjusts the penalty intensity based on the difference between the actual security gains and the security tolerance threshold, thereby controlling the action generation of the Actor network by game theory and achieving the goal of optimizing security gains.

[0215] Step 6.7: Target network soft update, to prevent the target Network and cost The network experiences severe oscillations. An exponential moving average is used to update the target network, and the calculation formulas are as follows:

[0216] ,

[0217] ,

[0218] in This is the soft update coefficient. For the target network parameters, For the current network parameters, For the target cost network parameters, These are the current cost network parameters.

[0219] To enable the agent to achieve safe, social, and efficient navigation in complex environments, this invention constructs a reinforcement learning training model that integrates pedestrian prediction, patience modeling, Nash game prediction, and the Soft Actor-Critic decision-making framework. The specific training steps are as follows:

[0220] Step 1: Initialize the Actor and Critic networks;

[0221] Step 2: Build a simulation environment and connect it to the pedestrian prediction model;

[0222] Step 3: Calculate pedestrian patience level;

[0223] Step 4: The Actor network generates agent actions based on the game theory equilibrium solution;

[0224] Step 5: The Critic network conducts a value assessment;

[0225] Step 6: Update network parameters;

[0226] Step 7: End training if the training requirements are met; if the training requirements are not met, repeat the above steps in sequence.

Claims

1. A multi-target navigation method for intelligent agents based on deep reinforcement learning in dynamic environments, characterized in that, Includes the following steps: Step 1: Establish a simulation environment. Use Webots simulation software to build a dynamic crowd environment. Add mobile intelligent agents and pedestrians to the environment to simulate complex dynamic scenarios of intelligent agents interacting with multiple pedestrians. Step 2: Design the state space and action space. Divide the environmental state into agent state and pedestrian state. Agent state includes information such as agent position, speed, direction of movement, and distance between agent and target point. Pedestrian state includes information such as pedestrian position, speed, and direction of movement. Based on this, introduce pedestrian behavior prediction variables to describe the changes in the pedestrian's psychological state during the interaction process. Agent action space includes control commands such as acceleration, deceleration, straight movement, and turning. Constraints are imposed on its maximum speed and maximum acceleration, and the pedestrian's motion state and behavior changes are comprehensively considered in the decision-making process. Step 3: Construct a meeting relationship induction factor by the meeting angle between the agent and the pedestrian velocity vector, calculate the pedestrian social pressure index, design a multi-objective reward function, and construct a safety reward function, a social reward function, and an efficiency reward function; Step 4: Analyze the relationship between pedestrian movement speed and expected comfort speed, establish a pedestrian patience evolution model, describe the pedestrian psychological adaptation process through an exponential decay function, and discretize the continuous-time model. Step 5: Establish the agent's reward function and the pedestrian's reward function, and incorporate efficiency reward, safety reward and social reward into the reward function. By solving the Nash equilibrium, the predicted pedestrian action and the agent's optimal action are obtained. Step 6: Construct a SAC reinforcement learning framework that integrates security constraints, introduces a security penalty term and a dynamic Lagrange multiplier, and uses the Critic network and the cost Critic network together as the Critic network in the SAC network.

2. The agent-based multi-target navigation method in dynamic environments according to claim 1, characterized in that, Step three involves constructing a meeting relationship induction factor based on the meeting angle between the agent and the pedestrian's velocity vector, calculating the pedestrian social pressure index, designing a multi-objective reward function, and constructing a safety reward function, a social reward function, and an efficiency reward function. This is implemented specifically according to the following steps: Based on the meeting angle relationship, define the meeting relationship induction factor: ， in It is intelligent agents and pedestrians The meeting angle; Based on inducing factors, a pedestrian social stress index can be defined: ， in As an inducing factor for encounter relationship, For pedestrians The degree of impatience; The security reward function is defined as follows: ， in For pedestrians The degree of impatience, For intelligent agents and pedestrians European distance, This represents the minimum distance between the agent and the pedestrian. A social reward function is constructed to measure the degree of respect that the agent shows for the pedestrian's movement intentions and behavioral state during navigation. This reward is used to classify the intensity of social conflict between pedestrians and agents in different encounter situations by integrating the geometric relationship between the pedestrian's psychological state and movement. The social reward function is defined as: ， in For pedestrians The degree of impatience, As an indicator of social stress, To achieve a low conflict threshold, High conflict threshold; Construct an efficiency reward function to encourage the agent to complete the navigation task with high efficiency while ensuring safety and social constraints, and avoid long periods of stagnation or unnecessary detours. The efficiency reward function is defined as: ， in This represents the real-time distance from the agent to the target point. It is the efficiency threshold.

3. The agent-based multi-objective navigation method in dynamic environments according to claim 1, characterized in that, Step four involves analyzing the relationship between pedestrian movement speed and expected comfort speed, establishing a pedestrian patience evolution model, describing the pedestrian psychological adaptation process using an exponential decay function, and discretizing the continuous-time model. The specific implementation steps are as follows: Assume that the rate of increase in pedestrian impatience is positively correlated with the degree of speed obstruction: ， in For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed; Pedestrians will psychologically adapt after a long wait, and their impatience will diminish over time, exhibiting an exponential decay pattern. ， in For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed, This is the time decay coefficient; The degree of impatience satisfies the differential equation of continuous-time evolution: ， Multiplying the time decay term by the velocity resistance intensity yields: ， in For pedestrians The degree of impatience, For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed, This is the time decay coefficient; Discretize it to fit the discrete time step used in simulation and reinforcement learning: ， in For pedestrians The degree of impatience, For time step; This represents a psychological quantity, and its value needs to be restricted to a certain range. Introducing a cutoff factor: ， in For pedestrians The degree of impatience, For pedestrians exist Speed ​​of motion at any moment For maximum comfortable walking speed, The time decay coefficient, For time step, This represents the interval clipping function.

4. The agent-based multi-target navigation method in dynamic environments according to claim 1, characterized in that, Step five involves establishing the agent's reward function and the pedestrian's reward function, incorporating efficiency rewards, safety rewards, and social rewards into the reward function, and obtaining the predicted pedestrian actions and the agent's optimal actions by solving the Nash equilibrium. This is specifically implemented according to the following steps: The pedestrian benefit function is: ， ， ， in This is the pedestrian efficiency weighting coefficient. This is the pedestrian conflict risk weighting coefficient. This represents the weighting coefficient for pedestrian social norms. For the social comfort cost function, For patience factor, For pedestrians The degree of impatience, The minimum psychological safety threshold that pedestrians can accept. As a social distancing variable, For social comfort scale parameters; Agent reward function: ， in For efficiency weighting coefficients, For safety weighting coefficients, Social weight coefficient, For efficiency reward function, For the safety reward function, For social reward functions; Solving for the Nash equilibrium solution based on the payoff function: ， ， ， ， ， in For agent actions, Accelerate the intelligent agent, The maximum threshold for agent acceleration. The maximum speed threshold for pedestrians, For the action space of the intelligent agent, Provide space for pedestrian movement. For the actions of pedestrians, Let be the agent's reward function. Let be the pedestrian benefit function. The optimal action for the agent. Predicted pedestrian movements.

5. The agent-based multi-target navigation method in dynamic environments according to claim 1, characterized in that, Step six describes the construction of the SAC reinforcement learning framework with integrated safety constraints. This involves introducing a safety penalty term and a dynamic Lagrange multiplier. The Critic network and the cost Critic network together form the Critic network within the SAC network. The specific implementation steps are as follows: The optimization objective of the SAC network includes maximizing the cumulative reward and entropy. First, the immediate reward at the current time is calculated, then the policy entropy is calculated, and finally, the values ​​are weighted according to their respective weights. The formula is as follows: ， in To optimize the objective function of the strategy, For instant rewards, For temperature parameters, For policy entropy, This is the current state; SAC introduces policy entropy: ， in For policy entropy, This is the current state. The current action of the agent; Construct an Actor network, introduce a safety penalty term on the original basis, and incorporate game theory into the Actor network. The optimization objective function of the Actor network is: ， in For policy network parameters, For the current reward Network parameters, For temperature parameters, This is the current state. For the agent's current action, This represents the policy probability distribution of the current Actor network. The cost Critic network parameters are used to measure the security benefits of the agent. For dynamic Lagrange multipliers; Constructing the Critic network, the SAC network employs a dual-mode approach. The structure, based on the original, adds a cost critic network. Specifically assess the security benefits of intelligent agents; It consists of current reward and future expected value, and its calculation formula is as follows: ， in For the target Q value, For instant rewards, As a discount factor, For the goal network, For the target network parameters, For the next action, For the state at the next moment, Entropy weights, usually ; Specifically assess the security benefits: ， in As a discount factor, For the sake of safety and benefit at the present moment, For cost network, For target cost network parameters; Update the Critic network by minimizing the current... Estimated value and target The mean squared error between values ​​is used to update the value network parameters. The loss function of the Critic network is applied to... Updated, the calculation formula is as follows: ， in For the goal Network parameters, Current goal Estimated value For the goal value; Update cost Critic network: ， in For cost Network parameters For natural costs Estimated value For cost value; Update the Actor network parameters. The Actor network samples actions from the current policy, calculates the log probability of that action, and calculates two... Find the minimum value of the network, and for Find the gradient descent; The loss function of the Actor network is: ， in The loss function of the Actor network. For policy network parameters, For the current goal Network parameters, For temperature parameters, For cost Critic network parameters, For dynamic Lagrange multipliers; Temperature parameters are updated in conjunction with Lagrange multipliers: ， in For entropy weights, For the target entropy value, The strategy for the current state; The Lagrange multiplier update formula is: ， in For the Lagrange learning rate, The expected cumulative cost of the current strategy. For safety tolerance threshold, For dynamic Lagrange multipliers; The target network is soft-updated using an exponential moving average, and the calculation formulas are as follows: ， ， in This is the soft update coefficient. For the target network parameters, For the current network parameters, For the target cost network parameters, These are the current cost network parameters.

Images