Multi-agent path planning method based on social value orientation and DRL fusion

Abstract

The invention relates to the technical field of intelligent manufacturing workshop logistics scheduling, in particular to a multi-agent path planning method based on social value orientation and DRL fusion, and the method comprises the steps: S1, building multi-channel tensor representation based on local observation, carrying out the spatial coding of dynamic information, and building an interpretable thinking feature space; s2, receiving neighbor agent dynamic information based on a heterogeneous graph attention network communication mechanism of SVO, separating and aggregating the information through a multi-head SVO perception heterogeneous graph attention network, and extracting information which is most critical to self decision; and S3, based on an SVO double-layer decision-making mechanism of a neighbor agent group, combining SVO and PPO algorithms, and dynamically adjusting an SVO strategy by adopting a local environment and a neighbor agent group to realize dynamic change of a self-adaptive environment of a lower-layer action strategy. According to the method, path conflicts and deadlocks are effectively eliminated, the throughput and success rate of the system are improved, an efficient collaborative solution is provided, and the workshop operation efficiency and the intelligent level are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent manufacturing workshop logistics scheduling, and specifically to a multi-agent path planning method based on the fusion of social value orientation and DRL. Background Art

[0002] In recent years, LMAPF variants have garnered widespread attention in academia, with significant application value in the collaborative scheduling of warehouse logistics robots. This application scenario represents a multi-billion dollar industry, exemplified by the 2023 online LMAPF Research Challenge, organized by Amazon Robotics. This competition aims to explore more challenging LMAPF applications. However, LMAPF presents additional challenges, particularly ensuring efficient coordination of agents and real-time, conflict-free decision-making in long-term tasks. Deep reinforcement learning (DRL), which combines the powerful representational capabilities of neural networks with the sequential decision-making mechanisms of reinforcement learning, has become a key technology for solving decision-making problems in complex dynamic environments. Experience data from agents is centrally collected to train a shared policy network, which is then optimized simultaneously. During execution, agents make independent decisions based solely on local observations and the shared policy, without requiring global information. This approach is highly scalable, applicable to any number of agents and environments of varying sizes, and supports real-time path computation in LMAPF tasks. However, the dynamic nature of LMAPF presents two challenges: Homogeneity defect of shared strategies: Existing deep reinforcement learning (DRL) methods train shared strategy networks by centrally collecting empirical data. Although this achieves distributed decision-making without global information and good scalability, this feature can easily cause deadlock (such as bidirectional blocking) and livelock (such as cyclic avoidance) in complex environments. As the task continues, the average throughput of the system decreases exponentially, severely limiting the effectiveness of long-term tasks.

[0003] Collaboration blind spots under local observation: Agents rely solely on local observations (e.g., their own position and obstacles) for decision-making, unable to directly perceive their neighbors' goals, priorities, or social preferences (e.g., avoidance tendencies). This results in coordination mechanisms relying solely on statistical inferences from historical trajectories, making it difficult to cope with dynamic conflicts. Even if individual strategy optimization improves local decision-making efficiency, global coordination can still fall into a "short-sighted game" due to information asymmetry, manifesting as stability issues such as path fluctuations and intensified resource competition. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-agent path planning method based on the fusion of social value orientation and DRL to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: The multi-agent path planning method based on the integration of social value orientation and DRL includes: Step S1: construct a multi-channel tensor representation based on local observations, spatially encode dynamic information, and establish an interpretable thinking feature space; Step S2: Based on the SVO-based heterogeneous graph attention network communication mechanism, it receives dynamic information from neighboring agents, separates and aggregates the information through the multi-head SVO perception heterogeneous graph attention network, and extracts the most critical information for its own decision-making; Step S3: Based on the SVO two-layer decision-making mechanism of the neighboring intelligent agent group, the SVO and PPO algorithms are combined, and the local environment and the neighboring intelligent agent group are used to dynamically adjust the SVO strategy to achieve the dynamic changes of the lower-level action strategy adaptive environment.

[0006] Preferably, the role attributes of the agent are dynamically defined by the relative relationship between its own SVO value and that of neighboring agents, and a role classification method based on a local perspective is further proposed, including: If the agent The SVO value is , agent There is a set of neighboring agents , agent , its SVO value is ; when When the agent Compared to intelligent agents Pay more attention to self-interest, that is, in the intelligent body From the perspective of intelligent agents Is a selfish intelligent agent; when , agent Compared to intelligent agents Pay more attention to collective interests, that is, in the intelligent body From the perspective of intelligent agents Is a selfless intelligent agent; when , agent With the agent Adopt a balanced strategy based on historical experience, that is, Vision, intelligent agent Is a neutral class agent; The same agent presents different roles from the perspectives of different agents. This relativity can well reflect the complexity of the multi-agent system and echoes the multifaceted nature of individual roles in human society. The roles and behaviors of the agents will dynamically adjust with the group structure in which they are located, further affecting the overall emergent characteristics of the system.

[0007] Preferably, the environmental complexity of the agent can be modeled as a function of the neighboring agent group and its own state, and its mathematical expression is as follows: in, For intelligent agents The distance to the neighboring agent is less than The number of The larger the size, the more complex the environment. is the state of the agent. If the agent is in deadlock, is 1, if the agent is in livelock, then is the number of livelock steps, otherwise is 0.

[0008] Preferably, the step S3 uses the local environment and the neighboring agent group to dynamically adjust the SVO strategy, and the strategy network dynamically generates the personality score by real-time perception of local observation information and the characteristics of neighboring agents. , and mapped to SVO angle value ; SVO Strategy As an additional layer embedded in the policy network structure, the SVO policy and the action policy share feedback signals, enabling parallel training and collaborative optimization of the two policies. Through the reward function, the agent can dynamically adjust its SVO value to adapt to different environmental conditions and select efficient actions based on the SVO policy, effectively breaking the behavioral homogeneity problem caused by the shared policy network; Preferably, the reward function includes: The SVO reward function calculates the average of the basic rewards of an agent and other agents in its neighborhood, establishing a dynamic balance between individual interests and group interests, guiding agents to choose social value orientations with a cooperative tendency. The action reward function is based on the SVO theoretical framework, which realizes the adaptive adjustment of the agent's behavior strategy. That is, in a low-density local environment, the agent maximizes the long-term cumulative reward. To optimize the efficiency of individual path planning; in high-density complex environments, agents tend to be pro-social, by maximizing To coordinate the conflict resolution among multiple agents, this effectively solves the behavior homogeneity problem caused by the fixed strategy network in traditional methods, and ensures the adaptability of the system in different environmental conditions through a dynamic adjustment mechanism; By maximizing the accumulation and , find the optimal SVO strategy and action strategy, .

[0009] Preferably, a variant of the PPO algorithm in step S3 is used to support the training of the agent, namely: SVO perceives multi-strategy PPO to construct SMPPO, and SMPPO needs to optimize SVO strategy at the same time and action strategies , and the optimization objectives of the two are executed independently, and the loss function associated with each strategy is formalized as and ; and is the advantage function, which measures the superiority of a specific SVO strategy or action strategy relative to the average strategy in a given state; By limiting the probability ratio of new and old strategies The scope of change to avoid excessive policy updates; Using Hyperparameters Define the clipping interval and use the min operation to combine the original probability ratio and the clipped value to optimize the strategy target, thereby improving sample efficiency while ensuring training stability; The SVO policy loss is calculated through the action advantage function, and the action policy loss is calculated through the SVO advantage function to ensure that the behavior of the agent is consistent with the SVO decision.

[0010] Preferably, the SMPPO encourages action strategies to overcome short-sightedness and continue to follow the guidance of the SVO strategy through policy gradient optimization. The gradient optimization is as follows: This two-way reinforcement between SVO strategies and action strategies creates a collaborative learning environment. It not only aligns individual behaviors with collective goals, but also dynamically optimizes SVO strategies based on action outcomes, thereby directly forming a continuous and mutually beneficial positive feedback loop between SVO decision-making and action execution.

[0011] Preferably, in order to improve the SVO stability of the agent, a supervised learning paradigm is adopted to construct the SVO stability enhancement loss The formula can be expressed as follows: Is the cross entropy between the SVO policy output by the neural network and the expected SVO probability distribution. Expected SVO value is the current value and previous value The weighted combination of gets: By introducing an under-relaxation factor based on the complexity of the environment , change the current SVO Compared with the previous SVO Mixing to determine expected SVO Stabilize the loss function; ensure that SVO adjusts to the expected value and achieves dynamic and smooth updates of SVO; is the boundary condition of environmental complexity; when the actual environmental complexity between agents is lower than When the complexity of the environment decreases, the system will adaptively relax the policy constraints, allowing the agent to make more flexible adjustments according to specific task requirements while ensuring stability.

[0012] Compared with the prior art, the present invention has the following beneficial effects: This paper proposes a method called SPHGL that integrates SVO and RL to solve the LMAPF problem. First, an SVO two-layer strategy architecture based on a group of neighboring intelligent agents is constructed, so that they can automatically adjust their action preferences according to the complexity of the environment. At the same time, in order to better increase the coordination between intelligent agents, a communication mechanism based on SVO perception is proposed, which uses the SVO value of the agent to establish a communication topology. Then, simulation experiments are successfully implemented under different conditions to evaluate the performance of SPGLP. It outperforms the most advanced RL-based MAPF method with higher throughput and improves the success rate of MAPF. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is the overall architecture diagram of the present invention; Figure 2 This is a diagram of the double-layer strategy architecture of the present invention; Figure 3 Schematic diagram of the communication mechanism of the present invention; Figure 4 These are the experimental maps of the present invention: (a) maze maps with different wall lengths; (b) random maps with different obstacle densities; (c) lak303d in a game environment; (d) warehouse maps in an industrial environment. Figure 5 Schematic diagram of DLR under different obstacle densities when there are 64 intelligent agents in the present invention; Figure 6Schematic diagram of DLR with different numbers of agents under 30% obstacle density of the present invention; Figure 7 This is a schematic diagram of the average throughput of the present invention under a maze map with corridor lengths of 3, 5, and 7; Figure 8 This is a schematic diagram of the average throughput of large-scale intelligent agents under 30% obstacle density of the present invention; Figure 9 This is a schematic diagram of average throughput under the warehouse map of the present invention; Figure 10 This is a schematic diagram of the average throughput under lak303d of the present invention; Figure 11 Schematic diagram of the success rate under different maps of the present invention. DETAILED DESCRIPTION

[0014] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0015] Example 1: The multi-agent path planning method based on the integration of social value orientation and DRL is suitable for solving the efficient collaboration, real-time path optimization and deadlock / livelock problems of multiple transport vehicles (agents) in workshop logistics scenarios. It includes the following steps: Step S1: construct a multi-channel tensor representation based on local observations, spatially encode dynamic information, and establish an interpretable thinking feature space; Step S2: Based on the SVO-based heterogeneous graph attention network communication mechanism, it receives dynamic information from neighboring agents, separates and aggregates the information through the multi-head SVO perception heterogeneous graph attention network, and extracts the most critical information for its own decision-making; Step S3: Based on the SVO two-layer decision-making mechanism of the neighboring intelligent agent group, the SVO and PPO algorithms are combined, and the local environment and the neighboring intelligent agent group are used to dynamically adjust the SVO strategy to achieve the dynamic changes of the lower-level action strategy adaptive environment.

[0016] Determination of SVO values ​​and role classification; To optimize the overall throughput of a multi-agent system, a dynamic, environmentally adaptable social value orientation (SVO) adjustment mechanism is required. In low-density visual environments (without other agents), agents should adopt a completely self-serving strategy, completing tasks quickly to improve both individual efficiency and system throughput. In highly complex environments (especially those experiencing deadlock or livelock), agents must proactively adjust their behavior, favoring altruism and prioritizing assistance in resolving system deadlocks. This ensures the collaborative stability and overall performance of the multi-agent system during long-term tasks. To address this need, this paper expands the traditional static SVO concept into a dynamic property, enabling real-time adjustments based on the agent's local environmental state and interactions with neighboring agents.

[0017] Standard SVO (with means) should be between self-interest ( ) and prosocial ( ), the social value orientation of the intelligent agent is defined as: When the number of agents is For any agent , whose personality score satisfy , each agent The social value orientation perspective is: .

[0018] However, traditional social value orientation (SVO) models classify agents' personalities using fixed thresholds (e.g., SVO values ​​greater than 45° are considered selfless, and less than 45° are considered selfish), but fail to fully consider the interactions between agents. In real-world applications, an agent's role attributes need to be dynamically defined based on its own SVO value relative to its neighboring agents. To address this, this paper proposes a role classification method based on a local perspective: Definition 4: If the agent The SVO value is , agent There is a set of neighboring agents , agent , its SVO value is when When the agent Compared to intelligent agents Pay more attention to self-interest, that is, in the intelligent body From the perspective of intelligent agents is a selfish agent, when , agent Compared to intelligent agents Pay more attention to collective interests, that is, in the intelligent body From the perspective of intelligent agents It is a selfless intelligent agent.

[0019] when , agent With the agent Adopt a balanced strategy based on historical experience, that is, Vision, intelligent agent It is a neutral class agent.

[0020] The same agent presents different roles under the observation perspective of different agents (agent In the intelligent body Vision is a selfish agent, but agent In the intelligent body This relativity can well reflect the complexity of multi-agent systems and echoes the multifaceted nature of individual roles in human society. The roles and behaviors of agents will dynamically adjust with the group structure in which they are located, further affecting the overall emergent characteristics of the system.

[0021] The complexity of an agent's environment can be modeled as a function of its neighboring agents and its own state. The mathematical expression is as follows: (2) in For intelligent agents The distance to the neighboring agent is less than The number of The larger the size, the more complex the environment. is the state of the agent. If the agent is in deadlock, is 1, if the agent is in livelock, then is the number of livelock steps, otherwise is 0.

[0022] Reward function; The strategy network dynamically generates personality scores by perceiving local observation information and the characteristics of neighboring agents in real time. , and mapped to SVO angle value SVO strategy As an additional layer embedded in the policy network structure, the policy network is as follows Figure 1 Description: The SVO strategy and the action strategy share feedback signals, enabling parallel training and collaborative optimization of the two strategies. Through the reward function, the agent can dynamically adjust its SVO value to adapt to different environmental states and select efficient actions based on the SVO strategy, effectively breaking the behavioral homogeneity problem caused by the shared strategy network. (Table 1) is a goal-oriented sparse reward that ensures the basic operating efficiency of the system.

[0023] Table 1 Basic Rewards However, in actual operation, when an agent insists on its own choice in a conflict scenario, its neighboring agents need to bear additional external penalties to deal with the resulting conflict consequences. Therefore, in terms of reward mechanism design, the present invention constructs a reward function with two components: The SVO reward function calculates the average of the basic rewards of an agent and other agents in its neighborhood, establishing a dynamic balance between individual interests and group interests, guiding agents to choose social value orientations with a cooperative tendency. The action reward function is based on the SVO theoretical framework, which realizes the adaptive adjustment of the agent's behavior strategy - in a low-density local environment, the agent maximizes the long-term cumulative reward To optimize the efficiency of individual path planning; in high-density complex environments, agents tend to be pro-social, by maximizing This effectively solves the behavioral homogeneity problem caused by fixed strategy networks in traditional methods, and ensures the system's adaptability in different environmental conditions through a dynamic adjustment mechanism.

[0024] (3) (4) By maximizing the accumulation and , find the optimal SVO strategy and action strategy, (5) (6) SMPPO algorithm; The present invention uses a variant of PPO to support the training of intelligent agents, namely SVO-aware Multi-strategy PPO (SMPPO), in which SMPPO needs to optimize SVO strategies at the same time. and action strategies , and the optimization objectives of the two are executed independently, so the present invention explicitly formalizes the loss function associated with each strategy as and ; (8) and It is an advantage function that measures the superiority of a specific SVO strategy or action strategy relative to the average strategy in a given state. The range of change is to avoid excessive policy updates. Use hyperparameters Define the clipping interval and use the min operation to combine the original probability ratio and the clipped value to optimize the policy target, thereby ensuring training stability while improving sample efficiency. Calculate the SVO policy loss through the action advantage function to ensure that the agent's behavior is consistent with the SVO decision.

[0025] SMPPO uses policy gradient optimization to encourage action strategies to overcome short-sightedness and continue to follow the guidance of the SVO strategy. The gradient optimization is as follows: (9) (10) This two-way reinforcement between SVO strategies and action strategies creates a collaborative learning environment. It not only aligns individual behaviors with collective goals, but also dynamically optimizes SVO strategies based on action outcomes, thereby directly forming a continuous and mutually beneficial positive feedback loop between SVO decision-making and action execution.

[0026] In order to improve the SVO stability of the agent, this paper adopts the supervised learning paradigm. Construct SVO stability enhancement loss The formula can be expressed as follows: Is the cross entropy between the SVO policy output by the neural network and the expected SVO probability distribution. Expected SVO value is the current value and previous value The weighted combination of gets: (12) By introducing an under-relaxation factor based on the complexity of the environment , change the current SVO Compared with the previous SVO Mixing to determine expected SVO Stabilize the loss function. Ensure that the SVO adjusts to the expected value and achieves dynamic and smooth updates of the SVO.

[0027] (13) is the boundary condition of environmental complexity; when the actual environmental complexity between agents is lower than When the complexity of the environment decreases, the system will adaptively relax the policy constraints, allowing the agent to make more flexible adjustments based on specific task requirements while ensuring stability.

[0028] Algorithm 1 SMPPO algorithm Input: Maximum number of steps , discount factor , GAE parameters , batch size , the number of parallel environments = 1 \* GB3 ① Initialization Parallel environment = 2 \* GB3 ② = 3 \* GB3 ③ = 4 \* GB3 ④ According to the strategy renew , select an action = 5 \* GB3 ⑤Interact with the environment and get = 6 \* GB3 ⑥end for = 7 \* GB3 ⑦ Collect samples = 8 \* GB3 ⑧ Computational advantage = 9 \* GB3 ⑨ end for = 10 \* GB3 ⑩ Aggregate samples from all parallel environments, generate random permutations, and divide the total samples according to Divided into indivual ⑪for K , ⑫Yes Standardize the advantages. ⑬Update using current batch data , , .

[0029] ⑭ Update parameters using gradient descent ⑮end for ⑯ Build a communication diagram; According to the above definition 2, a communication graph is constructed, such as Figure 3In the graph neural network, the adjacency matrix of the graph shift operator (GSO) is used to represent the dynamic communication topology at time t. In the communication mechanism of the present invention, three graph shift operators are constructed to represent the communication topology of the agent at time t with its selfish neighbor agents. , communication topology with selfless neighbor agents Communication topology with neutral neighbor agents (14) in Among them, if And the agent In the intelligent body If the vision is a selfish agent type, then set , others Algorithm 2: Constructing a communication graph Input: N agents, distance threshold , the social preference parameter of each agent Output: Each agent , near_agent = 1 \* GB3 ① = 2 \* GB3 ② Initialization , , , near_agent = 3 \* GB3 ③ = 4 \* GB3 ④ Computational Agent and agents Distance = 5 \* GB3 ⑤if = 6 \* GB3 ⑥ The intelligent agent Join near_agent = 7 \* GB3 ⑦end if = 8 \* GB3 ⑧if then = 9 \* GB3 ⑨ = 10 \* GB3 ⑩else if then ⑪ ⑫else: ⑬end if ⑭end for ⑮end for SVO_GAT; Based on the established communication topology, this paper proposes a new multi-head attention communication mechanism, SVO-GAT, which aggregates observation function messages. Formally, this paper defines SVO-GAT as follows: (15) is the attention matrix at time t, Is a weight matrix that is independent and learnable in each attention head. The element values ​​of are calculated as follows: (16) Calculate the score of each communication edge (17) Output features Used to predict time SVO-GAT considers information from different types of agents in the decision-making process.

[0030] In addition, borrowing research ideas, SVO-GAT (must ensure the stability of graph attention convolution operations, consider the dynamic neighborhood changes within the agent's perception field (FOV) (including the entry and exit of agents), and its communication topology will exhibit time-varying characteristics. Under this condition, this paper focuses on the following core properties: 1) Permutation Equivariance: As a core feature of SVO-GAT, permutation equivariance ensures that when the order of nodes in the communication graph is arbitrarily permuted (i.e., the order of agent numbers is rearranged), the network can still maintain the topological invariance of the attention mechanism.

[0031] The present invention defines a permutation operation , used to exchange the index of the agent. :express Represents the node feature matrix at time step t. If the node feature matrix is Graph shift operator : It is a re-representation of rows and columns at time step t. This operation is equivalent to re-labeling the nodes in the graph, thus ensuring the topological invariance of the graph convolution operation.

[0032] Lemma 1. Given a permutation , the convolution operation of GCN The following equation needs to be satisfied: : (18) For any permutation operation of the SVO-GAT defined in (15) and attention convolution operation , the following equation holds: Proof 1: Only local observation features and When the agent index changes, the rows and columns of the attention matrix change with the agent observation feature matrix Therefore, the attention matrix :Changes to . Can be displayed in SVO-GAT: Similarly, we can get: , 2) Time invariance: SVO-GAT must ensure that under the same input conditions (i.e. the same feature matrix and the same graph structure ), whenever an attention convolution operation is performed, its output remains consistent. This means that if the historical state is replayed, the network will produce exactly the same output, ensuring the stability and repeatability of the decision.

[0033] Theorem 2: Assume , , , and the attention convolution operation of SVO-GAT is defined in (15) , then the following equation holds (20) Proof 2: The partially observable DRL-based LMAPF strategy of the present invention essentially meets this criterion: the output of all agents at time t is only related to the input content.

[0034] Example 2: Based on Example 1, further experiments are disclosed.

[0035] This paper first provides details of the experimental setup and then introduces the metrics used in the evaluation. Next, we systematically compare the results of LMAPF and MAPF on different maps for agent sizes ranging from 8–512 (scaling up to 768). Overall, this paper aims to address the following research questions: RQ: Can SPGLP effectively break deadlock and livelock? Can it improve the average throughput of LMAPF? Can it improve the success rate of MAPF in reaching its target? Experimental environment: The experiment was implemented in Python and used an NVIDIA GeForce RTX 4090 GPU. During training and evaluation, we maintained the same map size and number of agents, but used randomly generated obstacle layouts and agent positions to ensure diverse test scenarios. The experimental map is shown below: Evaluation criteria: 1) Deadlock and Livelock: Deadlock detection relies on the agent's four-step quiescence to determine if deadlock has occurred. However, in practice, SPGLP often breaks deadlock after the agent has reached three steps of quiescence, resulting in a relatively low number of instances of agent deadlock. To more accurately assess the agent's behavioral efficiency and the overall performance of the system, the number of deadlock and livelock occurrences is combined. This comprehensively reflects the difficulties that the agent may encounter when performing tasks: (twenty one) 2) The LMAPF goal is to maximize the average throughput, that is, the average number of targets reached per time step (twenty two) in is the number of goals reached by all agents, is the termination time step L 3) MAPF: Since the present invention focuses on solving the problem of lifelong path planning, it is also compared with one-time path planning. Success rate: When all agents reach the goal within the specified maximum time step, they succeed, which is defined as: (twenty three) in, is the number of successful instances, is the number of all MAPF instances. In this invention, we set the number of all instances to 128. In MAPF, Makespan is used to measure model efficiency. Makespan refers to the maximum time step for all agents to reach the target position: (twenty four) in is the number of agents, It is an intelligent agent For all successful test instances, the present invention uses the average completion time to measure the overall efficiency of the method, which can be defined as: (25) in Indicates the The maximum completion time of each instance.

[0036] Comparative experiment: We compare SPGLP with the state-of-the-art (L)MAPF method for LMAPF and MAPF scenarios in various maps with different obstacle densities and number of agents. The baselines are detailed in Table 2.

[0037] Table 2 Comparison of SPGPL and baseline Model Learning Comm Guidance PAIMAL2 DRL+IL NO Subgoal vector SCIRIMP DRL YES Subgoal vector FOLLOWER DRL+A* NO Subgoal vector + variant A* SPGLP DRL+SVO YES Subgoal vector + heuristic vector In order to explore the problem of RQ, the present invention designed a systematic experimental scheme: in a random map with 10%-30% obstacle density (such as Figure 4 In (b), the number of agents ranges from 8 to 64, and 20 rounds of training are performed to count the occurrence of system deadlock and livelock. The results are shown in Figure 5 , Figure 6 As shown (the shaded area represents the error between the evaluation round results and the average training round results), due to insufficient initial experience, the policy framework is inefficient in resolving deadlocks and livelocks. However, as experience accumulates, the system gradually optimizes its decisions through online learning. When the number of agents is fixed at 64, the system's DLR shows a significant downward trend as the obstacle density increases. At a 10% obstacle density, the DLR significantly decreases to below 3%, and at a 30% obstacle density, it significantly decreases to below 7%. At 30% obstacle density, the DLR for all experimental groups with different agent numbers drops below 7%.

[0038] Table 3 shows the average throughput for a small number of agents on random maps. Compared to the PAIMAL2 and FOLLOWER algorithms, SPGLP demonstrates significant throughput advantages across all experimental conditions. SPGLP's performance advantage becomes increasingly pronounced with increasing environment complexity and agent size. At a 30% obstacle density, throughput for 64 agents increases by 46%. In the absence of static obstacles, throughput reaches a maximum of 30%. This demonstrates that SPGLP improves system throughput by breaking deadlock and livelock.

[0039] The present invention also tests the average throughput of random maps in a more challenging large-scale intelligent volume scenario (obstacle density 30%, 128, 256, 512, 768), and the results are as follows: Figure 8 As shown (the shaded area indicates the 95% confidence interval), SPGLP still exhibits a significant performance advantage. SPGLP improves the baseline by 22% on 512 agents. Overall, when the number of agents is not low, SPGLP outperforms FOLLOWER and PRIMAL2.

[0040] Table 3 Comparison of SPGLP with PAIMAL2 and FOLLOWER on random maps with different obstacle densities.

[0041] Maze map Figure 4 (a) Due to the narrow passages and dense path intersections, deadlock and livelock problems frequently occur. To further verify that the algorithm can effectively break deadlock and livelock, the present invention conducted comparative experiments on maze maps with 16-256 agents and different corridor widths. The results are shown in the figure. Figure 7 As shown in Figure 2, compared with the PAIMAL2 and FOLLOWER algorithms, the average throughput of SPGLP is improved by about 8%.

[0042] In order to verify the performance of SPGLP in industrial scenarios, a standard warehouse environment (such as Figure 4 (d)) The average throughput performance of the number of detection agents is 8-768. The results are as follows Figure 9 As shown in the figure, the throughput of medium-scale (64-256 agents) is improved by 7% compared to the baseline; in the ultra-large-scale (512,768 agents) scenario, the throughput is improved by 11% compared to the baseline. These results fully demonstrate that SPGLP can effectively meet the needs of industrial environments. In order to verify the performance of SPGLP in the game scene, the present invention is based on the lak303d map of Moving AI (such as Figure 4 (c)) was tested to evaluate the average throughput when the number of agents ranged from 8 to 512. Since the average throughput when the baseline number of agents was 8 was too low, this data point was not recorded. Figure 10 As shown in Figure 3), compared to the baseline method FOLLOWER, SPGLP’s throughput is improved by 20%.

[0043] In the multi-agent path planning (MAPF) scenario, this paper compares two baseline algorithms, PAIMAL2 and SCIRIMP, respectively, on the warehouse map (e.g. Figure 4 (d)) and random maps (e.g. Figure 4(b)) The test was conducted in the environment, where the maximum step size of 8-128 agents was set to 256, while the maximum step size of 256 and 512 agents was set to 512, in order to evaluate the maximum time step and success rate of SPGLP. The experimental results are shown in Figure 11 SPGLP can still maintain a high success rate in high-density agent scenarios (such as 256+), but its maximum time step is increased compared to the baseline algorithm, which shows that SPGLP sacrifices time efficiency to a certain extent when optimizing the success rate of path planning. Subsequent research will focus on optimizing the algorithm's time performance to achieve a better balance between success rate and time efficiency. Table 4 describes the maximum time step under the warehouse map. Table 4 Comparison of the maximum time step of SPGLP, PAIMAL2 and SCIRIMP under different maps.

[0044] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent path planning method based on the fusion of social value orientation and DRL, characterized by: include: Step S1: construct a multi-channel tensor representation based on local observations, spatially encode dynamic information, and establish an interpretable thinking feature space; Step S2: Based on the SVO-based heterogeneous graph attention network communication mechanism, it receives dynamic information from neighboring agents, separates and aggregates the information through the multi-head SVO perception heterogeneous graph attention network, and extracts the most critical information for its own decision-making; Step S3: Based on the SVO two-layer decision-making mechanism of the neighboring intelligent agent group, the SVO and PPO algorithms are combined, and the local environment and the neighboring intelligent agent group are used to dynamically adjust the SVO strategy to achieve the dynamic changes of the lower-level action strategy adaptive environment.

2. The multi-agent path planning method based on the integration of social value orientation and DRL according to claim 1 is characterized in that: The SVO two-tier decision-making mechanism in step S3 extends the traditional static SVO concept into a dynamic attribute, enabling it to make real-time adjustments based on the local environment state of the agent and the interaction relationship between the neighboring agent group; Among them, the social value orientation of the intelligent agent is defined as: When the number of agents is For any agent in a multi-agent system , whose personality score satisfy , each agent The social value orientation perspective is: 。 3. The multi-agent path planning method based on the integration of social value orientation and DRL according to claim 2 is characterized in that: The role attributes of the agent are dynamically defined by the relative relationship between its own SVO value and neighboring agents, and a role classification method based on local perspective is further proposed, including: If the agent The SVO value is , agent There is a set of neighboring agents , agent , its SVO value is ; when When the agent Compared to intelligent agents Pay more attention to self-interest, that is, in the intelligent body From the perspective of intelligent agents Is a selfish intelligent agent; when , agent Compared to intelligent agents Pay more attention to collective interests, that is, in the intelligent body From the perspective of intelligent agents Is a selfless intelligent agent; when , agent With the agent Adopt a balanced strategy based on historical experience, that is, Vision, intelligent agent Is a neutral class agent; The same agent presents different roles from the perspectives of different agents. This relativity can well reflect the complexity of the multi-agent system and echoes the multifaceted nature of individual roles in human society. The roles and behaviors of the agents will dynamically adjust with the group structure in which they are located, further affecting the overall emergent characteristics of the system.

4. The multi-agent path planning method based on the fusion of social value orientation and DRL according to claim 3 is characterized in that: The complexity of the agent's environment can be modeled as a function of the neighboring agent group and its own state, and its mathematical expression is as follows: in, For intelligent agents The distance to the neighboring agent is less than The number of The larger the size, the more complex the environment. is the state of the agent. If the agent is in deadlock, is 1, if the agent is in livelock, then is the number of livelock steps, otherwise is 0.

5. The multi-agent path planning method based on the integration of social value orientation and DRL according to claim 4 is characterized in that: The step S3 uses the local environment and the neighboring agent group to dynamically adjust the SVO strategy, and the strategy network dynamically generates the personality score by real-time perception of local observation information and the characteristics of neighboring agents. , and mapped to SVO angle value ; SVO Strategy As an additional layer embedded in the policy network structure, the SVO strategy and the action strategy share feedback signals, realizing parallel training and collaborative optimization of the two strategies. Through the reward function, the intelligent agent can dynamically adjust its SVO value to adapt to different environmental states and select efficient actions based on the SVO strategy, effectively breaking the behavioral homogeneity problem caused by the shared policy network.

6. The multi-agent path planning method based on the integration of social value orientation and DRL according to claim 5 is characterized in that: The reward function includes: The SVO reward function calculates the average of the basic rewards of an agent and other agents in its neighborhood, establishing a dynamic balance between individual interests and group interests, guiding agents to choose social value orientations with a cooperative tendency. The action reward function is based on the SVO theoretical framework, which realizes the adaptive adjustment of the agent's behavior strategy. That is, in a low-density local environment, the agent maximizes the long-term cumulative reward. To optimize the efficiency of individual path planning; in high-density complex environments, agents tend to be pro-social, by maximizing To coordinate the conflict resolution among multiple agents, this effectively solves the behavior homogeneity problem caused by the fixed strategy network in traditional methods, and ensures the adaptability of the system in different environmental conditions through a dynamic adjustment mechanism; By maximizing the accumulation and , find the optimal SVO strategy and action strategy, 。 7. The multi-agent path planning method based on the integration of social value orientation and DRL according to claim 6 is characterized in that: A variant of the PPO algorithm in step S3 supports the training of the intelligent agent, namely: SVO perception multi-strategy PPO constructs SMPPO, and SMPPO needs to optimize the SVO strategy at the same time and action strategies , and the optimization objectives of the two are executed independently, and the loss function associated with each strategy is formalized as and ; and is the advantage function, which measures the superiority of a specific SVO strategy or action strategy relative to the average strategy in a given state; By limiting the probability ratio of new and old strategies The scope of change to avoid excessive policy updates; Using Hyperparameters Define the clipping interval and use the min operation to combine the original probability ratio and the clipped value to optimize the strategy target, thereby improving sample efficiency while ensuring training stability; The SVO policy loss is calculated through the action advantage function, and the action policy loss is calculated through the SVO advantage function to ensure that the behavior of the agent is consistent with the SVO decision.

8. The multi-agent path planning method based on the integration of social value orientation and DRL according to claim 7 is characterized in that: The SMPPO encourages action strategies to overcome short-sightedness and continue to follow the guidance of the SVO strategy through policy gradient optimization. The gradient optimization is as follows: This two-way reinforcement between the SVO strategy and the action strategy creates an environment for collaborative learning. It not only achieves the alignment between individual behavior and collective goals, but also dynamically optimizes the SVO strategy based on the results of the actions, thereby directly forming a continuous and mutually beneficial positive feedback between SVO decision-making and action execution.

9. The multi-agent path planning method based on the integration of social value orientation and DRL according to claim 8 is characterized in that: In order to improve the SVO stability of the agent, a supervised learning paradigm is adopted to construct the SVO stability enhancement loss The formula can be expressed as follows: Is the cross entropy between the SVO policy output by the neural network and the expected SVO probability distribution. Expected SVO value is the current value and previous value The weighted combination of gets: By introducing an under-relaxation factor based on the complexity of the environment , change the current SVO Compared with the previous SVO Mixing to determine expected SVO Stabilize the loss function; ensure that SVO adjusts to the expected value and achieves dynamic and smooth updates of SVO; is the boundary condition of environmental complexity; when the actual environmental complexity between agents is lower than When the complexity of the environment decreases, the system will adaptively relax the policy constraints, allowing the agent to make more flexible adjustments according to specific task requirements while ensuring stability.

Images