Pedestrian path planning method and device based on reinforcement learning and medium

Abstract

The present disclosure provides a pedestrian path planning method and device based on reinforcement learning and a medium. The pedestrian path planning method based on reinforcement learning comprises: acquiring a pedestrian path data set and position coordinates, translation speed and angular velocity of a pedestrian in a to-be-tested crowd corresponding to a current time; performing model training on a preset fluid dynamics model to determine a crowd simulation model based on fluid dynamics; predicting the movement speed of the to-be-tested pedestrian at the next time based on the crowd simulation model based on fluid dynamics; and feeding back the predicted movement speed of the to-be-tested pedestrian at the next time to a simulation environment according to a dynamic loss function and a reward function to simulate the movement of the pedestrian in the to-be-tested crowd. Through the present disclosure, the crowd simulation model based on fluid dynamics determines the expectation of the agent speed to reach the destination, prevents the agent from colliding, and improves the authenticity and effectiveness of the crowd movement simulation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a pedestrian path planning method, apparatus, and medium based on reinforcement learning. Background Technology

[0002] Crowd path planning is of great significance in various fields such as architecture, urban planning, and safety management. Currently, crowd path planning still faces challenges in terms of realism, motion planning, and interactivity. In terms of realism, it is difficult to simulate human movements and posture changes; in terms of motion planning, it is difficult to capture pedestrians' intentions in the environment and make decisions about their movements; and in terms of interactivity, it is difficult to simulate the interactions between pedestrians, the environment, and other people.

[0003] From the perspective of the target audience, existing crowd simulation models include rule-based methods and agent-based methods. Rule-based methods determine the overall movement trend and simulate pedestrian movement through predefined rules, while agent-based methods simulate the behavioral decisions of each individual in the crowd and model the behavior of the entire crowd through the interactions between individuals. However, rule-based methods lack realism, as their rules and constraints are hard-coded, making optimization difficult and resulting in low simulation efficiency and an inability to handle large-scale scenarios. Agent-based methods have high computational complexity, are sensitive to initial conditions, and focus on the behavior of individual agents rather than the behavior of the entire group, thus failing to accurately simulate social interactions in complex scenarios or capture the global behavior of the group.

[0004] In densely populated scenarios, even when combining rule-based and agent-based methods to simulate crowd behavior, challenges remain, including high computational complexity and difficulties in model validation. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the purpose of this disclosure is to provide a pedestrian path planning method, device and medium based on reinforcement learning.

[0006] To achieve the above objectives, according to a first aspect of this disclosure, a pedestrian path planning method based on reinforcement learning is provided, comprising:

[0007] Obtain the pedestrian path dataset and the current position coordinates, translational velocity, and angular velocity of pedestrians in the test group. The pedestrian path dataset includes the position coordinates, translational velocity, and angular velocity of pedestrians at each moment.

[0008] The fluid dynamics model is trained based on the pedestrian path dataset to determine a crowd simulation model based on fluid dynamics.

[0009] The current position coordinates, translational velocity, and angular velocity of the pedestrians in the crowd to be tested are input into the crowd simulation model based on fluid dynamics to predict the movement velocity of the pedestrians in the next moment.

[0010] The predicted speed of the pedestrian in the next moment is fed back to the simulation environment based on the dynamic loss function and the preset reward function, thus simulating the movement of pedestrians in the crowd.

[0011] Optionally, the method further includes:

[0012] The dynamic loss function is determined based on the predicted next-move speed of the pedestrian under test by the crowd simulation model based on fluid dynamics, the actual next-move speed, and the preset loss function term.

[0013] Optionally, determining the dynamic loss function based on the pedestrian's predicted speed at the next moment, the actual speed at the next moment, and a preset loss function term, according to the crowd simulation model based on fluid dynamics, includes:

[0014] DL = MSE(v,p)

[0015]

[0016] Wherein, DL represents the preset loss function term, i.e., dynamic loss, v represents the actual movement speed at the next moment, p represents the movement speed of the pedestrian predicted by the crowd simulation model based on fluid dynamics at the next moment, Loss represents the dynamic loss function, PL represents the policy gradient loss, n represents the number of pedestrians in the preset radius area, EL represents the entropy loss, α represents the rotation angle between the vector connecting the pedestrians in the preset angle area directly in front of the pedestrian and the pedestrian, and VL represents the value loss.

[0017] Optionally, the preset reward function includes:

[0018]

[0019] R = a × DR - b × CR + FR - SR - c (c > 0)

[0020]

[0021]

[0022] Here, FR represents follow-up reward. Indicates the direction of movement of the pedestrian being measured. d represents the direction of movement of the other pedestrian, γ represents the distance between the pedestrian being tested and the other pedestrian, γ represents the coefficient of the weight of the follow reward value, a represents the weight of the pedestrian reaching the destination, b represents the weight of the collision between the pedestrian being tested and the other pedestrian, DR represents the sign that the pedestrian being tested has reached the destination, CR represents the sign that the pedestrian being tested has collided with the other pedestrian, SR represents the square of the difference between the actual speed and the preset speed of the pedestrian being tested, and c represents the overspeed penalty of the pedestrian being tested at each time step.

[0023] Optionally, the step of training the fluid dynamics model based on the pedestrian path dataset to determine a crowd simulation model based on fluid dynamics includes:

[0024] The deformation rate tensor of the pedestrian is determined based on the position coordinates, translational velocity, and angular velocity of the pedestrian at each moment in the pedestrian path dataset.

[0025] The deformation rate tensor, the position coordinates, translational velocity, and angular velocity of the pedestrians in the pedestrian path dataset at each moment are input into the fluid dynamics model for model training, thereby determining a crowd simulation model based on fluid dynamics.

[0026] Optionally, the step of training the fluid dynamics model based on the pedestrian path dataset to determine the crowd simulation model based on fluid dynamics further includes:

[0027] The preset fluid dynamics model includes:

[0028] V=V0+E·δr+ω×δr

[0029] Where V represents the pedestrian's movement speed, V0 represents the pedestrian's translational speed, E represents the deformation rate tensor, δr represents the length infinitesimal vector, and ω represents the pedestrian's angular velocity;

[0030] The deformation rate tensor E includes:

[0031]

[0032] in, Let represent the angular velocity of one side of the absolute bounding box b of the pedestrian. Let b represent the angular velocity of the other adjacent side of the absolute bounding box b of the pedestrian. x b' represents the component of the absolute bounding box b of the pedestrian at the current moment projected onto the x-axis of the Cartesian coordinate system. x b represents the component of the absolute bounding box b of the pedestrian projected onto the x-axis of the Cartesian coordinate system at the next moment. yb' represents the component of the absolute bounding box b of the pedestrian at the current moment projected onto the y-axis of the Cartesian coordinate system. y b represents the component of the absolute bounding box b of the pedestrian projected onto the y-axis of the Cartesian coordinate system at the next moment. z b' represents the component of the absolute bounding box b of the pedestrian at the current moment projected onto the z-axis of the Cartesian coordinate system. z This represents the component of the absolute bounding box b of the pedestrian projected onto the z-axis of the Cartesian coordinate system at the next moment.

[0033] The crowd simulation model based on fluid dynamics includes:

[0034]

[0035] Among them, v 预 v represents the predicted velocity of the pedestrian in the next moment. 0x v represents the component of the pedestrian's translational velocity along the x-axis at the current moment. 0y v represents the component of the pedestrian's translational velocity along the y-axis at the current moment. 0z ω represents the component of the pedestrian's translational velocity along the z-axis at the current moment. x ω represents the component of the pedestrian's angular velocity at the current moment projected onto the x-axis of the Cartesian coordinate system. y ω represents the component of the pedestrian's angular velocity at the current moment projected onto the y-axis of the Cartesian coordinate system. z This represents the component of the pedestrian's angular velocity at the current moment projected onto the z-axis of the Cartesian coordinate system.

[0036] Optionally, the method further includes:

[0037] Based on Gauss's law for the velocity field of the crowd to which the pedestrian is located, determine the variance of the crowd velocity.

[0038] Determine the area covered by pedestrians within the crowd of the pedestrian being tested, who walk at the same preset speed over a preset time period;

[0039] The model quality of the crowd simulation model based on fluid dynamics is determined by the variance of the crowd speed of the pedestrian being tested and the area covered by pedestrians in the crowd being tested walking at the same preset speed within a preset time period.

[0040] Optionally, determining the variance of the crowd velocity of the crowd to which the pedestrian to be tested is located, based on Gauss's law for the velocity field of the crowd, includes:

[0041] Let Ω' be a closed region consisting of macro-infinitesimals and micro-finitesimals formed by piecewise smooth surfaces Σ':

[0042]

[0043] Where σ represents the variance of the velocity of the crowd to which the pedestrian to be tested belongs, and v represents the velocity field of the crowd to which the pedestrian to be tested belongs.

[0044] According to a second aspect of this disclosure, a pedestrian path planning device based on reinforcement learning is provided, comprising:

[0045] The acquisition module is used to acquire the pedestrian path dataset and the current position coordinates, movement speed and angular velocity of pedestrians in the test group. The pedestrian path dataset includes the position coordinates, translation speed and angular velocity of pedestrians at each moment.

[0046] The model training module is used to train the fluid dynamics model based on the pedestrian path dataset to determine a crowd simulation model based on fluid dynamics.

[0047] The model prediction module is used to input the current position coordinates, movement speed and angular velocity of the pedestrians in the crowd to be tested into the crowd simulation model based on fluid dynamics, and predict the movement speed of the pedestrians in the next moment.

[0048] The crowd motion simulation module is used to feed back the predicted movement speed of the pedestrian under test to the simulation environment based on the dynamic loss function and reward function, thereby simulating the movement of pedestrians in the crowd under test.

[0049] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the pedestrian path planning method based on reinforcement learning provided in the first aspect of this disclosure.

[0050] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0051] Through the above technical solution, this disclosure combines rule-based and agent-based methods to simulate pedestrian movement. The rule-based method develops a crowd simulation model based on fluid dynamics, treating each pedestrian in the crowd as a fluid particle. The physical equations of fluid particle motion, i.e., a fluid dynamics model, guides pedestrian movement and predicts the pedestrian's speed at the next moment, improving the accuracy of the simulated pedestrian movement. The agent-based method uses a dynamic loss function and a preset reward function to feed back the predicted speed of the pedestrian at the next moment to the simulation environment. Pedestrians interact with the environment in real time to realistically and effectively simulate pedestrian movement in the crowd, improving the realism of the simulated pedestrian movement. This disclosure improves pedestrian movement planning by determining the agent's speed to achieve its desired destination and preventing collisions, thus realizing pedestrian movement simulation and improving the realism and effectiveness of crowd movement simulation. Attached Figure Description

[0052] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0053] Figure 1 This is a flowchart illustrating a pedestrian path planning method based on reinforcement learning, according to an exemplary embodiment.

[0054] Figure 2 This is a flowchart illustrating a pedestrian path planning method based on reinforcement learning, according to another exemplary embodiment.

[0055] Figure 3 This is a top view of the absolute bounding box of a pedestrian according to an exemplary embodiment.

[0056] Figure 4 This is a verification diagram of four population simulation models illustrated according to an exemplary embodiment.

[0057] Figure 5 This is a block diagram illustrating a pedestrian path planning device based on reinforcement learning, according to an exemplary embodiment. Detailed Implementation

[0058] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0059] This disclosure uses the CARLA platform and relies on the map provided by the CARLA platform. It uses the OpenAI Gym reinforcement learning environment to train the reinforcement learning algorithm with an end-to-end stable baseline, thereby realizing the pedestrian path planning method based on reinforcement learning provided in this disclosure.

[0060] Figure 1 This is a flowchart illustrating a pedestrian path planning method based on reinforcement learning, according to an exemplary embodiment. Figure 1 As shown, a pedestrian path planning method based on reinforcement learning includes S11 to S14.

[0061] S11, obtain the pedestrian path dataset and the current position coordinates, translational velocity, and angular velocity of the pedestrians in the test group at the current moment.

[0062] The pedestrian path dataset includes the pedestrian's position coordinates, translational velocity, and angular velocity at each moment.

[0063] As an example, the population to be tested in this disclosure consists of pedestrians randomly positioned at random locations on a map, moving toward random destinations.

[0064] S12 trains the fluid dynamics model based on the pedestrian path dataset to determine a crowd simulation model based on fluid dynamics.

[0065] The fluid dynamics model includes:

[0066] V=V0+E·δr+ω×δr

[0067] Where V represents the pedestrian's movement speed, V0 represents the pedestrian's translational speed, E represents the deformation rate tensor, δr represents the length infinitesimal vector, and ω represents the pedestrian's angular velocity.

[0068] The pedestrian path dataset, including the pedestrian's current position coordinates, translational velocity, and angular velocity, is input into the fluid dynamics model. The model outputs the predicted velocity of the pedestrian in the next moment, which is used to train the fluid dynamics model and determine the crowd simulation model based on fluid dynamics.

[0069] S13: Input the current position coordinates, translational velocity, and angular velocity of the pedestrian in the crowd to be tested into the crowd simulation model based on fluid dynamics to predict the movement velocity of the pedestrian in the next moment.

[0070] S14: Based on the dynamic loss function and the preset reward function, the predicted speed of the pedestrian in the next moment is fed back to the simulation environment to simulate the movement of pedestrians in the crowd.

[0071] Based on a crowd simulation model using fluid dynamics, the movement velocity of each person in the target crowd at the next moment is determined. Then, based on a dynamic loss function and a pre-defined reward function, the movement velocity of each person at the next moment is fed back into the simulation environment for display. This allows for the creation of dense crowds on a map and the simulation of crowd movement within a short time. The simulation environment can visually demonstrate the crowd movement patterns guided by the fluid dynamics-based crowd simulation model.

[0072] In this disclosure, a dynamic loss function is used to determine the loss value between the predicted and actual movement velocities of pedestrians in the next time step using a crowd simulation model based on fluid dynamics. Backpropagation is then used to update the parameters of the crowd simulation model to reduce the loss between the predicted and actual movement velocities, thus reinforcing the model. A pre-defined reward function is used to draw an analogy between the behavior of pedestrians in the crowd and the behavior of fluid particles, maintaining relative consistency in direction among pedestrians and encouraging consistency and clustering.

[0073] A preset reward function is used to encourage pedestrians to move in the same direction. In some possible embodiments, the preset reward function includes:

[0074]

[0075] R = a × DR - b × CR + FR - SR - c (c > 0)

[0076]

[0077]

[0078] Here, FR represents follow-up reward. Indicates the direction of movement of the pedestrian being measured. d represents the direction of movement of the other pedestrian, γ represents the distance between the pedestrian being tested and the other pedestrian, γ represents the coefficient of the weight of the follow reward value, a represents the weight of the pedestrian being tested reaching the destination, b represents the weight of the collision between the pedestrian being tested and the other pedestrian, DR represents the sign that the pedestrian being tested has reached the destination, CR represents the sign that the pedestrian being tested has collided with the other pedestrian, SR represents the square of the difference between the actual speed and the preset speed of the pedestrian being tested, and c represents the overspeed penalty of the pedestrian being tested at each time step.

[0079] Through the above technical solution, this disclosure combines rule-based and agent-based methods to simulate pedestrian movement. The rule-based method develops a crowd simulation model based on fluid dynamics, treating each pedestrian in the crowd as a fluid particle. The physical equations of fluid particle motion, i.e., a fluid dynamics model, guides pedestrian movement and predicts the pedestrian's speed at the next moment, improving the accuracy of the simulated pedestrian movement. The agent-based method uses a dynamic loss function and a preset reward function to feed back the predicted speed of the pedestrian at the next moment to the simulation environment. Pedestrians interact with the environment in real time to realistically and effectively simulate pedestrian movement in the crowd, improving the realism of the simulated pedestrian movement. This disclosure improves pedestrian movement planning by determining the agent's speed to achieve its desired destination and preventing collisions, thus realizing pedestrian movement simulation and improving the realism and effectiveness of crowd movement simulation.

[0080] Figure 2 This is a flowchart illustrating a pedestrian path planning method based on reinforcement learning, according to another exemplary embodiment. Figure 2 As shown, in some possible embodiments, the pedestrian path planning method based on reinforcement learning also includes S15.

[0081] S15. Based on the predicted movement speed of the pedestrian at the next moment, the actual movement speed at the next moment, and the preset loss function term of the crowd simulation model based on fluid dynamics, determine the dynamic loss function.

[0082] The dynamic loss function includes:

[0083] DL = MSE(v,p)

[0084]

[0085] Wherein, DL represents the preset loss function term, i.e., dynamic loss, v represents the actual movement speed of the pedestrian in the next moment, p represents the movement speed of the pedestrian in the next moment predicted by the crowd simulation model based on fluid dynamics, Loss represents the dynamic loss function, PL represents the policy gradient loss, n represents the number of pedestrians in the preset radius area, EL represents the entropy loss, α represents the rotation angle between the vector connecting the pedestrians in the preset angle area directly in front of the pedestrian and the pedestrian in the test, and VL represents the value loss.

[0086] In one possible embodiment, the dynamic loss includes policy gradient loss (PL), entropy loss (EL), and value loss (VL). The total loss predicted by the pedestrian's movement speed at the next moment using a crowd simulation model based on fluid dynamics is a weighted sum of policy gradient loss (PL), entropy loss (EL), and value loss (VL).

[0087] In another possible embodiment, the number n of pedestrians within a preset radius area, the rotation angle α between the vector connecting the pedestrians within a preset angle area directly in front of the pedestrian and the pedestrian under test can be preset. In this disclosure, the preset angle of the pedestrian under test within the preset angle area directly in front can be set to 180°, and the rotation angle α is normalized to [0°, 90°].

[0088] By defining n to adjust the entropy loss, defining α to adjust the value loss, and adding a preset loss function term DL, the dynamic loss function Loss of the crowd simulation model based on fluid dynamics is determined.

[0089] Through the above technical solutions, using agent-based and rule-based methods, and employing the PPO reinforcement learning framework of OpenAI Gym, the prediction results of the crowd simulation model based on fluid dynamics are incorporated into the dynamic loss function of the RL process. This allows pedestrians to interact with the environment in real time, improving the realism of simulated pedestrian movement.

[0090] In some possible embodiments, the step of training the fluid dynamics model based on the pedestrian path dataset to determine the crowd simulation model based on fluid dynamics includes S121 to S122.

[0091] S121, determine the pedestrian deformation rate tensor based on the pedestrian's position coordinates, translational velocity, and angular velocity at each moment in the pedestrian path dataset.

[0092] In one possible embodiment, the pedestrian's position coordinates are set as L, translational velocity as v, and angular velocity as ω, where the pedestrian's position coordinates are variables. The pedestrian's position coordinates, translational velocity, and angular velocity are mapped to a Cartesian coordinate system, where the x-axis component of the pedestrian's current position coordinate L in the Cartesian coordinate system is Lx. x The component along the y-axis is L. y The component along the z-axis is L. z The component of the pedestrian's translational velocity v in the Cartesian coordinate system at the current moment along the x-axis is v0. x The component along the y-axis is v. y The component along the z-axis is v z The components of the pedestrian's angular velocity ω at the current moment along the x-axis in the Cartesian coordinate system are w x The component along the y-axis is w y The component along the z-axis is w z .

[0093] Define the absolute bounding box of a pedestrian as b. Mapping the absolute bounding box of a pedestrian to a Cartesian coordinate system, the absolute bounding box of a pedestrian is defined as b, where b is the x-axis component. x The component along the y-axis is b. y The component along the z-axis is b. z .

[0094] Figure 3 This is a top view of the absolute bounding box of a pedestrian according to an exemplary embodiment.

[0095] like Figure 3 As shown, in the Cartesian coordinate system, quadrilateral OABC is defined by looking down at the xy-plane from the z-axis, representing the absolute bounding box of the pedestrian. Point O is the origin, point B lies on the xy-plane, point A is located at [location missing], and point C lies on the xy-plane. Sides OA and OB represent adjacent sides of quadrilateral OABC.

[0096] definition: The elongation per unit length is:

[0097]

[0098] Where, ∈ xx The linear deformation rate of the absolute bounding box b of the pedestrian in the x-axis direction, ∈ yy The linear deformation rate in the y-axis direction represents the absolute bounding box b of the pedestrian, ∈ zz The linear deformation rate in the z-axis direction represents the absolute bounding box b of the pedestrian.

[0099] Therefore, the angular velocity of OA is:

[0100]

[0101] The angular velocity of OB is:

[0102]

[0103] Based on the approximate analysis, the pedestrian's height is constant, so angular deformation along the z-axis can be ignored. Assume the pedestrian's centroid coincides with the center of the pedestrian's absolute bounding box b.

[0104] As another example, the elements on the main diagonal of the matrix of the deformation rate tensor E represent the linear deformation rates in the x-axis, y-axis, and z-axis directions, respectively, and the elements on the second diagonal represent the angular deformation rates.

[0105] If we disregard deformation along the z-axis, then element e xz e yz e zx e zy The value is 0. Based on symmetry, it is determined that...

[0106]

[0107] Among them, e xy e represents the angular deformation rate of the absolute bounding box b of the pedestrian in the x-axis direction relative to the y-axis direction. yx The absolute bounding box b of the pedestrian represents the angular deformation rate in the y-axis direction relative to the x-axis direction.

[0108] The deformation rate tensor E of the pedestrian is:

[0109]

[0110] S122, the deformation rate tensor, the position coordinates, translational velocity and angular velocity of pedestrians in the pedestrian path dataset at each time step are input into the fluid dynamics model for model training, and the crowd simulation model based on fluid dynamics is determined.

[0111] The preset fluid dynamics model includes:

[0112] V=V0+E·δr+ω×δr

[0113] Where V represents the pedestrian's movement speed, V0 represents the pedestrian's translational speed, E represents the deformation rate tensor, δr represents the length infinitesimal vector, and ω represents the pedestrian's angular velocity;

[0114] The deformation rate tensor E includes:

[0115]

[0116] in, Let represent the angular velocity of one side of the absolute bounding box b of the pedestrian. Let b represent the angular velocity of the other adjacent side of the absolute bounding box b of the pedestrian. x b' represents the component of the absolute bounding box b of the pedestrian at the current moment projected onto the x-axis of the Cartesian coordinate system. x b represents the component of the absolute bounding box b of the pedestrian projected onto the x-axis of the Cartesian coordinate system at the next moment. y b' represents the component of the absolute bounding box b of the pedestrian at the current moment projected onto the y-axis of the Cartesian coordinate system. y b represents the component of the absolute bounding box b of the pedestrian projected onto the y-axis of the Cartesian coordinate system at the next moment. z b' represents the component of the absolute bounding box b of the pedestrian at the current moment projected onto the z-axis of the Cartesian coordinate system. z This represents the component of the absolute bounding box b of the pedestrian projected onto the z-axis of the Cartesian coordinate system at the next moment.

[0117] The crowd simulation model based on fluid dynamics includes:

[0118]

[0119] Among them, v 预 v represents the predicted velocity of the pedestrian in the next moment. 0x v represents the component of the pedestrian's translational velocity along the x-axis at the current moment. 0y v represents the component of the pedestrian's translational velocity along the y-axis at the current moment. 0z ω represents the component of the pedestrian's translational velocity along the z-axis at the current moment. x ω represents the component of the pedestrian's angular velocity at the current moment projected onto the x-axis of the Cartesian coordinate system. y ω represents the component of the pedestrian's angular velocity projected onto the y-axis of the Cartesian coordinate system at the current moment. z This represents the component of the pedestrian's angular velocity at the current moment projected onto the z-axis of the Cartesian coordinate system.

[0120] Through the above technical solutions, a rule-based approach is used to design a crowd simulation model based on fluid dynamics, predict the speed of pedestrians, analogize each fluid to a fluid particle, and use the physical equations of fluid particle motion to guide pedestrian movement, thereby improving the accuracy of crowd movement simulation.

[0121] like Figure 2 As shown, in some possible embodiments, the pedestrian path planning method based on reinforcement learning further includes S16 to S18.

[0122] S16. Based on Gauss's law of velocity field of the crowd to which the pedestrian is located, determine the variance of the crowd velocity.

[0123] Let Ω' be the macro-infinite and micro-finite closed region composed of piecewise smooth surfaces Σ'.

[0124]

[0125] Where σ represents the variance of the velocity of the crowd to which the pedestrian is located, and v represents the velocity field of the crowd to which the pedestrian is located.

[0126] When the flow of the pedestrian in the environment tends to be stable, the velocity flux through the cross-section decreases, and the value of σ decreases.

[0127] S17, determine the area covered by pedestrians in the crowd of the pedestrian to be tested walking at the same preset speed within a preset time period.

[0128] The area covered by pedestrians in the same preset speed within a preset time period can be represented by S.

[0129] S18. Determine the model quality of the crowd simulation model based on fluid dynamics by considering the variance of the crowd speed of the pedestrian being tested and the area covered by the pedestrians in the crowd being tested walking at the same preset speed within a preset time.

[0130] In this disclosure, the variance σ of the crowd speed of the pedestrian being tested and the area S covered by the pedestrians in the crowd walking at the same preset speed within a preset time period can be used as macroscopic evaluation indicators for the pedestrian simulation to determine the stability and mobility of the crowd, and thus determine the model quality of the crowd simulation model based on fluid dynamics.

[0131] Specifically, the smaller the variance σ of the crowd speed of the pedestrian being tested, the better the simulation performance of the crowd simulation model based on fluid dynamics. The larger the area S covered by pedestrians in the crowd being tested walking at the same preset speed within a preset time period, the better the simulation performance of the crowd simulation model based on fluid dynamics.

[0132] Figure 4 This is a verification diagram of four population simulation models illustrated according to an exemplary embodiment.

[0133] In one possible embodiment, such as Figure 4 As shown, the performance of the models was validated based on four crowd simulation models: (a) baseline method, (b) Boids method, (c) social force model, and (d) OurAIR (crowd simulation model based on fluid dynamics).

[0134] like Figure 4 As shown in Figures (a) and (b), the variance σ of the pedestrian speed in the crowd is relatively small. In Figure (d), the area S covered by the pedestrians in the crowd based on fluid dynamics of this disclosure is relatively large when they walk at the same preset speed within a preset time period. Therefore, the crowd simulation model based on fluid dynamics of this disclosure can cover more areas in the same time period, indicating that the pedestrian speed distribution simulated by the model of this disclosure is more stable and the crowd movement simulation is more efficient.

[0135] In some possible embodiments, the reinforcement learning-based pedestrian path planning method of this disclosure is adapted to dynamic path planning tasks from a first-person perspective by using reinforcement learning tasks.

[0136] The publicly available hydrodynamic-based crowd simulation model was trained on the Town10 map provided by the CARLA platform, and the human-environment interaction was customized.

[0137] The interaction between pedestrians and the environment is realized, and the termination condition of the interaction between pedestrians and the environment is based on at least one of the following:

[0138] (1) Pedestrians arrive at random destinations;

[0139] (2) A pedestrian collides with another pedestrian or an obstacle;

[0140] (3) In the environment, the duration of interaction between pedestrians exceeds the preset duration.

[0141] In one possible embodiment, this disclosure may set up a semantic segmentation camera for pedestrians, which is used to simulate the process by which pedestrians acquire images of their current surrounding environment through vision and perform image processing.

[0142] It is also possible to set up a bird's-eye view for pedestrians to simulate their perception of the surrounding environment. The bird's-eye view is used to provide information about 30 meters in front of the pedestrian and 12 meters behind them, including real-time information about people, lanes and obstacles in front of and behind the pedestrian.

[0143] The semantic segmentation camera and bird's-eye view are input into a pre-trained frozen-weight VAE encoder, which outputs a one-dimensional feature vector.

[0144] The one-dimensional feature vector is concatenated with the pedestrian's current position to obtain the observation value at the current time.

[0145] In its public version, OpenAI Gym's PPO reinforcement learning framework uses a dynamic PPO algorithm, where a = 1000, b = 500, and c = 0.1 in a preset reward function.

[0146] In one possible embodiment, as shown in Table 1, the dynamic PPO algorithm of this disclosure is compared with the DQN algorithm, DiscretePPO algorithm, and Continuous PPO algorithm in terms of success rate and collision rate. The experimental results in Table 1 are all averages of multiple experiments.

[0147]

[0148] Table 1

[0149] In this context, Our represents the dynamic PPO algorithm used in the PPO reinforcement learning framework of OpenAI Gym disclosed herein.

[0150] As shown in Table 1, the dynamic PPO algorithm used in the reinforcement learning framework of this disclosure converges faster than other existing DQN algorithms, Discrete PPO algorithms, and Continuous PPO algorithms. It also exhibits superior performance in terms of success rate and bit error rate. Therefore, the pedestrian path planning method based on reinforcement learning disclosed in this disclosure has excellent performance.

[0151] Through the above technical solutions, the pedestrian path planning method based on reinforcement learning disclosed herein achieves good results in crowd simulation tasks, with its evaluation metrics being the variance σ of the crowd speed of the pedestrian group and the area S covered by pedestrians walking at the same preset speed within a preset time. In path planning tasks, its dynamic PPO algorithm also produces good results, thus proving the effectiveness of the pedestrian path planning method based on reinforcement learning that combines rule-based and agent-based methods disclosed herein.

[0152] Based on the same concept, this disclosure also provides a pedestrian path planning device based on reinforcement learning. Figure 5 This is a block diagram illustrating a pedestrian path planning device based on reinforcement learning, according to an exemplary embodiment. Figure 5 As shown, the pedestrian path planning device 100 based on reinforcement learning includes: an acquisition module 110, a model training module 120, a model prediction module 130, and a crowd movement simulation module 140.

[0153] The acquisition module 110 is used to acquire the pedestrian path dataset and the current position coordinates, movement speed and angular velocity of the pedestrians in the test group. The pedestrian path dataset includes the position coordinates, translation speed and angular velocity of the pedestrians at each moment.

[0154] The model training module 120 is used to train the fluid dynamics model based on the pedestrian path dataset to determine a crowd simulation model based on fluid dynamics.

[0155] The model prediction module 130 is used to input the current position coordinates, movement speed and angular velocity of the pedestrian in the crowd to be tested into the crowd simulation model based on fluid dynamics, and predict the movement speed of the pedestrian in the next moment.

[0156] The crowd motion simulation module 140 is used to feed back the predicted motion speed of the pedestrian to be tested at the next moment to the simulation environment according to the dynamic loss function and the reward function, so as to simulate the movement of pedestrians in the crowd to be tested.

[0157] Through the above technical solution, this disclosure combines rule-based and agent-based methods to simulate pedestrian movement. The rule-based method develops a crowd simulation model based on fluid dynamics, treating each pedestrian in the crowd as a fluid particle. The physical equations of fluid particle motion, i.e., a fluid dynamics model, guides pedestrian movement and predicts the pedestrian's speed at the next moment, improving the accuracy of the simulated pedestrian movement. The agent-based method uses a dynamic loss function and a preset reward function to feed back the predicted speed of the pedestrian at the next moment to the simulation environment. Pedestrians interact with the environment in real time to realistically and effectively simulate pedestrian movement in the crowd, improving the realism of the simulated pedestrian movement. This disclosure improves pedestrian movement planning by determining the agent's speed to achieve its desired destination and preventing collisions, thus realizing pedestrian movement simulation and improving the realism and effectiveness of crowd movement simulation.

[0158] Optionally, the reinforcement learning-based pedestrian path planning device 100 further includes:

[0159] The loss function determination module is used to determine the dynamic loss function based on the predicted movement speed of the pedestrian under test at the next moment, the actual movement speed at the next moment, and the preset loss function term of the crowd simulation model based on fluid dynamics.

[0160] Optionally, the loss function determination module includes:

[0161] DL = MSE(v,p)

[0162]

[0163] Wherein, DL represents the preset loss function term, i.e., dynamic loss, v represents the actual movement speed at the next moment, p represents the movement speed of the pedestrian predicted by the crowd simulation model based on fluid dynamics at the next moment, Loss represents the dynamic loss function, PL represents the policy gradient loss, n represents the number of pedestrians in the preset radius area, EL represents the entropy loss, α represents the rotation angle between the vector connecting the pedestrians in the preset angle area directly in front of the pedestrian and the pedestrian, and VL represents the value loss.

[0164] Optionally, the preset reward function includes:

[0165]

[0166] R = a × DR - b × CR + FR - SR - c (c > 0)

[0167]

[0168]

[0169] Here, FR represents follow-up reward. Indicates the direction of movement of the pedestrian being measured. d represents the direction of movement of the other pedestrian, γ represents the distance between the pedestrian being tested and the other pedestrian, γ represents the coefficient of the weight of the follow reward value, a represents the weight of the pedestrian being tested reaching the destination, b represents the weight of the collision between the pedestrian being tested and the other pedestrian, DR represents the sign that the pedestrian being tested has reached the destination, CR represents the sign that the pedestrian being tested has collided with the other pedestrian, SR represents the square of the difference between the actual speed and the preset speed of the pedestrian being tested, and c represents the overspeed penalty of the pedestrian being tested at each time step.

[0170] Optionally, the model training module 120 includes:

[0171] The first determining submodule is used to determine the deformation rate tensor of the pedestrian based on the pedestrian's position coordinates, translational velocity, and angular velocity at each moment in the pedestrian path dataset.

[0172] The second determining submodule is used to input the deformation rate tensor, the position coordinates, translational velocity, and angular velocity of the pedestrians in the pedestrian path dataset at each moment into the fluid dynamics model for model training, and to determine the crowd simulation model based on fluid dynamics.

[0173] Optionally, the model training module 120 further includes:

[0174] The fluid dynamics model includes:

[0175] V=V0+E·δr+ω×δr

[0176] Where V represents the pedestrian's movement speed, V0 represents the pedestrian's translational speed, E represents the deformation rate tensor, δr represents the length infinitesimal vector, and ω represents the pedestrian's angular velocity;

[0177] The deformation rate tensor E includes:

[0178]

[0179] in, Let represent the angular velocity of one side of the absolute bounding box b of the pedestrian. Let b represent the angular velocity of the other adjacent side of the absolute bounding box b of the pedestrian. x b' represents the component of the absolute bounding box b of the pedestrian at the current moment projected onto the x-axis of the Cartesian coordinate system. x b represents the component of the absolute bounding box b of the pedestrian projected onto the x-axis of the Cartesian coordinate system at the next moment.y b' represents the component of the absolute bounding box b of the pedestrian at the current moment projected onto the y-axis of the Cartesian coordinate system. y b represents the component of the absolute bounding box b of the pedestrian projected onto the y-axis of the Cartesian coordinate system at the next moment. z b' represents the component of the absolute bounding box b of the pedestrian at the current moment projected onto the z-axis of the Cartesian coordinate system. z This represents the component of the absolute bounding box b of the pedestrian projected onto the z-axis of the Cartesian coordinate system at the next moment.

[0180] The crowd simulation model based on fluid dynamics includes:

[0181]

[0182] Among them, v 预 v represents the predicted velocity of the pedestrian in the next moment. 0x v represents the component of the pedestrian's translational velocity along the x-axis at the current moment. 0y v represents the component of the pedestrian's translational velocity along the y-axis at the current moment. 0z ω represents the component of the pedestrian's translational velocity along the z-axis at the current moment. x ω represents the component of the pedestrian's angular velocity at the current moment projected onto the x-axis of the Cartesian coordinate system. y ω represents the component of the pedestrian's angular velocity at the current moment projected onto the y-axis of the Cartesian coordinate system. z This represents the component of the pedestrian's angular velocity at the current moment projected onto the z-axis of the Cartesian coordinate system.

[0183] Optionally, the reinforcement learning-based pedestrian path planning device 100 further includes:

[0184] The first determining module is used to determine the variance of the velocity of the crowd to which the pedestrian to be tested is located, based on Gauss's law of velocity field of the crowd to which the pedestrian to be tested is located;

[0185] The second determining module is used to determine the area covered by pedestrians in the crowd of the pedestrian to be tested walking at the same preset speed within a preset time period;

[0186] The third determining module is used to determine the model quality of the crowd simulation model based on fluid dynamics based on the variance of the crowd speed of the crowd to which the pedestrian to be tested belongs and the area covered by the pedestrians in the crowd to which the pedestrian to be tested belongs walking at the same preset speed within a preset time.

[0187] Optionally, the first determining module includes:

[0188] Let Ω' be a closed region consisting of macro-infinitesimals and micro-finitesimals formed by piecewise smooth surfaces Σ':

[0189]

[0190] Where σ represents the variance of the velocity of the crowd to which the pedestrian to be tested belongs, and v represents the velocity field of the crowd to which the pedestrian to be tested belongs.

[0191] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0192] In another exemplary embodiment, a non-transitory computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the reinforcement learning-based pedestrian path planning method of the first aspect described above. For example, the computer-readable storage medium may be the aforementioned memory including program instructions, which may be executed by a processor of an electronic device to complete the aforementioned reinforcement learning-based pedestrian path planning method.

[0193] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described reinforcement learning-based pedestrian path planning method when executed by the programmable device.

[0194] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A pedestrian path planning method based on reinforcement learning, characterized in that, include: Obtain the pedestrian path dataset and the current position coordinates, translational velocity, and angular velocity of pedestrians in the test group. The pedestrian path dataset includes the position coordinates, translational velocity, and angular velocity of pedestrians at each moment. The fluid dynamics model is trained based on the pedestrian path dataset to determine a crowd simulation model based on fluid dynamics. The current position coordinates, translational velocity, and angular velocity of the pedestrians in the crowd to be tested are input into the crowd simulation model based on fluid dynamics to predict the movement velocity of the pedestrians in the next moment. The predicted speed of the pedestrian in the next moment is fed back to the simulation environment based on the dynamic loss function and the preset reward function to simulate the movement of pedestrians in the crowd. The step of training the fluid dynamics model based on the pedestrian path dataset to determine the crowd simulation model based on fluid dynamics includes: The deformation rate tensor of the pedestrian is determined based on the position coordinates, translational velocity, and angular velocity of the pedestrian at each moment in the pedestrian path dataset. The deformation rate tensor, the position coordinates, translational velocity, and angular velocity of the pedestrians in the pedestrian path dataset at each moment are input into the fluid dynamics model for model training to determine the crowd simulation model based on fluid dynamics. The step of training the fluid dynamics model based on the pedestrian path dataset to determine the crowd simulation model based on fluid dynamics further includes: The fluid dynamics model includes: ； in, Indicates the speed of pedestrian movement. This indicates the translational speed of the pedestrian. This represents the deformation rate tensor. Represents a vector of infinitesimal length. This represents the angular velocity of the pedestrian; The deformation rate tensor E includes: E = ； in, Absolute bounding box representing a pedestrian The angular velocity of one of its edges. Absolute bounding box representing a pedestrian The angular velocity of the other adjacent side, Represents the absolute bounding box of the pedestrian at the current moment. The component projected onto the Cartesian coordinate system along the x-axis. Represents the absolute bounding box of the pedestrian at the next moment. The component projected onto the Cartesian coordinate system along the x-axis. Represents the absolute bounding box of the pedestrian at the current moment. The component projected onto the y-axis of the Cartesian coordinate system. Represents the absolute bounding box of the pedestrian at the next moment. The component projected onto the y-axis of the Cartesian coordinate system. Represents the absolute bounding box of the pedestrian at the current moment. The component projected onto the Cartesian coordinate system along the z-axis. Represents the absolute bounding box of the pedestrian at the next moment. The component projected onto the z-axis of the Cartesian coordinate system; The crowd simulation model based on fluid dynamics includes: = + + ； in, This indicates the predicted speed of the pedestrian in the next moment. This represents the component of the pedestrian's translational velocity in the x-axis direction at the current moment. This represents the component of the pedestrian's translational velocity in the y-axis direction at the current moment. This represents the component of the pedestrian's translational velocity in the z-axis direction at the current moment. This represents the component of the pedestrian's angular velocity at the current moment projected onto the x-axis of the Cartesian coordinate system. This represents the component of the pedestrian's angular velocity at the current moment projected onto the y-axis of the Cartesian coordinate system. This represents the component of the pedestrian's angular velocity at the current moment projected onto the z-axis of the Cartesian coordinate system.

2. The method according to claim 1, characterized in that, The method further includes: The dynamic loss function is determined based on the predicted next-move speed of the pedestrian under test by the crowd simulation model based on fluid dynamics, the actual next-move speed, and the preset loss function term.

3. The method according to claim 2, characterized in that, The dynamic loss function is determined based on the pedestrian's predicted speed at the next moment, the actual speed at the next moment, and a preset loss function term, according to the crowd simulation model based on fluid dynamics. include: ； ； in, This represents the preset loss function term, i.e., the dynamic loss. This indicates the actual velocity at the next moment. This represents the predicted velocity of a pedestrian at the next moment, based on a crowd simulation model using fluid dynamics. This represents the dynamic loss function. Represents the policy gradient loss. This indicates the number of pedestrians within a preset radius area. Represents entropy loss, This represents the rotation angle between the vector connecting the pedestrians within a preset angle area directly in front of the pedestrian and the pedestrian being measured. This indicates a loss of value.

4. The method according to claim 1, characterized in that, The preset reward function includes: ； （ >0）； ； ； in, Indicates follow-up rewards. Indicates the direction of movement of the pedestrian being measured. Indicates the direction of movement of another pedestrian. This indicates the distance between the pedestrian being measured and the other pedestrian. A coefficient representing the weight of the reward value. This represents the weight of the pedestrian's arrival at their destination. This represents the collision weight between the pedestrian being tested and the other pedestrian. This indicates that the pedestrian being tested has arrived at their destination. A sign indicating that the pedestrian being tested has collided with another pedestrian. This represents the square of the difference between the actual speed and the preset speed of the pedestrian being measured. This represents the overspeed penalty for the pedestrian being tested at each time step.

5. The method according to claim 1, characterized in that, The method further includes: Based on Gauss's law for the velocity field of the crowd to which the pedestrian is located, determine the variance of the crowd velocity. Determine the area covered by pedestrians within the crowd of the pedestrian being tested, who walk at the same preset speed over a preset time period; The model quality of the crowd simulation model based on fluid dynamics is determined by the variance of the crowd speed of the pedestrian being tested and the area covered by pedestrians in the crowd being tested walking at the same preset speed within a preset time period.

6. The method according to claim 5, characterized in that, The step of determining the variance of the velocity of the crowd to which the pedestrian belongs, based on Gauss's law for the velocity field of the crowd, includes: set up For piecewise smooth surfaces The macro-infinite and micro-finite closed regions formed by: ； in, This represents the variance of the crowd speed of the group containing the pedestrian being tested. This represents the velocity field of the crowd to which the pedestrian being measured is located.

7. A pedestrian path planning device based on reinforcement learning, characterized in that, include: The acquisition module is used to acquire the pedestrian path dataset and the current position coordinates, movement speed and angular velocity of pedestrians in the test group. The pedestrian path dataset includes the position coordinates, translation speed and angular velocity of pedestrians at each moment. The model training module is used to train the fluid dynamics model based on the pedestrian path dataset to determine a crowd simulation model based on fluid dynamics. The model prediction module is used to input the current position coordinates, movement speed and angular velocity of the pedestrians in the crowd to be tested into the crowd simulation model based on fluid dynamics, and predict the movement speed of the pedestrians in the next moment. The crowd motion simulation module is used to feed back the predicted motion speed of the pedestrian to be tested at the next moment to the simulation environment based on the dynamic loss function and reward function, so as to simulate the movement of pedestrians in the crowd to be tested. The model training module 120 includes: The first determining submodule is used to determine the deformation rate tensor of the pedestrian based on the pedestrian's position coordinates, translational velocity, and angular velocity at each moment in the pedestrian path dataset. The second determining submodule is used to input the deformation rate tensor, the position coordinates, translational velocity and angular velocity of the pedestrians in the pedestrian path dataset at each time moment into the fluid dynamics model for model training, and determine the crowd simulation model based on fluid dynamics. The model training module 120 further includes: The fluid dynamics model includes: ； in, Indicates the speed of pedestrian movement. This indicates the translational speed of the pedestrian. This represents the deformation rate tensor. Represents a vector of infinitesimal length. This represents the angular velocity of the pedestrian; The deformation rate tensor E includes: E = ； in, Absolute bounding box representing a pedestrian The angular velocity of one of its edges. Absolute bounding box representing a pedestrian The angular velocity of the other adjacent side, Represents the absolute bounding box of the pedestrian at the current moment. The component projected onto the Cartesian coordinate system along the x-axis. Represents the absolute bounding box of the pedestrian at the next moment. The component projected onto the Cartesian coordinate system along the x-axis. Represents the absolute bounding box of the pedestrian at the current moment. The component projected onto the y-axis of the Cartesian coordinate system. Represents the absolute bounding box of the pedestrian at the next moment. The component projected onto the y-axis of the Cartesian coordinate system. Represents the absolute bounding box of the pedestrian at the current moment. The component projected onto the Cartesian coordinate system along the z-axis. Represents the absolute bounding box of the pedestrian at the next moment. The component projected onto the z-axis of the Cartesian coordinate system; The crowd simulation model based on fluid dynamics includes: = + + ； in, This indicates the predicted speed of the pedestrian in the next moment. This represents the component of the pedestrian's translational velocity along the x-axis at the current moment. This represents the component of the pedestrian's translational velocity in the y-axis direction at the current moment. This represents the component of the pedestrian's translational velocity in the z-axis direction at the current moment. This represents the component of the pedestrian's angular velocity at the current moment projected onto the x-axis of the Cartesian coordinate system. This represents the component of the pedestrian's angular velocity at the current moment projected onto the y-axis of the Cartesian coordinate system. This represents the component of the pedestrian's angular velocity at the current moment projected onto the z-axis of the Cartesian coordinate system.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

Images