Autonomous navigation method based on spatiotemporal contrast reinforcement learning from bird's-eye view

Abstract

The present invention discloses an autonomous navigation method based on bird's-eye view spatiotemporal contrast reinforcement learning. The method first sets the state space, action space and reward function. Secondly, a bird's-eye view navigation network model based on bird's-eye view spatiotemporal contrast learning is constructed, each network parameter is initialized, and the three-dimensional point cloud of the initial observation is input into the bird's-eye view navigation network model. The intelligent agent randomly samples data and stores the obtained data pairs into the experience pool to initialize the experience pool. Finally, the navigation event is looped, the target point and the pedestrian position are set, the bird's-eye view navigation network model is updated according to the data stored in the experience pool, and the action is selected according to the updated bird's-eye view navigation network and the state of the intelligent agent is continuously updated until the mobile intelligent agent reaches the target position. The present invention significantly improves the success rate of autonomous exploration and navigation efficiency of the intelligent agent in complex and dynamic environments, and has important theoretical and practical application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of robots, and in particular relates to an autonomous navigation method based on spatiotemporal contrast reinforcement learning from a bird's-eye view. Background Art

[0002] In the real world, how to enable intelligent agents to achieve efficient and safe autonomous navigation in complex and changing environments is a prerequisite for the successful application of intelligent agents.

[0003] Traditional navigation algorithms usually involve global and local path planning, both of which have their own advantages and limitations when dealing with static and dynamic environments. Global path planning relies on a known map of the environment to navigate by selecting an overall path. Common methods include ant colony optimization and fast search random tree methods. However, these methods perform poorly when dealing with dynamic environments because they rely too much on static maps and cannot flexibly respond to changes in the environment. On the other hand, local path planning methods focus on replanning the path around the agent to adapt to dynamic changes in the environment. Artificial potential field method and dynamic window method are common local path planning strategies. Although these methods are able to handle dynamic environments, the computational overhead required to replan the navigation path is often limited by practical applications. Specifically, for applications where agents perform service tasks in dynamic environments, frequent path replanning may result in huge computational overhead, which may be impractical in actual operations. Therefore, traditional global and local path planning methods may appear inflexible in dynamic environments.

[0004] With the rapid development of deep learning and reinforcement learning, deep reinforcement learning has attracted widespread attention as an intelligent decision-making method. With the powerful data processing capabilities of deep neural networks and the excellent decision-making capabilities of reinforcement learning, a variety of deep reinforcement learning methods have emerged to handle the autonomous navigation tasks of intelligent agents. These methods model the navigation task as a Markov decision process and take sensor observations as states. Through interaction with the environment, deep reinforcement learning methods are committed to finding the optimal strategy to guide the agent to the target location by maximizing the cumulative reward of actions. Among these methods, some use a network based on a fully connected layer to process state information, but this type of method mainly considers the interactive relationship between the agent and the environment in the spatial dimension, and does not fully consider the state change characteristics of the agent and dynamic and static obstacles in the environment in the temporal dimension. Other methods use recurrent neural networks (RNNs) to process interactive information in the temporal dimension, which has been widely used due to their advantages in time series prediction. However, it is difficult for recurrent neural networks to extract global states at the same time, which imposes certain limitations on navigation performance. In complex dynamic environments, extracting comprehensive spatiotemporal state representations is still challenging and underdeveloped. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes an autonomous navigation method based on spatiotemporal contrast reinforcement learning from a bird's-eye view, using spatial point cloud features to characterize the environment and using bird's eye view (BEV) features to extract network encoding global spatial feature information. The network is trained through spatiotemporal self-supervision to learn the temporal correlation between consecutive frames and infer the dynamic relationship of spatial position transformation, thereby more accurately predicting the motion state of pedestrians. Based on the BEV feature representation, the SAC (Soft Actor-Critic) reinforcement learning method is used to obtain the optimal navigation strategy. The success rate and navigation efficiency of autonomous navigation of intelligent agents in complex and dynamic environments are greatly improved, which has important theoretical value and practical significance.

[0006] The autonomous navigation method based on spatiotemporal contrast reinforcement learning from a bird's-eye view specifically includes the following steps:

[0007] Step 1: Set the state space, action space, and reward function.

[0008] As a preference, the state space, action space and reward function are set as follows:

[0009] The agent observes the environment in front of it, avoids collisions with pedestrians and static obstacles, and reaches the target point smoothly; in this navigation process, p t Represents the 3D radar point cloud observation of the agent at time t, using Represents the BEV semantic features of the agent at time t, using represents the position of the agent at time t, a t =(v t ,ω t ) represents the action of the agent at time t, where v t represents the forward speed of the agent at time t, ω t represents the turning speed of the agent at time t, θ represents the heading angle of the agent, and g = [g x ,g y ] represents the location of the target point, Represents the state of the agent at time t.

[0010] At time t, the state space is s t , including the status information of static obstacles, pedestrians and agents in the space.

[0011] The action space consists of a continuous space of possible actions.

[0012] When designing the reward function, the following principles are followed: a positive reward is given when the agent reaches the target, and a negative reward is given when the agent collides. In addition, if the agent is too close to pedestrians during navigation, a penalty should also be imposed to guide the agent to maintain an appropriate comfortable distance from pedestrians during navigation. At the same time, a positive reward is given when the agent is close to the target, and a negative reward is given when it is far away from the target, so as to encourage the agent to plan the optimal path to the target while avoiding pedestrians. The reward function is defined as follows:

[0013]

[0014] In the above formula, R(s t ,a t )Depend on and Composition; is the reward for the agent to move toward the goal, It is to punish the agent for colliding with dynamic pedestrians or static obstacles.

[0015]

[0016] where r dist is the reward coefficient for each agent action, is the distance from the agent to the target point at time t.

[0017]

[0018] in is the shortest distance between the agent and the human within the decision time Δt, d c Represents the minimum comfortable distance that humans can tolerate, r min Represents the penalty coefficient for the agent's impact on pedestrians.

[0019] Step 2: Construct a bird's-eye view navigation network model based on bird's-eye view spatiotemporal contrast learning, wherein the model includes a BEV feature extraction network, a spatiotemporal self-supervised training network, and a reinforcement learning network.

[0020] (a) BEV feature extraction network

[0021] The current 3D radar point cloud p t As input, a sparse-dense BEV convolutional network is designed. The 3D point cloud is projected from top to bottom onto a 2D plane, and a 2D sparse convolutional network is used to extract the main spatial features from the 2D plane. Then, a series of dense convolutional networks are used to extract deeper spatial features to obtain the BEV spatial features.

[0022] (b) Spatiotemporal self-supervised training network

[0023] In order to enhance the agent's ability to represent and reason about the spatiotemporal characteristics of the observed state. t ,[a t ,a t+1 ,…,a t+k ],p t+k} as input, an asymmetric network structure is designed to train the BEV feature extraction network in a self-supervised manner.

[0024] In terms of spatial representation, the three-dimensional point cloud p t Perform random data augmentation to obtain and And use ψ to extract its features respectively and in Pass through the fully connected network M g and the fully connected network M h Get prediction features and Only through the fully connected network M g Forming target features Finally, the similarity between the predicted features and the target features is calculated. This can be used to train the agent to represent the observation space well.

[0025] In terms of temporal reasoning, the current observation is combined with a multi-frame continuous action sequence to predict the features of future frames. Specifically, the 3D point cloud at time t and t+k is input into the ψ network to obtain and At the same time, the continuous action sequence [a t ,a t+1 ,…,a t+k ] Get the embedded features s of multi-frame actions through the fully connected network a , here we use a fully connected network for the input and a Output predicted features for future frames At the same time Get the target features through the fully connected network Finally, the similarity between the predicted results and the target features is calculated to train the intelligent agent to make good inference predictions about the future.

[0026] Reinforcement Learning Network

[0027] Based on the status t 、Action a t =(v t ,ω t ) and the target position g = [g x ,g y], designed a reinforcement learning SAC network, in which the Actor network consists of a BEV feature extraction network ψ, a target fully connected network and a policy network. The Critic network and the Actor network share the BEV feature extraction network and the target fully connected network. The difference between the Critic network and the Actor network is that the Critic network adds an action fully connected network, and the state Q value is estimated by two value networks; the Actor network input is the observed state s t and the target position g = [g x ,g y ], output the optimal action a under the current state t In the Actor network, the BEV feature extraction network encodes the three-dimensional point cloud into BEV features, and the fully connected network maps the target position to a high-dimensional space to generate a target embedding vector. Then, the policy network composed of fully connected layers uses the BEV features and the target embedding vector as input to obtain the optimal action in the current state; the Critic network input is the observed state s t 、Action a t =(v t ,ω t ) and the target position g = [g x ,g y ], output the Q value of the current state, and in the Critic network, obtain the same BEV features and target embedding vector as the Actor in the same way, and use the fully connected network to embed the action a t Map to high-dimensional space to generate action embedding vector, and then use the value network to get the Q value of the current state with BEV features, target embedding vector and action embedding vector as input; the specific formula is as follows:

[0028] a t =ζ([s t ,g];W ζ )

[0029]

[0030] where ζ represents the Actor network, Represents the Critic network, parameter W ζ and They are ζ and The embedding weight matrix of .

[0031] Step 3: Initialize the network parameters, input the initial observed 3D point cloud into the bird’s-eye view navigation network model, and the agent randomly samples the data and obtains the obtained <s t ,a t ,s t+1 ,r t,done>The data pair is stored in the experience pool D for initialization of the experience pool D, where r t is the reward obtained at time t, and done indicates whether the navigation task is completed at time t.

[0032] Step 4: Loop navigation events, set the target point and pedestrian position, update the bird's-eye view navigation network model according to the data stored in the experience pool D, and select action a according to the updated bird's-eye view navigation network t And constantly update the state s of the agent t , until the mobile agent reaches the target point.

[0033] The specific process of loop navigation event is as follows:

[0034] Randomly initialize the state s t , and use the updated policy network to select the optimal action a t , constantly updating the current state and reward value, as well as updating the BEV feature extraction network ψ, the Actor network ζ and the Critic network Until the agent reaches the target point or exceeds the navigation time; calculate the gradient to update the network parameters. When the number of training times is reached, the training of the network model is completed. In each step of the network update, based on the updated policy network π(·|s t ), the optimal value network Q(s) at time t t ,a t ) is expressed as:

[0035]

[0036]

[0037] Where T is the number of decisions from the state at time t to the final state, r(s t ,a t ) represents the reward obtained by executing action at at time t, γ∈[0,1) represents the discount factor, E represents the expectation, D represents the experience pool, π specifically refers to the strategy function, V(s t ) represents the estimation of the current state, H(π(·|s t )) represents the action entropy value. The updated policy network is expressed as:

[0038]

[0039] H(π(·|s t ))=E[-log(·|s t )]

[0040] Where r(s t ,a t ) is the reward obtained at time t, and α is the entropy regularization coefficient.

[0041] The present invention has the following beneficial effects:

[0042] A novel autonomous navigation method based on bird's-eye view spatiotemporal contrast learning is proposed to solve the challenging task of autonomous navigation in complex environments. The BEV feature representation is effectively extracted from the observation point cloud of the intelligent agent, and rich spatial location information is obtained, while further improving the navigation efficiency. Based on this, the reinforcement learning network can accurately generate the optimal action according to the state, thereby providing support for efficient navigation strategies. Experimental results under various evaluation settings show that this method significantly improves the success rate of autonomous exploration and navigation efficiency of the intelligent agent in complex and dynamic environments, and has important theoretical and practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the network structure for BEV feature extraction network and spatiotemporal self-supervised training;

[0044] Figure 2 This is a schematic diagram of the autonomous navigation network model structure based on bird's-eye view spatiotemporal contrast reinforcement learning;

[0045] Figure 3 This is a flow chart of the autonomous navigation algorithm based on spatiotemporal contrast reinforcement learning from a bird's eye view;

[0046] Figure 4 Flowchart of training value network for reinforcement learning SAC algorithm. DETAILED DESCRIPTION

[0047] The present invention is further explained below in conjunction with the accompanying drawings.

[0048] like Figure 3 As shown, an autonomous navigation algorithm method based on bird's-eye view spatiotemporal contrast reinforcement learning includes the following steps:

[0049] Step 1: State space, action space, and reward function setting.

[0050] As a preference, the state space, action space and reward function are set as follows:

[0051] In a three-dimensional simulation environment, the agent can observe the environment in front of it. The agent needs to avoid collisions with pedestrians and reach the target point smoothly. In this navigation process, p t Represents the 3D radar point cloud observation of the agent at time t, using Represents the BEV semantic features of the agent at time t, using represents the position of the agent at time t, a t =(v t ,ω t) represents the action of the agent at time t, where v t represents the forward speed of the agent at time t, ω t represents the turning speed of the agent at time t, θ represents the heading angle of the agent, and g = [g x ,g y ] represents the location of the target point, Represents the state of the agent at time t.

[0052] At time t, the state space is s t ; Contains the status information of static obstacles, pedestrians and agents in the space.

[0053] The action space consists of a discrete space of possible actions.

[0054] When designing the reward function, the following principles are followed: a positive reward is given when the agent reaches the target, and a negative reward is given when the agent collides. In addition, if the agent is too close to pedestrians during navigation, a moderate penalty should be imposed to guide the agent to maintain an appropriate and comfortable distance from pedestrians during navigation. At the same time, a positive reward is given when the agent is close to the target, and a negative reward is given when it is far away from the target, so as to encourage the agent to plan the optimal path to the target while avoiding pedestrians. The reward function is defined as follows:

[0055]

[0056] In the above formula, R(s t ,a t )Depend on and Composition; is the reward for the agent to move toward the goal, It is to punish the agent for colliding with dynamic pedestrians or static obstacles.

[0057]

[0058] where r dist =5 is the reward coefficient for each agent action, is the distance from the agent to the target point at time t.

[0059]

[0060] in is the shortest distance between the agent and the human within the decision time Δt, d c Represents the minimum comfortable distance that humans can tolerate, r min =5 represents the penalty coefficient for the agent affecting pedestrians, d c The value range is (0.2,0.4).

[0061] Step 2: Reinforcement learning network model construction.

[0062] like Figure 1 , 2 As shown, a bird's-eye view navigation network model based on bird's-eye view spatiotemporal contrast learning is constructed, and the model includes a BEV feature extraction network, a spatiotemporal self-supervised training network, and a reinforcement learning network.

[0063] (a) BEV feature extraction network

[0064] The current three-dimensional point cloud p t As input, a sparse-dense BEV convolutional network is designed. The 3D point cloud is projected from top to bottom onto a 2D plane, and a 2D sparse convolutional network is used to extract the main spatial features from the 2D plane. Then, a series of dense convolutional networks are used to extract deeper spatial features to obtain the BEV spatial features. This spatial feature will be used for subsequent navigation strategy learning, namely:

[0065]

[0066] Where ψ is the BEV feature extraction network, parameter W ψ is the embedding weight matrix.

[0067] (b) Spatiotemporal self-supervised training network

[0068] In order to enhance the agent's ability to represent and reason about the spatiotemporal characteristics of the observed state. t ,[a t ,a t+1 ,…,a t+k ],p t+k} as input, an asymmetric network structure is designed to train the BEV feature extraction network in a self-supervised manner.

[0069] In terms of spatial representation, the three-dimensional point cloud p t Perform random data augmentation to obtain and And use ψ to extract its features respectively and in Pass through the fully connected network M g and the fully connected network M h Get prediction features and Only through the fully connected network M g Forming target features Finally, the similarity between the predicted features and the target features is calculated. This training agent can well represent the observation space. The specific formula is:

[0070]

[0071]

[0072]

[0073] Where N represents the feature vector dimension.

[0074] In terms of temporal reasoning, the current observation is combined with a multi-frame continuous action sequence to predict the features of future frames. Specifically, the 3D point cloud at time t and t+k is input into the ψ network to obtain and At the same time, the continuous action sequence [a t ,a t+1 ,…,a t+k ] Get the embedded features s of multi-frame actions through the fully connected network a , where the prediction network is used for the input and a Output predicted features for future frames At the same time Get the target features through the fully connected network Finally, the similarity between the predicted result and the target feature is calculated to train the agent to make good inference predictions about the future. The specific formula is:

[0075] s a =MLP([a t ,a t+1 ,…,a t+k ])

[0076]

[0077]

[0078]

[0079] (c) Reinforcement Learning Network

[0080] like Figure 4 As shown, based on the state s t 、Action a t =(v t ,ω t ) and the target position g = [g x ,g y], designed a reinforcement learning SAC network, in which the Actor network consists of a BEV feature extraction network, a target fully connected network and a policy network. The Critic network and the Actor network share the BEV feature extraction network and the target fully connected network. The difference between the Critic network and the Actor network is that the Critic network adds an action fully connected network, and the state Q value is estimated by two value networks to avoid the problem of overestimation of a single Q value; the Actor network input is the observation state s t and the target position g = [g x ,g y ], output the optimal action a under the current state t In the Actor network, the BEV feature extraction network encodes the three-dimensional point cloud into BEV features, and the fully connected network maps the target position to a high-dimensional space to generate a target embedding vector. Then, the policy network composed of fully connected layers uses the BEV features and the target embedding vector as input to obtain the optimal action in the current state; the Critic network input is the observed state s t 、Action a t =(v t ,ω t ) and the target position g = [g x ,g y ], output the Q value of the current state, and in the Critic network, obtain the same BEV features and target embedding vector as the Actor in the same way, and use the fully connected network to embed the action a t Map to high-dimensional space to generate action embedding vector, and then use the value network to get the Q value of the current state with BEV features, target embedding vector and action embedding vector as input; the specific formula is as follows:

[0081] a t =ζ([s t ,g];W ζ )

[0082]

[0083] where ζ represents the Actor network, Represents the Critic network, parameter W ζ and They are ζ and The embedding weight matrix of .

[0084] Step 3: Initialize the network parameters, input the initial observed 3D point cloud into the bird’s-eye view navigation network model, and the agent randomly samples the data and obtains the obtained <s t ,a t ,s t+1 ,r t,done>The data pair is stored in the experience pool D for initialization of the experience pool D, where r t is the reward obtained at time t, and done indicates whether the navigation task is completed at time t, that is:

[0085]

[0086] Step 4: Loop through navigation events, set the target point and pedestrian position, and pass the policy network π(·|s t ) Select the best action a t And constantly update the state s of the agent t , until the mobile agent reaches the target position, where the policy network is expressed as:

[0087]

[0088] H(π(·|s t ))=E[-log(·|s t )]

[0089] Where r(s t ,a t ) is the reward obtained at time t, and α is the entropy regularization coefficient.

[0090] In each step of network update, based on the updated policy network π(·|s t ), the optimal value network Q(s) at time t t ,a t ) is expressed as:

[0091]

[0092]

[0093] Where T is the number of decisions from the state at time t to the final state, r(s t ,a t ) means executing action a at time t t The reward obtained, γ∈[0,1) represents the discount factor, E represents the expectation, D represents the experience pool, π specifically refers to the strategy function, V(s t ) represents the estimation of the current state, H(π(·|s t )) represents the action entropy value.

[0094] The specific process of the loop navigation event in step 4 is as follows:

[0095] Randomly initialize the state s t , and use the updated policy network to select the optimal action a t, constantly updating the current state and reward value, as well as updating the BEV feature extraction network ψ, the Actor network ζ and the Critic network Until the agent reaches the target point or exceeds the navigation time; calculate the gradient and update the network parameters. When the number of training times is reached, the training of the network model is completed. Preferably, the decision time Δt=0.1s.

[0096] Experimental training details: During the initial network imitation learning process, the learning rate was 0.001 and the training was 1 million steps. During the reinforcement learning training process, the initial learning rate was 0.005 and the batch size was 64. As the training continued, the learning rate was updated to 0.9 times the original every 1000 rounds, and the AdamW optimizer was used to optimize the data with a batch size of 100. At the same time, a data enhancement technique was used, that is, random translation was introduced into the data with a translation amplitude of 0.01. The trained network model was tested for 500 navigations. Compared with other network models, the test results are shown in Table 1 below:

[0097] Table 1

[0098] method Success rate Average speed Success Path Length DMCL 0.88 0.75 0.83 SAC-P 0.84 0.73 0.80 The present invention 0.94 0.76 0.85

[0099] It can be seen from Table 1 above that the success rate of the present invention is significantly higher than that of other methods, and efficient autonomous navigation can be achieved to reach the target location.

Claims

1. An autonomous navigation method based on bird's-eye view spatiotemporal contrast reinforcement learning, characterized in that: The following steps are involved: Step 1: Set the state space, action space and reward function; The settings of the state space, action space and reward function are as follows: The agent observes the environment in front of it, avoids collisions with pedestrians and static obstacles, and reaches the target point smoothly. During the navigation process, it uses p t Represents the 3D radar point cloud observation of the agent at time t, using Represents the BEV semantic features of the agent at time t, using represents the position of the agent at time t, a t =(v t ,ω t ) represents the action of the agent at time t, where v t represents the forward speed of the agent at time t, ω t represents the turning speed of the agent at time t, θ represents the heading angle of the agent, and g = [g x ,g y ] represents the location of the target point, Represents the state of the agent at time t; At time t, the state space is s t , including the status information of static obstacles, pedestrians and agents in the space; The action space consists of a continuous space of possible actions; When the agent reaches the target, it is given a positive reward, and when the agent collides, it is given a negative reward; if the agent is too close to the pedestrian during navigation, a penalty is imposed; the agent is given a positive reward when it is close to the target, and a negative reward when it is far away from the target; the reward function is defined as follows: In the above formula, R(s t ,a t )Depend on and Composition; is the reward for the agent to move toward the goal, It is to punish the agent for colliding with dynamic pedestrians or static obstacles; where r dist is the reward coefficient for each agent action, is the distance from the agent to the target point at time t; in is the shortest distance between the agent and the human within the decision time Δt, d c Represents the minimum comfortable distance that humans can tolerate, r min represents the penalty coefficient for the agent affecting pedestrians; Step 2: Construct a bird's-eye view navigation network model based on bird's-eye view spatiotemporal contrast learning, including a BEV feature extraction network, a spatiotemporal self-supervised training network, and a reinforcement learning network; The BEV feature extraction network is as follows: The current 3D radar point cloud p t As input, a sparse-dense BEV convolutional network is designed. The 3D point cloud is projected from top to bottom onto a 2D plane, and a 2D sparse convolutional network is used to extract spatial features from the 2D plane. Then, a column-dense convolutional network is used to extract spatial features to obtain BEV spatial features. Where ψ is the BEV feature extraction network, parameter W ψ is the embedding weight matrix; The spatiotemporal self-supervised training network is specifically as follows: {p t ,[a t ,a t+1 ,…,a t+k ],p t+k } as input, design an asymmetric network structure to train the BEV feature extraction network in a self-supervised manner; In terms of spatial representation, the three-dimensional point cloud p t Perform random data augmentation to obtain and And use ψ to extract its features respectively and in Pass through the fully connected network M g and the fully connected network M h Get prediction features and Only through the fully connected network M g Forming target features Finally, calculate the similarity between the predicted features and the target features: Where N represents the feature vector dimension; In terms of temporal reasoning, the 3D point cloud at time t and t+k is input into the ψ network to obtain and At the same time, the continuous action sequence [a t ,a t+1 ,…,a t+k ] Get the embedded features s of multi-frame actions through the fully connected network a , here we use a fully connected network for the input and a Output predicted features for future frames At the same time Get the target features through the fully connected network Finally, calculate the similarity between the predicted results and the target features: s a =MLP([a t ,a t+1 ,…,a t+k ]) The reinforcement learning network is as follows: Based on the status t 、Action a t =(v t ,ω t ) and the target position g = [g x ,g y ], designed a reinforcement learning SAC network, in which the Actor network consists of a BEV feature extraction network ψ, a target fully connected network and a policy network. The Critic network and the Actor network share the BEV feature extraction network ψ and the target fully connected network. The difference between the Critic network and the Actor network is that the Critic network adds an action fully connected network, and the state Q value is estimated by two value networks; the Actor network input is the observation state s t and the target position g = [g x ,g y ], output the optimal action a under the current state t In the Actor network, the BEV feature extraction network encodes the three-dimensional point cloud into BEV features, and the fully connected network maps the target position to a high-dimensional space to generate a target embedding vector. Then, the policy network composed of fully connected layers uses the BEV features and the target embedding vector as input to obtain the optimal action in the current state; the Critic network input is the observed state s t 、Action a t =(v t ,ω t ) and the target position g = [g x ,g y ], output the Q value of the current state, and in the Critic network, obtain the same BEV features and target embedding vector as the Actor in the same way, and then use the fully connected network to embed the action a t Map to high-dimensional space to generate action embedding vector, and then use the value network to get the Q value of the current state with BEV features, target embedding vector and action embedding vector as input; the specific formula is as follows: a t =ζ([s t ,g];W ζ ) Where ζ represents the Actor network, Represents the Critic network, parameter W ζ and They are ζ and The embedding weight matrix of Step 3: Initialize network parameters, input the initial observed 3D point cloud into the bird's-eye view navigation network model, the agent randomly samples data and stores the obtained data pairs into the experience pool D, and initializes the experience pool D; The agent randomly samples data as <s t ,a t ,s t+1 ,r t ,done>, where r t is the reward obtained at time t, and done indicates whether the navigation task is completed at time t, that is: Step 4: Loop navigation events, set the target point and pedestrian position, update the bird's-eye view navigation network model according to the data stored in the experience pool D, and select action a according to the updated bird's-eye view navigation network model t And constantly update the state s of the agent t , until the mobile agent reaches the target point; The specific process of the cyclic navigation event is as follows: Randomly initialize the state s t , and use the current policy network to select the optimal action a t , constantly updating the current state and reward value, as well as updating the BEV feature extraction network ψ, the Actor network ζ and the Critic network Until the agent reaches the target point or exceeds the navigation time; Calculate the gradient to update the network parameters. In each step of the network update, based on the updated policy network π(·|s t ), t The optimal value network Q(st,at) at the moment is expressed as: Where T is from t The number of decisions from the state at the moment to the final state, r(st,at) represents t Execute action a at any time t The reward obtained, γ∈[0,1) represents the discount factor, E represents the expectation, D represents the experience pool, π specifically refers to the strategy function, V(s t ) represents the estimation of the current state, H(π(·|s t )) represents the action entropy value; The updated policy network is expressed as: H(π(·|s t ))=E[-log(·|s t )] Where r(s t ,a t ) is in t The reward obtained at the moment, α is the entropy regularization coefficient.

Images