A hierarchical attention-based robot navigation method and system

Abstract

This invention relates to a robot navigation method and system based on hierarchical attention, belonging to the field of path planning technology. It includes: acquiring the robot's pose and the position information of the target point, and acquiring environmental images and pedestrian data ahead of the robot's current movement path; extracting spatial semantic feature vectors from the images using a hierarchical attention architecture; extracting dynamic pedestrian feature vectors using a multilayer perceptron; preprocessing the position information to obtain the target point feature vector, dynamically fusing the three feature vectors after concatenation, and inputting them into a policy network and a value network to obtain the robot's next action and state value; obtaining new pose, position information, and environmental state based on the new action, and calculating the reward value using a reward function; continuously updating the parameters of the two networks based on the reward value and the new state value, and obtaining the optimal reinforcement learning model after multiple iterations, using this reinforcement learning model to complete robot navigation. This invention improves the accuracy and stability of navigation.

Description

Technical Field

[0001] This invention relates to the field of path planning technology, and in particular to a robot navigation method and system based on hierarchical attention. Background Technology

[0002] With the continuous development of automation technology, mobile robots have been widely used in various industries such as hospitals, homes, hotels, exhibition halls, and restaurants, mainly undertaking tasks such as logistics transportation, goods handling, and order delivery. Among existing mobile robot navigation technologies, traditional navigation algorithms such as A*, RRT, and DWA have formed relatively mature application systems. However, these traditional navigation methods have significant drawbacks in practical applications. Due to the complexity and uncertainty of real-world application scenarios, traditional navigation methods require parameter tuning based on specific environmental parameters, which is not only cumbersome but also prone to poor navigation performance. For example, they cannot safely avoid obstacles and are difficult to meet navigation needs in complex environments.

[0003] Traditional navigation methods employ a rule-based design approach, which lacks flexibility. Typically, traditional navigation methods decouple the navigation process into two independent modules: global path planning and local path planning, implementing navigation functionality through modular processing. However, this decoupled design struggles to handle the dynamic and complex situations in real-world environments. To overcome the performance bottlenecks of traditional navigation strategies, researchers have introduced Deep Reinforcement Learning (DRL) technology into the field of mobile robot navigation, providing a new technological path to improve navigation performance. DRL models the mobile robot navigation problem as a Markov Decision Process (MDP). By enabling the agent (i.e., the mobile robot) to interact and experiment with its surroundings, deep neural networks learn an end-to-end mapping strategy from the initial perception state to the optimal control action. This technology effectively integrates perception, planning, and control functions, allowing the mobile robot to learn complex nonlinear decision-making strategies, thereby achieving a high degree of environmental adaptability and fundamentally eliminating the reliance on manual parameter tuning and precise environmental modeling inherent in traditional methods. At the algorithm research level, mainstream deep reinforcement learning algorithms such as DQN, PPO, and TD3 have been validated in robot navigation simulation environments, demonstrating their great potential in solving path planning and obstacle avoidance problems.

[0004] While deep reinforcement learning has brought breakthroughs to mobile robot navigation, it still faces many challenges in practical applications. Currently, the application of deep reinforcement learning in mobile robot navigation largely relies on single or homogeneous sensor information (such as RGB images). This raw, unprocessed sensor data cannot comprehensively and accurately represent all the features of the environment. This problem directly leads to poor obstacle avoidance performance in mobile robots using this technology. Furthermore, due to the simplistic state space design, it cannot completely and accurately describe the robot's real-time environment, thus limiting the effectiveness of deep reinforcement learning in real-world mobile robot navigation scenarios. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the defects of the prior art, which are incomplete expression of environmental features, insufficient obstacle avoidance performance, and simple state space design that cannot accurately describe the real-time environment due to reliance on single or homogeneous sensor information.

[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention provides a robot navigation method based on hierarchical attention, comprising:

[0007] S1. Obtain the robot's pose and the location information of the target point, and obtain the color image, depth image and pedestrian data in front of the robot's current movement path;

[0008] S2. Extract spatial semantic feature vectors of the color image and the depth image using a hierarchical attention architecture; extract dynamic pedestrian feature vectors of the pedestrian data using a multilayer perceptron.

[0009] S3. Preprocess the location information to obtain the target point feature vector; concatenate the target point feature vector, the spatial semantic feature vector, and the dynamic pedestrian feature vector to obtain the concatenated feature vector; dynamically fuse the concatenated feature vector to obtain the deep fusion feature vector.

[0010] S4. Input the deep fusion feature vector into the policy network to obtain the robot's next action; input the deep fusion feature vector into the value network to obtain the state value at the next moment;

[0011] S5. Based on the executed action, obtain the robot's next pose, the target point's next position information, and the next environmental state in front of the robot's movement path; calculate the reward value based on the next pose, the next position information, the next environmental state, and the reward function.

[0012] S6. Calculate the policy gradient and value loss based on the reward value and the state value; update the network parameters of the policy network using the policy gradient and update the network parameters of the value network using the value loss, and perform multiple iterations of training until the reward function converges, thereby obtaining the optimal policy network and the optimal value network respectively.

[0013] S7. Obtain the robot's control actions based on the optimal reinforcement learning model, and complete robot navigation based on the control actions; wherein, the optimal reinforcement learning model includes the optimal policy network and the optimal value network.

[0014] In one embodiment of the present invention, step S2, which involves extracting the spatial semantic feature vectors of the color image and the depth image using a hierarchical attention architecture, is as follows:

[0015] Convert the color image and the depth image to Channel feature map; where Represents positive integers;

[0016] The The channel feature map is divided into multiple local windows. Each local window undergoes convolution to enhance the positional information of the features within the window, and average pooling is used to generate... Each unique carrier token, among which Represents positive integers;

[0017] All the local windows corresponding to the Each dedicated carrier token is aggregated to form a global carrier token set; multi-head self-attention and multi-layer perceptron are applied to the global carrier token set to update the corresponding token for each local window. The characteristics of a unique carrier token;

[0018] The updated version Each dedicated carrier token is concatenated with the corresponding local window to obtain a first fusion feature for each local window; multi-head self-attention calculation is performed on each first fusion feature to obtain a second fusion feature for each local window;

[0019] The second fusion feature of each local window is spatially restored, and the corresponding feature of each local window is... Each dedicated carrier token is upsampled to obtain a sub-feature map; all the sub-feature maps are spliced ​​and merged in the spatial dimension to obtain a complete feature map; a spatial semantic feature vector is generated based on the complete feature map.

[0020] In one embodiment of the present invention, step S2, which involves extracting the dynamic pedestrian feature vector from the pedestrian data using a multilayer perceptron, comprises:

[0021] The pedestrian data is preprocessed to obtain dynamic pedestrian features; wherein the preprocessing includes sorting the pedestrian data according to the Euclidean distance between the pedestrian and the robot body;

[0022] The dynamic pedestrian features are input into a multilayer perceptron to extract dynamic pedestrian feature vectors; wherein the multilayer perceptron includes multiple fully connected layers, and each fully connected layer is connected to a ReLU activation function.

[0023] In one embodiment of the present invention, the reward function includes a target reward, a proximity reward, a collision reward, and a pedestrian reward.

[0024] In one embodiment of the present invention, the expression for the proximity reward is:

[0025] ;

[0026] in, Indicates a reward for getting closer to the target. Indicates the weighting coefficient. This represents the Euclidean distance between the robot and the target point at the previous moment. This represents the Euclidean distance between the robot and the target point at the current moment. This represents the threshold for determining whether the target point has been reached.

[0027] In one embodiment of the present invention, the expression for the collision reward is:

[0028] ;

[0029] in, Indicates collision reward, This represents the minimum distance between the robot and the obstacle. This indicates the distance judgment threshold.

[0030] In one embodiment of the present invention, the step of obtaining the pedestrian reward is as follows:

[0031] Obtain the pedestrian's current speed; dynamically generate a rectangular threat region ahead of the robot's movement path based on the current speed; wherein the length of the rectangular threat region and the current speed satisfy the following relationship:

[0032] ;

[0033] in, Indicates the first The length of the rectangular threat area, Indicates the constraint time. Indicates the pedestrian's current speed. Indicates the base distance;

[0034] Merge all dynamically generated rectangular threat regions to obtain a set of rectangular threat regions;

[0035] Obtain the robot's current position and determine whether the robot's current position is within the rectangular threat area set; if yes, the pedestrian reward is negative; otherwise, the pedestrian reward is not negative.

[0036] In one embodiment of the present invention, step S6, updating the network parameters of the policy network using the policy gradient, includes training the policy network according to a training objective function; wherein the expression of the training objective function is:

[0037] ;

[0038] ;

[0039] in, Represents the training objective function. This represents the estimation of the advantage function. Indicates hyperparameters; Indicates the current state Next, according to the strategy Select the action to perform. The probability of; Indicates the current state Next, according to the old strategy Select the action to perform. The probability, Represents the shearing function. This represents the expected function.

[0040] In one embodiment of the present invention, step S6, which involves updating the network parameters of the value network using the value loss, includes calculating a loss function and updating the network parameters using the loss function; wherein the expression of the loss function is:

[0041] ;

[0042] in, Indicates the loss value. Represents the value network in relation to the current state. The predictive value Indicates the target value.

[0043] In one embodiment of the present invention, both the policy network and the value network include One fully connected neural network layer; among which... Each of the fully connected neural network layers is connected to an activation function; wherein... It is a positive integer greater than 1.

[0044] Secondly, to solve the above-mentioned technical problems, the present invention provides a robot navigation system based on hierarchical attention, used to implement the above-mentioned robot navigation method based on hierarchical attention, comprising:

[0045] The acquisition module is used to acquire the robot's pose and the location information of the target point, and to acquire the color image, depth image and pedestrian data in front of the robot's current movement path;

[0046] The feature extraction module is used to extract spatial semantic feature vectors of the color image and the depth image using a hierarchical attention architecture; and to extract dynamic pedestrian feature vectors of the pedestrian data using a multilayer perceptron.

[0047] The feature fusion module is used to preprocess the location information to obtain a target point feature vector; concatenate the target point feature vector, the spatial semantic feature vector, and the dynamic pedestrian feature vector to obtain a concatenated feature vector; and dynamically fuse the concatenated feature vector to obtain a deep fusion feature vector.

[0048] The strategy generation module is used to input the deep fusion feature vector into the policy network to obtain the robot's next action; input the deep fusion feature vector into the value network to obtain the state value at the next moment; obtain the robot's next pose, the target point's next position information, and the environment state ahead of the robot's movement path at the next moment based on the action; and calculate the reward value based on the next pose, the next position information, the next environment state, and the reward function.

[0049] The training module is used to calculate the policy gradient and value loss based on the reward value and the state value; update the network parameters of the policy network using the policy gradient, update the network parameters of the value network using the value loss, and perform multiple iterations of training until the reward function converges, thereby obtaining the optimal policy network and the optimal value network respectively.

[0050] A navigation module is used to obtain the robot's control actions based on an optimal reinforcement learning model, and to complete robot navigation based on the control actions; wherein, the optimal reinforcement learning model includes the optimal policy network and the optimal value network.

[0051] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0052] (1) The robot navigation method and system based on hierarchical attention described in this invention adopts a dual-branch feature extraction strategy, combines multi-source environmental data of color and depth images, extracts spatial semantic features through a hierarchical attention architecture, and extracts dynamic pedestrian features using a multilayer perceptron. This achieves comprehensive perception of static environment and dynamic target information, effectively solving the problem that single sensor information cannot fully express environmental features, and improving the comprehensiveness and accuracy of environmental perception. By fusing multi-dimensional features through an attention mechanism, this invention strengthens the weight of key environmental features and effectively suppresses the interference of redundant information, thereby improving the effectiveness of feature fusion and providing a more reliable basis for subsequent decision-making. Precise linear velocity and angular velocity control commands are output through the policy network and value network, and iterative optimization is performed in combination with environmental rewards. This enables the robot to accurately avoid dynamic pedestrians and various obstacles, further improving the safety and efficiency of navigation.

[0053] (2) In terms of decision-making capability, the policy network generates execution actions based on deeply fused feature vectors, ensuring that the robot can flexibly respond in complex environments. The value network evaluates the state value at the next moment, helping the robot predict long-term benefits and thus select the optimal path, improving the foresight and reliability of the decision. By adopting a proximal policy optimization algorithm, the policy update amplitude is limited, avoiding instability and performance drops during training, thereby improving the training efficiency and stability of the reinforcement learning model. The dynamic fusion mechanism dynamically fuses multi-dimensional features through a self-attention mechanism, enhancing the adaptability of the reinforcement learning model to different scenarios and further improving the learning effect. Attached Figure Description

[0054] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0055] Figure 1 This is a flowchart of a robot navigation method based on hierarchical attention in a preferred embodiment of the present invention;

[0056] Figure 2 This is a diagram of the reinforcement learning model architecture in a preferred embodiment of the present invention;

[0057] Figure 3 This is a diagram of a hierarchical attention architecture in a preferred embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the risk area for pedestrian rewards in a preferred embodiment of the present invention;

[0059] Figure 5 This is a structural diagram of a robot navigation system based on hierarchical attention in a preferred embodiment of the present invention. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0061] Example 1:

[0062] Reference Figure 1 and Figure 2 As shown, this embodiment of the invention provides a robot navigation method based on hierarchical attention, including but not limited to the following steps:

[0063] S1. Obtain the robot's pose and the location information of the target point, and obtain the color image, depth image and pedestrian data in front of the robot's current movement path;

[0064] S2. Extract spatial semantic feature vectors from color and depth images using a hierarchical attention architecture; extract dynamic pedestrian feature vectors from pedestrian data using a multilayer perceptron.

[0065] S3. Preprocess the location information to obtain the target point feature vector; concatenate the target point feature vector, spatial semantic feature vector, and dynamic pedestrian feature vector to obtain the concatenated feature vector; dynamically fuse the concatenated feature vector to obtain the deep fusion feature vector.

[0066] S4. Input the deep fusion feature vector into the policy network to obtain the robot's next action; input the deep fusion feature vector into the value network to obtain the state value at the next moment.

[0067] S5. Based on the executed action, obtain the robot's next pose, the target point's next position information, and the next environmental state in front of the robot's movement path; calculate the reward value based on the next pose, the next position information, the next environmental state, and the reward function.

[0068] S6. Calculate the policy gradient and value loss based on the reward value and state value; update the network parameters of the policy network using the policy gradient and the network parameters of the value network using the value loss, and perform multiple iterations of training until the reward function converges, thereby obtaining the optimal policy network and the optimal value network respectively.

[0069] S7. Obtain the robot's control actions based on the optimal reinforcement learning model, and complete the robot navigation based on the control actions; wherein, the optimal reinforcement learning model includes the optimal policy network and the optimal value network.

[0070] This invention discloses a robot navigation method based on hierarchical attention. It employs a dual-branch feature extraction strategy, combining multi-source environmental data from color and depth images. Through a hierarchical attention architecture, spatial semantic features are extracted, and a multilayer perceptron extracts dynamic pedestrian features. This achieves comprehensive perception of both static environment and dynamic target information, solving the problem that single-sensor information cannot fully represent environmental features and improving the comprehensiveness and accuracy of environmental perception. By fusing multi-dimensional features through an attention mechanism, the weights of key environmental features are strengthened, effectively suppressing redundant information interference and improving the effectiveness of feature fusion, providing a more reliable basis for subsequent decision-making. This method, based on an end-to-end decision-making model using deep reinforcement learning, integrates perception, planning, and control, eliminating the reliance on manual parameter tuning and precise environmental modeling in traditional navigation methods. It can adapt to complex and changing environments, significantly improving the adaptability and stability of robot navigation. Furthermore, the method described in this invention outputs precise linear and angular velocity control commands through a policy network and a value network. Combined with iterative optimization of environmental rewards, this enables the robot to accurately avoid dynamic pedestrians and various obstacles, improving the safety and efficiency of navigation.

[0071] Specifically, in step S1, the robot's pose and the target point's position are obtained. The target point's position is used as the state input model to establish the robot's kinematic model, the specific expression of which is:

[0072] ;

[0073] in, Indicates the robot's pose. The symbol for the transpose of a matrix; This represents the linear velocity of the robot. Indicates the robot's heading angle. This represents the robot's angular velocity. Indicates the current moment. Indicates the next moment, This represents the Euclidean distance between the robot and the target point at the current moment.

[0074] Furthermore, in this embodiment of the invention, the robot is equipped with a depth camera, which can acquire color and depth images of the area directly in front of the robot's current movement path in real time. The image resolution is 480×480 pixels, ensuring image clarity and detail. In addition, the robot can also acquire pedestrian information through LiDAR, ordinary cameras, or other external sensors, thereby achieving comprehensive perception of the surrounding dynamic environment.

[0075] Furthermore, before step S1, the robot's initial pose and the target point's position information are initialized, and the initial state of the Markov Decision Process (MDP) is set accordingly.

[0076] Specifically, in step S2, a dual-branch feature extraction strategy is employed. First, the color image and depth image are overlaid along the channel dimension, and then spatial semantic features are extracted using a hierarchical attention (HAT) architecture. Simultaneously, dynamic pedestrian data is processed using a multilayer perceptron (MLP) to extract dynamic pedestrian features. The specific steps of this dual-branch feature extraction are as follows:

[0077] S210, in the first branch, refer to Figure 3 As shown, a hierarchical attention (HAT) architecture is used to extract spatial semantic feature vectors from color and depth images. The HAT architecture includes window partitioning, carrier token (CT) initialization, and hierarchical interactive fusion. The specific steps are as follows:

[0078] S211. Perform feature map scaling and normalization on the color image and depth image to convert them into... Channel feature map; among which, Represents a positive integer.

[0079] For example, such as Figure 2 As shown, the color image (RGB image) is first resized to 224×224 pixels to ensure consistency with the input feature map size of the depth image. For the depth image, a 3×3 median filter is used to remove isolated noise points; for the color image, a 3×3 Gaussian filter is applied to smooth texture noise, thereby improving the reliability and stability of the input data. After the above preprocessing, the processed color image and depth image are merged to obtain a feature input with a size of 224×224×4. The value is 4.

[0080] S212, Local window partitioning and carrier token initialization stage, The channel feature map is divided into multiple local windows. Each local window undergoes convolution to enhance the positional information of the features within the window, and average pooling is used to generate... Each has its own dedicated carrier token. Represents a positive integer.

[0081] For example, the 4-channel feature map is divided into 7×7 local windows. Each window is enhanced with location information through a 3×3 convolution operation, and then subjected to average pooling to generate 4 dedicated carrier tokens. At this point, The value is 4. These dedicated carrier tokens serve as a digest of window information, effectively reducing the computational complexity of cross-window interactions while retaining key information.

[0082] S213, All local windows corresponding to Each unique carrier token is aggregated to form a global carrier token set. This global carrier token set is then subjected to multi-head self-attention (MHSA) and multilayer perceptron (MLP) to achieve global information interaction across windows and update the corresponding tokens for each local window. The characteristics of a unique carrier token.

[0083] For example, for each local window Each unique carrier token is aggregated, and cross-window information interaction is achieved using a multi-head attention mechanism and a multi-layer perceptron.

[0084] S214, the updated version Each unique carrier token is concatenated with its corresponding local window to obtain the first fusion feature of each local window. In this embodiment of the invention, within each local window... The channel feature map does not interact with other windows, but is only concatenated with its corresponding updated dedicated carrier token to form the first fused feature of each local window; multi-head attention is then performed on each first fused feature to integrate the local features into the global information, thereby obtaining the second fused feature of each local window.

[0085] For example, multi-head self-attention computation is performed by concatenating the 4-channel feature map within each local window with its corresponding 4 updated dedicated carrier tokens to enhance the local key features within the window, while injecting global semantics across windows.

[0086] S215. Restore the spatial position of the second fusion feature of each local window, and assign the corresponding feature to each local window. Each dedicated carrier token undergoes upsampling to obtain sub-feature maps. All sub-feature maps are then spliced ​​and merged spatially to obtain a complete feature map. Specifically, the second fused feature of each local window is segmented and then restored to its corresponding spatial location. Based on this, the dedicated carrier token is upsampled. Finally, these sub-feature maps are spliced ​​and merged spatially to form a complete feature map containing both local details and global contextual information.

[0087] S216. Generate spatial semantic feature vectors based on the complete feature map. The specific steps of this process are as follows: First, the complete feature map is transformed into a one-dimensional vector through a flattening layer; then, the feature is refined through a fully connected layer to finally obtain the spatial semantic feature vector. .

[0088] In this embodiment of the invention, the use of a hierarchical attention architecture in the first branch can effectively construct the local and global relationships of the image, while establishing the association between the depth image and the color image, thereby achieving more efficient feature extraction and helping the robot to understand the surrounding scene more accurately.

[0089] S220. In the second branch, a multilayer perceptron is used to extract the dynamic pedestrian feature vector of the pedestrian data. The specific steps are as follows:

[0090] S221. Preprocess the pedestrian data to obtain dynamic pedestrian features; the preprocessing includes: first, sorting the pedestrian data according to the Euclidean distance between the pedestrian and the robot body; second, collecting the position of the pedestrian in the Cartesian coordinate system of the robot body. and speed And take the corresponding dimension of the pedestrian data closest to the robot.

[0091] For example, the robot acquires data on the five nearest pedestrians in its surroundings via external sensors. This data includes the pedestrians' positions and speeds. Each pedestrian has four features, resulting in a total of 20 features. Before inputting the pedestrian data into the multilayer perceptron (MLP), the pedestrians are first sorted according to their Euclidean distance from the robot. This sorting ensures the consistency of the input feature order, thereby enhancing the model's robustness to variations in the number of pedestrians.

[0092] This invention employs heterogeneous pedestrian data preprocessed by external sensors as dynamic feature input, rather than directly using the raw data collected by the sensors. The preprocessed data clearly extracts dynamic pedestrian features and removes redundant information, thereby effectively improving the training speed and performance of the model.

[0093] S222. Input the dynamic pedestrian features into the multilayer perceptron to extract the dynamic pedestrian feature vector. The multilayer perceptron includes multiple fully connected layers, each connected to a ReLU activation function.

[0094] For example, the multilayer perceptron employs a three-layer fully connected structure, with 256, 256, and 128 neurons in each layer. Each fully connected layer is followed by a ReLU activation function, enabling the model to extract nonlinear features more effectively. Through this structural design, this branch ultimately generates a dynamic pedestrian feature vector. .

[0095] This invention employs a dual-branch feature extraction strategy for heterogeneous data. On one hand, it utilizes a hierarchical attention architecture to extract spatial semantic features to capture key information in the environment. On the other hand, it extracts low-dimensional and sparse dynamic obstacle state features through a multilayer perceptron, focusing on the representation of dynamic elements. By fusing these two types of features, comprehensive perception of complex scenes is achieved, enabling the simultaneous processing of static environmental information and dynamic obstacle information.

[0096] Specifically, in step S3, the location information of the target point is processed using maximum value normalization, and the normalization threshold is set to 10 meters. The spatial semantic feature vector is then processed. (256-dimensional) dynamic pedestrian feature vector The concatenated feature vector is obtained by concatenating the 128-dimensional (128-dimensional) and the normalized target point feature vector (2-dimensional). .

[0097] Furthermore, a two-layer self-attention mechanism is used to process heterogeneous data (i.e., concatenating feature vectors). Dynamic fusion is performed to obtain a deep fused feature vector. This design achieves deep fusion of spatial features, dynamic obstacle features, and target point features, ensuring that features from different sources can work together and provide a more comprehensive basis for subsequent decision-making.

[0098] Specifically, in step S4, the deep fused feature vector is input into the common layer of the policy-value network. In the policy network (Actor), this network outputs continuous execution actions. This action includes linear velocity. and angular velocity and perform this action This serves as the control command for the robot. Simultaneously, the value network (Critic) analyzes the current state. value An evaluation is performed to obtain the state value at the next moment. .

[0099] Furthermore, both the strategy network and the value network include One fully connected neural network layer; Each fully connected neural network layer is connected to an activation function; among them... It is a positive integer greater than 1. In this embodiment, The optimal value is 3. Specifically, the first two fully connected layers of both networks are followed by a ReLU activation function. The last fully connected layer of the policy network is followed by a tanh activation function, which limits the numerical value of the action to the range (-1, 1). When control commands are transmitted to the robot, they are based on the robot's maximum linear velocity. With maximum angular velocity Scaling is performed to ensure that speed commands conform to the robot's physical limitations.

[0100] Specifically, in this embodiment of the invention, an optimal reinforcement learning model is used to achieve autonomous navigation of the robot. The optimal reinforcement learning model is obtained by optimizing and iteratively training a deep reinforcement learning model. The deep reinforcement learning model includes a feature extraction part, a feature fusion part, and a policy generation part. These parts are sequentially connected and work together to achieve the extraction and fusion of multi-source features and the output of action policies.

[0101] Furthermore, the feature extraction section includes a first branch and a second branch, which extract visual features and pedestrian state features in parallel. The first branch adopts a hierarchical attention mechanism architecture to extract spatial semantic features from the acquired color and depth images, and strengthens the feature representation ability of key regions through attention weight allocation, as detailed in step S210. The second branch performs feature extraction on pedestrian motion state information, including acquiring pedestrian position parameters in the robot's Cartesian coordinate system. With speed parameters Feature data of the pedestrian closest to the robot is selected, and feature mapping and dimensionality reduction are completed through multilayer perceptron (MLP). For details, please refer to step S220.

[0102] Furthermore, the feature fusion part receives the concatenated dual-branch feature vectors, performs adaptive weighted fusion of heterogeneous features based on the attention mechanism, flattens the fused feature tensor into a one-dimensional feature vector, and then inputs it into the policy generation part, as detailed in step S3.

[0103] Furthermore, the strategy generation part adopts an Actor-Critic dual-network architecture, in which a fully connected neural network is used to construct the Actor policy network and the Critic value network respectively, so as to jointly optimize the action policy output and value evaluation. For details, please refer to step S4.

[0104] In this embodiment of the invention, the above-mentioned deep reinforcement learning model is subjected to multiple rounds of training iterations and parameter tuning to obtain the optimal reinforcement learning model that meets the preset navigation accuracy requirements. For details, please refer to steps S5 and S6 below.

[0105] Specifically, in step S5, the robot performs the actions in step S4. This process obtains the robot's next pose, the target point's next position, and the environment's state ahead of the robot's path. Based on the next pose, position, environment state, and reward function, the reward value is calculated. Thus constructing experience tuples .in, Indicates the current state. Indicates the action to be performed. This indicates the next state. These experience tuples are stored in a replay buffer to provide batch data samples for subsequent training iterations of the deep reinforcement learning model.

[0106] Furthermore, the reward function is used to train the deep reinforcement learning model. In this embodiment of the invention, the reward function... Including target rewards Reward for getting closer to the target Collision Rewards pedestrian rewards Reward function The specific expression is:

[0107] .

[0108] Furthermore, regarding target rewards This is used to reward the robot for reaching the target point. In this embodiment of the invention, the target reward... The expression is:

[0109] ;

[0110] in, This represents the Euclidean distance between the current robot and the target point; This represents the threshold for determining whether the target point has been reached, preferably 0.5.

[0111] Furthermore, regarding the reward for being close to the target Its specific expression is:

[0112] ;

[0113] in, Indicates the weighting coefficient. This represents the Euclidean distance between the robot and the target point at the previous moment. This represents the Euclidean distance between the robot and the target point at the current moment.

[0114] Furthermore, regarding collision rewards Its specific expression is:

[0115] ;

[0116] in, This represents the minimum distance between the robot and the obstacle. The distance judgment threshold is preferably 0.2 in this embodiment of the invention. In this embodiment, the minimum distance between the robot and the obstacle is... Data can be obtained through simulation environments, depth maps, or LiDAR data. The collision detection threshold is set at 0.2 meters. When the minimum distance between the robot and an obstacle is less than 0.2 meters, a collision is considered to have occurred, and the navigation task fails. In this case, the deep reinforcement learning model will receive a negative reward, and the environment will be reset to restart training.

[0117] Furthermore, referring to Figure 4 As shown, pedestrian rewards The steps to obtain it are as follows:

[0118] First, detect pedestrians (or dynamic obstacles) and obtain the pedestrians' current speed. Based on the current speed Dynamically generate a rectangular threat area in front of the robot's movement path. Among them, the rectangular threat area Length and current speed They are positively correlated, and the relationship they satisfy is:

[0119] ;

[0120] in, Indicates the first The length of the rectangular threat area ( Figure 4 (represented by L in Chinese) Indicates the constraint time. Indicates the pedestrian's current speed ( Figure 4 (represented by V in Chinese) This represents the base distance. In this embodiment of the invention, the base distance... Default shoulder width for adult males, constraint time The timer can be dynamically adjusted to 1.5 seconds, and the width of the rectangular threat area is set to... .

[0121] Secondly, all dynamically generated rectangular threat areas Merging them yields a set of rectangular threat regions. .

[0122] Finally, obtain the robot's current pose. Determine the robot's current pose Is it within the rectangular threat area set? If yes, the pedestrian reward is negative; otherwise, the pedestrian reward is not negative.

[0123] For example, pedestrian rewards The specific expression is:

[0124] .

[0125] Specifically, in step S6, the Proximal Policy Optimization (PPO) algorithm is employed, and empirical data is used to iteratively train the deep reinforcement learning model until the reward function converges, yielding the optimal reinforcement learning model. The PPO algorithm limits the update magnitude of the policy network in each iteration, ensuring that the difference between the old and new policies remains within an acceptable range. This effectively avoids training instability or performance degradation caused by excessive policy update magnitude. This enables the PPO algorithm to achieve more stable and efficient policy iterative learning when handling mobile robot navigation tasks with continuous action spaces and high-dimensional state spaces, significantly improving the model's training effect and navigation performance.

[0126] Furthermore, in the early stages of training a deep reinforcement learning model, in order to enhance the model's exploration ability, the exploration rate is increased. Take a larger value (e.g.) With a higher exploration rate, the model can acquire richer empirical data, thereby helping the policy network learn the optimal policy more effectively. At this point, the model actively collects a large number of empirical tuples. The rich set of empirical tuples provides ample data support for the model during the initial training phase. As the number of training iterations increases, the exploration rate... The exploration rate will gradually decrease. This process allows the robot to gradually reduce random exploration during training and rely more on the learned strategy for navigation. Ultimately, the model can stably converge to the optimal navigation strategy, achieving efficient and stable navigation performance. Furthermore, during training, the exploration rate... The number of training sessions will gradually decrease to actively explore untried actions, thus avoiding being limited to local optima based on known experience.

[0127] Furthermore, in this embodiment of the invention, the current state is obtained before calculating the policy gradient and value loss using the PPO algorithm. Execute actions Reward Value The robot's termination status and the probability of actions in the old strategy Based on reward value And combine it with step S4 to obtain the state value. Value of the state at the next moment Calculate the difference residuals. Then, derive the generalized dominance estimate from these residuals. The current state Input the new policy network to obtain the action probabilities under the new policy. Then, based on the ratio of the probabilities of the new and old policy actions, a PPO-tailored policy training objective function is constructed.

[0128] Furthermore, in step S6, updating the network parameters of the policy network using policy gradients includes training the policy network according to the training objective function; wherein the expression of the training objective function is:

[0129] ;

[0130] ;

[0131] in, Represents the training objective function. This represents the estimation of the advantage function. Indicates hyperparameters; Indicates the current state Next, according to the strategy Select the action to perform. The probability of; Indicates the current state Next, according to the old strategy Select the action to perform. The probability of; Represents the shearing function, symbol for Simplified representation; This represents the expectation function. This mechanism, by restricting policy updates from exceeding the trust region, effectively avoids drastic policy fluctuations and potential collapse, significantly enhancing the stability of the training process.

[0132] Furthermore, in step S6, the process of updating the network parameters of the value network using value loss includes calculating the value loss function and updating the network parameters using the value loss function; wherein, the expression of the loss function is:

[0133] ;

[0134] in, Indicates the loss value. Represents the value network in relation to the current state. The predictive value; This represents the target value, typically an estimated return calculated from the advantage function or obtained through a specific algorithm.

[0135] The robot navigation method described in this embodiment of the invention has the following advantages through the above design:

[0136] First, in terms of environmental perception, by fusing color images, depth images, and pedestrian data, the robot can comprehensively and accurately perceive static environments and dynamic targets, solving the problem of insufficient information from a single sensor. A hierarchical attention architecture and a multilayer perceptron are used to extract spatial semantic features and dynamic pedestrian features, respectively, enhancing key information and suppressing redundant data, thereby improving the effectiveness of feature extraction.

[0137] Secondly, in terms of decision-making ability, the policy network generates precise execution actions based on deeply fused feature vectors, ensuring the robot can flexibly respond in complex environments. The value network evaluates the value of the state at the next moment, helping the robot predict long-term benefits and thus select the optimal path, improving the foresight and reliability of decision-making. By employing the proximal policy optimization (PPO) algorithm, the magnitude of policy updates is limited, avoiding instability and performance drops during training, significantly improving the model's training efficiency and stability. The dynamic fusion mechanism dynamically fuses multi-dimensional features through self-attention, enhancing the model's adaptability to different scenarios and further improving learning performance.

[0138] Third, in terms of navigation performance, the policy network and value network output precise control commands, combined with iterative optimization through environmental rewards, enabling the robot to effectively avoid dynamic pedestrians and various obstacles, thus improving navigation safety and efficiency. The end-to-end decision-making model eliminates the reliance on manual parameter tuning and precise environmental modeling found in traditional navigation methods, allowing the robot to adapt to complex and ever-changing environments, improving navigation stability and adaptability. During training, a high exploration rate is used to acquire rich empirical data; as training progresses, the exploration rate is gradually reduced, allowing the model to steadily converge to the optimal policy, optimizing the training process. Collision detection and reward mechanisms enable the model to quickly learn collision avoidance strategies during training, further enhancing navigation safety and reliability.

[0139] Fourth, regarding model generalization ability, through dual-branch feature extraction and dynamic fusion mechanisms, the model can handle various types of data, enhancing its generalization ability to different scenarios and tasks. Preprocessing pedestrian data and sorting it by distance enhances the model's robustness to changes in pedestrian numbers, improving its practicality and reliability.

[0140] In summary, these steps, by optimizing perception, decision-making, learning, and training mechanisms, comprehensively improve the robot's navigation performance, enabling it to complete tasks efficiently and safely in complex and dynamic environments.

[0141] Example 2:

[0142] Based on the same inventive concept, this embodiment provides a robot navigation system based on hierarchical attention. The principle of solving the problem is similar to that of the robot navigation method based on hierarchical attention provided in Embodiment 1, and the repeated parts will not be described again.

[0143] Reference Figure 5 As shown, this embodiment provides a robot navigation system based on hierarchical attention, used to implement the robot navigation method based on hierarchical attention described in Embodiment 1, including:

[0144] The acquisition module is used to acquire the robot's pose and the location information of the target point, as well as to acquire color images, depth images, and pedestrian data in front of the robot's current movement path;

[0145] The feature extraction module is used to extract spatial semantic feature vectors from color and depth images using a hierarchical attention architecture; and to extract dynamic pedestrian feature vectors from pedestrian data using a multilayer perceptron.

[0146] The feature fusion module is used to preprocess the location information to obtain the target point feature vector; concatenate the target point feature vector, spatial semantic feature vector, and dynamic pedestrian feature vector to obtain the concatenated feature vector; and dynamically fuse the concatenated feature vector to obtain the deep fusion feature vector.

[0147] The policy generation module is used to input the deep fusion feature vector into the policy network to obtain the robot's next action; input the deep fusion feature vector into the value network to obtain the state value at the next moment; based on the action, obtain the robot's next pose, the next position information of the target point, and the next environmental state in front of the robot's movement path; and calculate the reward value based on the next pose, the next position information, the next environmental state, and the reward function.

[0148] The training module is used to calculate the policy gradient and value loss based on the reward value and state value; the network parameters of the policy network are updated using the policy gradient, and the network parameters of the value network are updated using the value loss. The training is performed multiple times until the reward function converges, and the optimal policy network and optimal value network are obtained respectively.

[0149] The navigation module is used to obtain the robot's control actions based on the optimal reinforcement learning model, and to complete the robot's navigation based on the control actions; the optimal reinforcement learning model includes the optimal policy network and the optimal value network.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A robot navigation method based on hierarchical attention, characterized in that, include: S1. Obtain the robot's pose and the location information of the target point, and obtain the color image, depth image and pedestrian data in front of the robot's current movement path; S2. Extract the spatial semantic feature vectors of the color image and the depth image using a hierarchical attention architecture; The dynamic pedestrian feature vector of the pedestrian data is extracted using a multilayer perceptron. S3. Preprocess the location information to obtain the target point feature vector; The target point feature vector, the spatial semantic feature vector, and the dynamic pedestrian feature vector are concatenated to obtain a concatenated feature vector; the concatenated feature vector is then dynamically fused to obtain a deep fused feature vector. S4. Input the deep fusion feature vector into the policy network to obtain the robot's next action; input the deep fusion feature vector into the value network to obtain the state value at the next moment; S5. Based on the executed action, obtain the robot's next pose, the target point's next position information, and the next environmental state in front of the robot's movement path. The reward value is calculated based on the next-time pose, the next-time position information, the next-time environmental state, and the reward function; S6. Calculate the policy gradient and value loss based on the reward value and the state value; update the network parameters of the policy network using the policy gradient and update the network parameters of the value network using the value loss, and perform multiple iterations of training until the reward function converges, thereby obtaining the optimal policy network and the optimal value network respectively. S7. Obtain the robot's control actions based on the optimal reinforcement learning model, and complete robot navigation based on the control actions; wherein, the optimal reinforcement learning model includes the optimal policy network and the optimal value network.

2. The robot navigation method based on hierarchical attention according to claim 1, characterized in that, The step S2, which involves extracting the spatial semantic feature vectors of the color image and the depth image using a hierarchical attention architecture, is as follows: Convert the color image and the depth image to Channel feature map; where Represents positive integers; The The channel feature map is divided into multiple local windows. Each local window undergoes convolution to enhance the positional information of the features within the window, and average pooling is used to generate... Each unique carrier token, among which Represents positive integers; All the local windows corresponding to the Each dedicated carrier token is aggregated to form a global carrier token set; multi-head self-attention and multi-layer perceptron are applied to the global carrier token set to update the corresponding token for each local window. The characteristics of a unique carrier token; The updated version Each unique carrier token is concatenated with the corresponding local window to obtain the first fusion feature of each local window; Multi-head self-attention calculation is performed on each of the first fusion features to obtain the second fusion feature of each local window; The second fusion feature of each local window is spatially restored, and the corresponding feature of each local window is... Each dedicated carrier token is upsampled to obtain a sub-feature map; all the sub-feature maps are spliced ​​and merged in the spatial dimension to obtain a complete feature map; a spatial semantic feature vector is generated based on the complete feature map.

3. The robot navigation method based on hierarchical attention according to claim 1, characterized in that, The step S2, which involves extracting the dynamic pedestrian feature vector from the pedestrian data using a multilayer perceptron, is as follows: The pedestrian data is preprocessed to obtain dynamic pedestrian features; The preprocessing includes sorting the pedestrian data according to the Euclidean distance between the pedestrian and the robot body; The dynamic pedestrian features are input into a multilayer perceptron to extract dynamic pedestrian feature vectors; The multilayer perceptron includes multiple fully connected layers, each of which is connected to a ReLU activation function.

4. The robot navigation method based on hierarchical attention according to claim 1, characterized in that, The reward function includes target reward, proximity reward, collision reward, and pedestrian reward.

5. A robot navigation method based on hierarchical attention according to claim 4, characterized in that, The expression for the reward for being close to the target is: ； in, Indicates a reward for getting closer to the target. Indicates the weighting coefficient. This represents the Euclidean distance between the robot and the target point at the previous moment. This represents the Euclidean distance between the robot and the target point at the current moment. This represents the threshold for determining whether the target point has been reached.

6. The robot navigation method based on hierarchical attention according to claim 4, characterized in that, The expression for the collision reward is: ； in, Indicates collision reward, This represents the minimum distance between the robot and the obstacle. This indicates the distance judgment threshold.

7. A robot navigation method based on hierarchical attention according to claim 4, characterized in that, The steps for obtaining the pedestrian reward are as follows: Obtain the pedestrian's current speed; dynamically generate a rectangular threat region ahead of the robot's movement path based on the current speed; wherein the length of the rectangular threat region and the current speed satisfy the following relationship: ； in, Indicates the first The length of the rectangular threat area, Indicates the constraint time. Indicates the pedestrian's current speed. Indicates the base distance; Merge all dynamically generated rectangular threat regions to obtain a set of rectangular threat regions; Obtain the robot's current position and determine whether the robot's current position is within the rectangular threat area set; if yes, the pedestrian reward is negative; otherwise, the pedestrian reward is not negative.

8. The robot navigation method based on hierarchical attention according to claim 1, characterized in that, Step S6, updating the network parameters of the policy network using the policy gradient, includes training the policy network according to a training objective function; wherein the expression of the training objective function is: ； ； in, Represents the training objective function. This represents the estimation of the advantage function. Indicates hyperparameters; Indicates the current state Next, according to the strategy Select the action to perform. The probability of; Indicates the current state Next, according to the old strategy Select the action to perform. The probability, Represents the shearing function. This represents the expected function.

9. A robot navigation method based on hierarchical attention according to claim 1, characterized in that, In step S6, the process of updating the network parameters of the value network using the value loss includes calculating a loss function and updating the network parameters using the loss function; wherein, the expression of the loss function is: ； in, Indicates the loss value. Represents the value network in relation to the current state. The predictive value Indicates the target value.

10. A robot navigation method based on hierarchical attention according to claim 1, characterized in that, Both the strategy network and the value network include One fully connected neural network layer; among which... Each of the fully connected neural network layers is connected to an activation function; wherein... It is a positive integer greater than 1.

11. A robot navigation system based on hierarchical attention, used to implement the robot navigation method based on hierarchical attention as described in any one of claims 1 to 10, characterized in that, include: The acquisition module is used to acquire the robot's pose and the location information of the target point, and to acquire the color image, depth image and pedestrian data in front of the robot's current movement path; The feature extraction module is used to extract the spatial semantic feature vectors of the color image and the depth image using a hierarchical attention architecture; The dynamic pedestrian feature vector of the pedestrian data is extracted using a multilayer perceptron. The feature fusion module is used to preprocess the location information to obtain the target point feature vector; The target point feature vector, the spatial semantic feature vector, and the dynamic pedestrian feature vector are concatenated to obtain a concatenated feature vector; the concatenated feature vector is then dynamically fused to obtain a deep fused feature vector. The strategy generation module is used to input the deep fusion feature vector into the policy network to obtain the robot's next action; input the deep fusion feature vector into the value network to obtain the state value at the next moment; and obtain the robot's next pose, the target point's next position information, and the next environmental state in front of the robot's movement path based on the action. The reward value is calculated based on the next-time pose, the next-time position information, the next-time environmental state, and the reward function; The training module is used to calculate the policy gradient and value loss based on the reward value and the state value; update the network parameters of the policy network using the policy gradient, update the network parameters of the value network using the value loss, and perform multiple iterations of training until the reward function converges, thereby obtaining the optimal policy network and the optimal value network respectively. A navigation module is used to obtain the robot's control actions based on an optimal reinforcement learning model, and to complete robot navigation based on the control actions; wherein, the optimal reinforcement learning model includes the optimal policy network and the optimal value network.

Images