Unmanned system sensing, obstacle avoidance and navigation method based on deep reinforcement learning

Abstract

The invention discloses an unmanned system sensing, obstacle avoidance and navigation method based on deep reinforcement learning. The method comprises the following steps: constructing an obstacle state prediction system based on BEV; sensor data and the motion state of the unmanned system are fused into intermediate environment representation through a dynamic environment sensing module; the actor network inputs intermediate environment characterization and a navigation task target point and outputs navigation action data, and the critic network evaluates an action value through a multi-target reward function; the submerged space obstacle state prediction module performs prediction by comparing observation differences of front and back frames, and the gradient is used for updating parameters of the submerged space obstacle state prediction module and synchronously updating the parameters of the actor and the critic network; and after training is completed, deploying the actor network to the unmanned system, inputting the navigation task target point and the intermediate environment representation into the actor network, outputting a navigation control action, and executing navigation and obstacle avoidance tasks of the unmanned system based on the navigation control action. The navigation efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of autonomous navigation and obstacle avoidance of unmanned systems, and particularly relates to a method for perception, obstacle avoidance and navigation of unmanned systems based on deep reinforcement learning. BACKGROUND

[0002] With the continuous development and innovation of unmanned systems, they have been widely applied to autonomous inspection, indoor guidance, logistics distribution, warehouse transportation and other scenarios. Such scenarios usually have mixed static obstacles (such as tables, chairs, walls, and shelves) and dynamic obstacles (such as pedestrians, mobile robots, and collaborative equipment). In complex environments, it is a basic requirement for unmanned systems to achieve safe and efficient autonomous navigation and effectively avoid obstacles to complete autonomous tasks. The main technologies and corresponding key problems of existing work include: (1) Deep reinforcement learning and high-dimensional representation method: Deep reinforcement learning has been widely applied to autonomous navigation and obstacle avoidance due to its strong representation ability for high-dimensional states and data efficiency advantage. Existing methods usually directly use raw sensor data, design a convolutional variational autoencoder, or use a feature extractor with an attention module for simple processing to map them to an effective feature space.

[0003] Problem: Sensor observations usually contain a large amount of information irrelevant to navigation. Directly using raw observations as input or only performing simple processing will seriously interfere with the generation of downstream unmanned system navigation decisions. At the same time, raw depth images or two-dimensional radar observations are difficult to directly distinguish static and dynamic obstacles; and the perspective transformation, parallax, and time consistency mismatch caused by the movement of the unmanned system device will further amplify the difficulty of differentiation in continuous observations.

[0004] (2) Model-based navigation and obstacle avoidance method: Traditional model-based navigation and obstacle avoidance methods explicitly model the kinematics of the unmanned system and the environment to predict and evaluate future actions. Such methods construct cost maps or use time-parameterized pose sequences to search or optimize collision-free trajectories that satisfy kinematic constraints in offline or online frameworks.

[0005] Problem: Such methods rely on accurate obstacle motion models, high-fidelity sensor data, and a large number of parameter hand-tuning. In dynamic and complex environments, not only is the modeling and tuning cost high, but also the unknown dynamic behavior shows poor adaptability and generalization ability.

[0006] (3) Semantic extraction based on target detection: In order to effectively distinguish static obstacles and dynamic obstacles in the original depth image or the scanning data of the laser radar, the existing method usually uses an external target detection tool such as a YOLO (You Only Look Once) algorithm to extract semantic information of the obstacles in the environment.

[0007] Problem: Relying on a third-party detection module to extract environmental semantics will cause the environmental perception ability of the system to be highly coupled with the detection accuracy and performance of the external tool, and will introduce additional uncertainty due to detection errors, thereby increasing the risk of subsequent environmental perception and obstacle avoidance decision-making. At the same time, such overly detailed semantic information as obstacle category is not the focus of the navigation task, and more attention should be paid to relevant information in the semantic granularity of obstacle motion state.

[0008] (4) BEV (Bird's eye view, bird's eye view) based intermediate representation method: By using BEV, redundant noise caused by perspective or height differences and high-dimensional perception details can be naturally filtered out to better represent the obstacle motion mainly occurring in the horizontal plane. For example, an occupancy grid is used to fill in the pedestrian pose, or a dense-sparse network is designed to obtain BEV features from the original point cloud input.

[0009] Problem: The existing BEV construction, artificial design features or direct encoding strategy has limited performance in selecting core features for navigation, is difficult to actively capture spatiotemporal representations related to dynamic environments, and is prone to introduce redundant or navigation-irrelevant features, thereby weakening decision-making efficiency and generalization ability. At the same time, some methods use external target detection modules to construct obstacle maps, which will introduce additional errors.

[0010] (5) Representation learning based on auxiliary tasks: To extract low-dimensional features from high-dimensional sensor data that are effective for navigation tasks, existing work usually introduces auxiliary tasks to achieve this goal, such as image reconstruction or static scene prediction, so as to use the supervision signal provided by the auxiliary task to constrain the representation space and enhance environmental understanding.

[0011] Problem: Such reconstruction or static prediction auxiliary tasks mainly focus on restoring the appearance of the scene or the short-term static structure, and are difficult to capture the motion information of the obstacles in the dynamic scene, such as speed, trajectory or short-term motion trend. Therefore, the encoder is difficult to sufficiently improve the prediction ability of dynamic behavior through such auxiliary tasks, limiting the improvement of the obstacle avoidance performance and generalization ability of the navigation strategy in complex dynamic environments. SUMMARY

[0012] Therefore, the purpose of the present application is to provide an unmanned system perception, obstacle avoidance and navigation method based on deep reinforcement learning.

[0013] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows: This invention provides a method for perception, obstacle avoidance, and navigation of unmanned systems based on deep reinforcement learning, comprising the following steps: Step 1: Construct an obstacle state prediction system based on BEV, including a dynamic environment perception module, a latent space obstacle state prediction module, and a deep reinforcement learning framework, wherein the deep reinforcement learning framework includes an actor network and a critic network. Step 2: The unmanned system interacts with the environment to obtain sensor data and the motion state of the unmanned system. The sensor data and the motion state of the unmanned system are fused into an intermediate environment representation through the dynamic environment perception module. The intermediate environment representation and the navigation task target point are input into the actor network and the critic network. The intermediate environment representation is input into the latent space obstacle state prediction module. Step 3: The intermediate environment representation and navigation task target point are processed through the actor network to output navigation action data. The critic network evaluates the value of the navigation action data based on the multi-objective reward function, intermediate environment representation and navigation task target point. Step 4: The latent space obstacle state prediction module performs the prediction task by comparing the differences between the observation data of the intermediate environment in the latent space before and after the previous frame, and feeds back the generated prediction gradient to the latent space obstacle state prediction module and updates the parameters of the latent space obstacle state prediction module to assist in training the dynamic perception ability of the actor network, and simultaneously updates the parameters of the actor network and the critic network. Step 5: Deploy the trained actor network on the unmanned system, input the navigation task target point and the intermediate environment representation obtained by the dynamic environment perception module in the current dynamic environment into the trained actor network, and output the navigation control actions of the unmanned system. Step 6: Execute the navigation and obstacle avoidance tasks of the unmanned system based on the navigation control actions.

[0014] Furthermore, step 2 specifically includes: Step 21: Collect sensor data, including depth images acquired using a depth camera and environmental point cloud data acquired using a 2D LiDAR. Step 22: Project the depth image and environmental point cloud data onto a unified three-dimensional point cloud coordinate system to obtain a three-dimensional point cloud, and extract the spatial attributes of the three-dimensional point cloud at the same time. Step 23: Using the unmanned system as the center, discretize the target area within the preset range into a raster map; Step 24: Project the 3D point cloud onto the raster map in the vertical direction; for each raster cell, extract the maximum height value in the 3D point cloud as the spatial feature of that raster cell; Step 25: Combine the motion state of the unmanned system itself, calculate and fill in the relative velocity components of each grid cell in the coordinate system of the unmanned system to form velocity features; Step 26: Assign the default maximum height value to the spatial characteristics of the grid in the unobservable region, and set the velocity characteristic to zero; the unobservable region includes areas beyond the sensor's maximum range or field of view, as well as occluded areas; Step 27: Connect the spatial features and velocity features to form an intermediate environment representation, and normalize the intermediate environment representation. Step 28: Input the normalized intermediate environment representation and navigation task target point into the actor network and critic network, and input the normalized intermediate environment representation into the latent space obstacle state prediction module.

[0015] Furthermore, step 25 specifically includes: Step 251: Calculate the relative velocity in the unmanned system coordinate system at the center of each grid by transforming the rotation matrix:

[0016] in, This represents the relative velocity of the navigation target point in the coordinate system of the unmanned system. express of x Axial components, express of y Axial components; This represents the relative velocity of the navigation mission target point in the world coordinate system relative to the coordinate system of the unmanned system. express of x Axial components, express of y Axial components; It is a rotation matrix. Yaw angle; Step 252, and Fill the two velocity features of the grid respectively; Step 253: Generate two types of BEV velocity maps: one type is unlabeled velocity features directly generated based on the current observations, which serves as the raw input to the actor network; the other type is labeled velocity features generated by combining dynamic obstacle attitude data from the Gazebo simulator, which serves as a global information sequence for obstacle prediction.

[0017] Furthermore, the process of generating the labeled velocity features is as follows: Unlabeled velocity features are refined using dynamic obstacle attitude data to generate new velocity features; Replace the data at the corresponding grid positions with the relative velocity of the pedestrian in the coordinate system of the unmanned system. For static obstacles or free space, let... For moving pedestrians, make To obtain the velocity components in the coordinate system of the unmanned system; where, This represents the velocity vector of the unmanned system itself. Indicates the speed of pedestrians; The processed new velocity features are labeled to obtain labeled velocity features, while simultaneously encoding the spatial position and velocity information of dynamic obstacles.

[0018] Furthermore, step 27 specifically includes: Step 271: Connect the spatial features with the unlabeled velocity features and the labeled velocity features respectively to form two three-channel intermediate environment representations; Step 272: Using the maximum absolute value scaling process, normalize the observation data of the three channels to the following formula. Within the range:

[0019] in, express Time Channel Observational data, This represents the minimum value of the observed data for that channel. This indicates the maximum value of the observed data for that channel.

[0020] Furthermore, step 3 specifically includes: Step 31: The deep reinforcement learning box adopts the Soft Actor-Critic algorithm framework, including an actor network and a critic network. The actor network includes a first encoder and an action output head, which are used to process intermediate environmental representations and navigation task target points to perceive the environment and output navigation action data. The critic network includes a second encoder and two Q-value networks, which are used to process intermediate environmental representations and navigation task target points and evaluate the quality of the navigation action data given under the current observation. Step 32: The actor network receives intermediate environment representations and navigation task target points. The intermediate environment representations include continuous... Frame observation data The observation data for each time frame consists of the 3D point cloud at the current time. The motion state of unmanned systems and sub-target point location Composition; Output is navigation action data of the unmanned system; The navigation action data includes linear velocity and angular velocity ,Right now ; Step 33: The critic network evaluates and optimizes the navigation action data based on a multi-objective reward function, intermediate environment representation, and navigation task target point. The multi-objective reward function includes at least target arrival reward, collision penalty reward, heuristic distance reward, and action smoothness reward.

[0021] Furthermore, step 33 specifically includes: The multi-objective reward function Represented as:

[0022] in, A reward is given for reaching the target point, i.e., a positive reward is given when the unmanned system successfully reaches the target point of the navigation task; Collision penalty reward, that is, the unmanned system gives a negative reward when a collision occurs; Heuristic distance reward; Reward for smoothness of motion; The target reached reward Defined as: in, express The distance between the unmanned system and the navigation mission target point. This indicates the preset threshold range for navigation task target points. At that time, it is considered that the unmanned system has successfully reached the navigation mission target point; This indicates the specific value of the reward received by the unmanned system after successfully reaching the navigation mission's target point. The collision penalty reward Defined as:

[0023] in, This indicates that a collision has occurred. The specific numerical penalty incurred upon a collision; The heuristic distance reward Defined as:

[0024] in, These are the weighting coefficients. express The distance between the unmanned system and the target point of the navigation mission; Represents the distance change between consecutive frames, heuristic distance reward Based on the distance change between consecutive frames Adjustments are made when the unmanned system approaches the navigation mission target point. A positive reward is awarded when the unmanned system moves far away from the navigation mission target point. If so, a negative reward will be given; The smoothness reward of the action Defined as:

[0025] in, and Weighting coefficients; action smoothness reward Calculated from the current state of motion, linear velocity The larger the angular velocity, the higher the positive reward. The smaller the value, the lower the negative reward / penalty.

[0026] Furthermore, in step 4, the latent space obstacle state prediction module performs a prediction task by comparing the differences between the observation data of the intermediate environment in the latent space before and after the previous frame, and feeds back the generated prediction gradient to the latent space obstacle state prediction module to update the parameters of the latent space obstacle state prediction module, so as to assist in training the dynamic perception capability of the actor network, specifically including: Step 41: The latent space obstacle state prediction module adopts an asymmetric self-supervised contrastive learning architecture, including the original encoder and its corresponding momentum encoder, the original projection layer and its corresponding momentum projection layer, the prediction layer, and the Transformer-based prediction decoder. Step 42: Sample the original observation sequence, action sequence, and labeled observation sequence, which are composed of intermediate environment representations, from the experience replay buffer; Step 43: Use the original encoder to map the original observation sequence into the latent space to obtain the latent state sequence; Step 44: Input the latent state sequence and action sequence into the prediction decoder to predict the future latent state sequence and obtain the predicted state sequence. Step 45: Process the predicted state sequence through the original projection layer and the prediction layer to obtain the prediction result with semantic segmentation parsing information; Step 46: Encode the labeled observation sequence into a momentum latent sequence using a momentum encoder, and then generate a target representation from the momentum latent sequence using a momentum projection layer; Step 47: The parameters of the momentum encoder and the momentum projection layer are updated using an exponential moving average of the momentum encoder weights and the momentum projection layer weights. The update formula is as follows:

[0027]

[0028] in, Indicates momentum encoder The parameters, It is the momentum coefficient. Indicates the momentum encoder weights; Indicates momentum projection layer The parameters, Indicates the momentum projection layer weights; Step 48: By calculating the cosine similarity between the prediction result and the target representation, construct an auxiliary loss function for the auxiliary prediction task to update the parameters of the latent space obstacle state prediction module.

[0029] Furthermore, step 4 involves synchronously updating the parameters of the actor network and the critic network, specifically including: Step 49: Initialize the prediction layer parameters, actor network parameters, critic network parameters and their target network parameters, entropy parameters in the Soft State-Action function, batch size and experience replay buffer of the latent space obstacle state prediction module; Step 410: Perform multiple training rounds, each training round including: (1) Initialize the current state, the pose of the unmanned system and the target state from the environment, and generate the corresponding intermediate environment representation; (2) Execute multiple time steps within a single training round, each time step including: Sampling actions are performed according to the current strategy; Perform actions, interact with the environment, and obtain rewards and the next state; Generate intermediate environment representations for the state at the next time step; Store the interaction data in the experience replay buffer; Batch data is sampled from the experience replay buffer, and the critic network parameters, actor network parameters, prediction layer parameters, or target network parameters are updated based on the sampled batch data; where: The process of updating the critic network parameters includes: calculating the critic network loss using the minimization of the temporal difference loss function, and updating the critic network parameters; The process of updating the actor network parameters includes: calculating the actor network loss by maximizing the soft state-action function, and updating the actor network parameters; The update process of the prediction layer parameters includes: sampling batch data from the experience replay buffer and updating the prediction layer parameters according to the auxiliary loss function; The update process of the target network parameters includes: implementing it through soft update or hard update, including copying or partially copying the critic network parameters to the target network.

[0030] Furthermore, step 5 includes: Step 51: Deploy the trained actor network on the unmanned system; Step 52: Acquire sensor data and unmanned system motion state in the current dynamic environment, and fuse sensor data and unmanned system motion state into an intermediate environment representation through the dynamic environment perception module; Step 53: Input the intermediate environment representation corresponding to the current dynamic environment and the navigation task target point into the trained actor network; the intermediate environment representation includes continuous Frame observation data The observation data for each time frame consists of the 3D point cloud at the current time. The motion state of unmanned systems and sub-target point location The system consists of components; the output is navigation action commands for the unmanned system, which include linear velocity and angular velocity.

[0031] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: 1. Improve the navigation success rate of unmanned systems: In independent evaluations with different population densities, the method of this invention significantly improves the navigation success rate of unmanned systems compared with the best state-of-the-art method and the baseline method (Table 1), and is also significantly better than the compared methods in completely unseen test scenarios (Table 3), indicating that the present invention has excellent robustness and can effectively improve the success rate of unmanned systems reaching the mission target point in complex scenarios.

[0032] 2. Improved Navigation Efficiency and Reduced Collision Risk: While surpassing the success rate of state-of-the-art methods and traditional baselines, this invention generally achieves superior path efficiency (SPL) results (Tables 1 and 3). Furthermore, in ablation and comparison experiments in high-density pedestrian scenes, the collision rate (CR) of this invention is significantly better than existing methods (Tables 1, 2, and 3). This indicates that this invention, through enhanced perception and potential spatial motion prediction, effectively reduces the collision probability of unmanned systems in unstructured scenes and significantly improves navigation efficiency.

[0033] 3. Enhance the cross-scenario generalization capability of unmanned systems: In independent evaluation scenarios where no training was conducted (Table 3), the method of the present invention can still maintain a high arrival success rate and navigation efficiency, indicating that the present invention can be effectively transferred to unknown scenarios for use, thereby improving the cross-scenario generalization and adaptability of unmanned systems in different scenarios.

[0034] 4. Enhanced spatiotemporal dynamic information extraction capability: This invention uses the spatial representation provided by the DEP module and the self-supervised auxiliary task of the LOSP module to enable the encoder to effectively capture the motion features such as the position, velocity and trajectory of obstacles in the potential space, thereby significantly improving the ability to perceive and predict the trajectory of static and dynamic targets (Table 2).

[0035] 5. Eliminating dependence on external detection modules to improve perception robustness: This invention fuses depth images, 2D LiDAR data, and the unmanned system's own motion state in a bird's-eye view coordinate system to construct a BEV-DEP dynamic environment representation. This eliminates the need for external target detection or tracking modules to obtain a unified intermediate environment representation. This intermediate environment representation effectively filters out information and noise irrelevant to the navigation task, improving the robustness of the environmental perception results.

[0036] 6. Joint Optimization of Perception and Decision-Making to Enhance Navigation Performance: This invention shares an encoder between the LOSP module's self-supervised auxiliary task and the main navigation task, and trains them jointly. This enables the encoder to receive gradients from both navigation and prediction tasks, thus endowing it with the dual capabilities of environmental perception and prediction of future scene dynamics. This joint optimization mechanism helps the strategy more accurately identify static and dynamic obstacles and predict their behavioral states, thereby improving the navigation efficiency and obstacle avoidance success rate of unmanned systems in complex dynamic environments. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is an execution flowchart of a perception, obstacle avoidance and navigation method for unmanned systems based on deep reinforcement learning, provided by an embodiment of the present invention.

[0039] Figure 2 This is the overall structure of the BEV-OSP framework in this invention. Among them, the sub- Figure 2 (a) represents the sensor model and raw data input, sub- Figure 2(b) indicates the intermediate feature processing flow, sub- Figure 2 (c) represents the deep reinforcement learning framework structure, sub- Figure 2 (d) Prediction task learning network, sub- Figure 2 (e) indicates the encoder structure (the structure is consistent in the actor network and the critic network).

[0040] Figure 3 This describes the intermediate feature generation process of the BEV-OSP framework in this invention.

[0041] Figure 4 This is the velocity annotation process of the BEV-OSP framework in this invention.

[0042] Figure 5 This is the learning architecture for obstacle prediction auxiliary tasks in the latent space of the BEV-OSP framework in this invention.

[0043] Figure 6 The obstacle prediction decoder of the BEV-OSP framework in this invention The structure.

[0044] Figure 7 This is a schematic diagram of the Gazebo simulation environment in Embodiment 1 of the present invention. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Please see Figures 1-7 The present invention provides a method for perception, obstacle avoidance, and navigation of unmanned systems based on deep reinforcement learning, comprising the following steps: Step 1: Construct an obstacle state prediction system based on Bird's-Eye View (BEV-OSP), including a dynamic environment perception module, a latent space obstacle state prediction module, and a deep reinforcement learning framework, which includes an actor network and a critic network. BEV-OSP fuses sensor data and the motion state of an unmanned system into a grid map as an intermediate environmental representation, explicitly encoding the spatial distribution and short-term motion information of obstacles. Within a deep reinforcement learning framework, it introduces a self-supervised auxiliary task based on BYOL (Bootstrap Your Own Latent, a self-supervised learning algorithm) contrastive learning. This allows the encoder to implicitly predict obstacle motion states in the latent space and gain robust environmental perception capabilities from the auxiliary task, thus enhancing dynamic obstacle recognition and prediction capabilities without the need for a third-party detector. The core of the BEV-OSP framework lies in representing observation data from a BEV perspective and further predicting and labeling the future states of dynamic obstacles observed, achieving efficient perception of the dynamic environment. The framework mainly consists of two modules: 1) Dynamic Environment Perception (DEP) and 2) Latent Obstacle State Prediction (LOSP). The main function of DEP is to output the intermediate environmental representation, while LOSP is used to predict future environmental changes.

[0047] The overall framework structure of BEV-OSP is as follows: Figure 2 As shown, using from Figure 2 The sensor data in (a) combined with the current action and posture of the unmanned system Figure 2 (b) The dynamic environment perception module obtains The BEV-DEP grid map is used as an intermediate environment representation, and then the BEV-DEP images from three consecutive time steps and the sub-target points obtained using a pure tracking method are used as input to the deep reinforcement learning framework. Figure 2 As shown in (c), the deep reinforcement learning box adopts the Soft Actor-Critic (SAC) algorithm framework, including an actor network and a critic network. The actor network includes a first encoder and an action output head for processing intermediate environment representations. The system uses consecutive BEV-DEP frames and navigation mission target points to perceive the environment and output navigation action data (linear velocity and angular velocity). The critic network includes a second encoder and two Q-value networks to evaluate the quality of the navigation action data given under the current observation. Furthermore, it uses... Figure 2 (d) illustrates a BYOL-like contrastive learning architecture for assisting in dynamic obstacle state prediction. Both the first encoder in the actor network and the second encoder in the critic network employ a structurally consistent ResNet (Residual Network) architecture, such as... Figure 2As shown in (e), the second encoder network in the critic network is updated by simultaneously receiving gradients from the reinforcement learning task and the prediction auxiliary task.

[0048] Step 2: The unmanned system interacts with the environment to acquire sensor data and its motion state. The dynamic environment perception module fuses this sensor data and motion state into an intermediate environment representation (this intermediate environment representation serves as training data for the actor network, critic network, and latent space obstacle state prediction module). The intermediate environment representation and the navigation task target point are then input into the actor network and critic network, and the intermediate environment representation is input into the latent space obstacle state prediction module. The main function of this dynamic environment perception module is to output the intermediate environment representation. Figure 3 The process of generating intermediate environmental representations by the dynamic environment perception module is demonstrated.

[0049] In this embodiment, step 2 specifically includes: Step 21, Sensor Data Acquisition: Acquire sensor data, including depth images obtained using a depth camera and environmental point cloud data obtained using a 2D LiDAR. Step 22, Data Unification and Projection: Project the depth image and environmental point cloud data onto a unified three-dimensional point cloud coordinate system to obtain a three-dimensional point cloud, and extract the spatial attributes of the three-dimensional point cloud at the same time. Step 23, Area Rasterization: Centered on the unmanned system, rasterize the target area within a preset range ( (Region) by The resolution is discretized into Raster map; Step 24, Spatial Channel Encoding (BEV-VelocityX and BEV-VelocityY): Project the 3D point cloud onto the raster map in the vertical direction; for each raster cell, extract the maximum height Z value in the 3D point cloud as the spatial feature of the raster cell, denoted as BEV-MaxHeight. Step 25, Velocity Channel Encoding (BEV-MaxHeight): Combining the unmanned system's own motion state, calculate and fill in the relative velocity components of each grid cell in the unmanned system's coordinate system to form velocity features; specifically including: Step 251: Calculate the relative velocity in the unmanned system coordinate system at the center of each grid by transforming the rotation matrix:

[0050] in, This represents the relative velocity of the navigation target point in the coordinate system of the unmanned system. express of xAxial components, express of y Axial components; This represents the relative velocity of the navigation mission target point in the world coordinate system relative to the coordinate system of the unmanned system. express of x Axial components, express of y Axial components; It is a rotation matrix. Yaw angle; Step 252, and Fill the two velocity features of the grid respectively; Step 253, in order to obtain and ,like Figure 4 As shown, two types of BEV velocity maps are generated: one is an unlabeled velocity feature (BEV-Velocity) directly generated based on the current observation, which serves as the raw input to the actor network; the other is an labeled velocity feature (BEV-AnnotatedVelocity) generated by combining obstacle pose (pedestrian position and velocity) from the Gazebo simulator, which is used as a global information sequence for obstacle prediction. The specific process for generating the labeled velocity features is as follows: The unlabeled velocity features (BEV-Velocity) are refined using dynamic obstacle attitude data to generate new velocity features (BEV-AnnotatedVelocity). Replace the data at the corresponding grid position with the relative velocity of the pedestrian in the coordinate system of the unmanned system. Specifically, as follows: Figure 4 As shown, for static obstacles or free space, let For moving pedestrians, make To obtain the velocity components in the coordinate system of the unmanned system; where, This represents the velocity vector of the unmanned system itself. Indicates the speed of pedestrians; The processed new velocity features are labeled to obtain labeled velocity features, while simultaneously encoding the spatial position and velocity information of dynamic obstacles; Step 26: Process unobservable areas: Assign the default maximum height value to the spatial characteristics of the grid in the unobservable area, and set the velocity characteristic to zero; the unobservable area includes areas beyond the sensor's maximum range or field of view, as well as occluded areas (such as behind walls). Step 27, Channel Connection and Normalization: Connect the spatial features and velocity features to form an intermediate environment representation, and normalize the intermediate environment representation; specifically including: Step 271: Connect the spatial feature (BEV-MaxHeight) with the unlabeled velocity features (BEV-VelocityX, BEV-VelocityY) and the labeled velocity feature (BEV-AnnotatedVelocity) respectively to form two three-channel intermediate environment representations; such as Figure 4 As shown, the two intermediate environment representations are used for: ① providing raw observation input to the actor network (policy network); ② pairing with labeled observations from Gazebo as global supervision for obstacle motion prediction.

[0051] Step 272: Using the maximum absolute value scaling process, normalize the observation data of the three channels to the following formula. Within the range:

[0052] in, express Time Channel Observational data, This represents the minimum value of the observed data for that channel. This indicates the maximum value of the observation data for that channel (such as the relative speed boundary of vehicles and pedestrians).

[0053] Step 28: Input the normalized intermediate environment representation and navigation task target point into the actor network and critic network, and input the normalized intermediate environment representation into the latent space obstacle state prediction module.

[0054] Step 3: The intermediate environment representation and navigation task target point are processed through the actor network to output navigation action data. The critic network evaluates the value of the navigation action data based on the multi-objective reward function, intermediate environment representation and navigation task target point. In this embodiment, step 3 specifically includes: Step 31: The deep reinforcement learning box adopts the Soft Actor-Critic algorithm framework, including an actor network and a critic network. The actor network includes a first encoder and an action output head, which are used to process intermediate environmental representations and navigation task target points to perceive the environment and output navigation action data. The critic network includes a second encoder and two Q-value networks, which are used to process intermediate environmental representations and navigation task target points and evaluate the quality of the navigation action data given under the current observation. Step 32: The actor network receives intermediate environment representations and navigation task target points. The intermediate environment representations include continuous... Frame observation data The observation data for each time frame consists of the 3D point cloud at the current time. The motion state of unmanned systems and sub-target point location Composition; Output is navigation action data of the unmanned system; The navigation action data includes linear velocity and angular velocity ,Right now ; Step 33: The critic network evaluates and optimizes the navigation action data based on a multi-objective reward function, intermediate environment representation, and navigation task target point. The multi-objective reward function includes at least target arrival reward, collision penalty reward, heuristic distance reward, and action smoothness reward.

[0055] In this embodiment, step 33 specifically includes: The multi-objective reward function Represented as:

[0056] in, A reward is given for reaching the target point, i.e., a positive reward is given when the unmanned system successfully reaches the target point of the navigation task; Collision penalty reward, that is, the unmanned system gives a negative reward when a collision occurs; Heuristic distance reward; Reward for smoothness of motion; The target reached reward Defined as: in, express The distance between the unmanned system and the navigation mission target point. This indicates the preset threshold range for navigation task target points. At that time, it is considered that the unmanned system has successfully reached the navigation mission target point; This indicates the specific value of the reward received by the unmanned system after successfully reaching the navigation mission's target point. The collision penalty reward Defined as:

[0057] in, This indicates that a collision has occurred. This indicates the specific numerical penalty incurred when a collision occurs. The heuristic distance reward Defined as:

[0058] in, These are the weighting coefficients. express The distance between the unmanned system and the target point of the navigation mission; Represents the distance change between consecutive frames, heuristic distance reward Based on the distance change between consecutive frames Adjustments are made when the unmanned system approaches the navigation mission target point. A positive reward is awarded when the unmanned system moves far away from the navigation mission target point. If so, a negative reward will be given; The smoothness reward of the action Defined as:

[0059] in, and Weighting coefficients; action smoothness reward Calculated from the current state of motion, linear velocity The larger the angular velocity, the higher the positive reward. The smaller the value, the lower the negative reward / penalty.

[0060] Step 4: The latent space obstacle state prediction module (LOSP) performs a prediction task by comparing the differences between the observation data of the intermediate environment in the latent space before and after the previous frame, and feeds back the generated prediction gradient to the latent space obstacle state prediction module and updates the parameters of the latent space obstacle state prediction module to assist in training the dynamic perception ability of the actor network, and simultaneously updates the parameters of the actor network and the critic network. Inspired by the application of self-supervised learning methods in reinforcement learning representation learning, the LOSP module introduces a BYOL-style asymmetric contrast structure to enhance the ability of unmanned systems to capture environmental dynamics, thereby assisting in navigation tasks. Within this LOSP module framework, the system calculates the distance between raw observation data and semantically labeled observation data in the latent feature space. Through this mechanism, the encoder can simultaneously perform semantic parsing of the observation data and infer the evolution of the environmental state, thus providing unmanned system navigation with more accurate environmental understanding and dynamic prediction capabilities.

[0061] In this embodiment, the latent space obstacle state prediction module in step 4 performs the prediction task by comparing the differences between the observation data of the intermediate environment in the latent space before and after the previous frame, and feeds back the generated prediction gradient to the latent space obstacle state prediction module to update the parameters of the latent space obstacle state prediction module, so as to assist in training the dynamic perception capability of the actor network, specifically including: Step 41, Composition of the LOSP module: The latent space obstacle state prediction module adopts an asymmetric self-supervised contrastive learning architecture, such as... Figure 5 As shown, including the original encoder and its corresponding momentum encoder Original projection layer and its corresponding momentum projection layer Prediction layer and Transformer-based prediction decoder ; like Figure 6 As shown, the predictive decoder As a Transformer-based architecture, the predictive decoder receives latent state tags. and action markers The relative position encoding is embedded into the state and action tags, and then the tag sequence is passed through... The layered Transformer encoder processes the predictions that generate a sequence of future potential states.

[0062] Step 42, Training Data Sampling: Sampling the original observation sequence composed of intermediate environment representations from the experience replay buffer. Action sequence and labeled observation sequences ;in Each in the sequence Obtained from the DEP module (see) Figure 4 (b)); Step 43, Raw Observation Encoding: Using the raw encoder Original observation sequence Mapping to the latent space yields a sequence of latent states. ,Right now ; Step 44: Transformer-based latent state prediction: Predict the latent state sequence and action sequence Input to the predictive decoder Predicting the future potential state sequence to obtain the future Predicted state sequence of steps ; Step 45: Cascaded processing of prediction and projection layers: through the original projection layer and prediction layer The predicted state sequences are concatenated to obtain prediction results with semantic segmentation parsing information. ,Right now ; Step 46, Momentum Encoding and Projection Processing at the Target End: Using a momentum encoder Label the observation sequence Encoded as a momentum latent sequence ,Right now Then through the momentum projection layer Momentum latent sequence Generate target representation ,Right now Step 47, Momentum network parameter update rules: Momentum encoder parameters and momentum projection layer parameters Through momentum encoder weights and momentum projection layer weight The Exponential Moving Average (EMA) is used for updating, and the update formula is as follows:

[0063]

[0064] in, Indicates momentum encoder The parameters, It is the momentum coefficient. Indicates the momentum encoder weights; Indicates momentum projection layer The parameters, This represents the momentum projection layer weight.

[0065] Step 48, Auxiliary Prediction Loss: The goal of the LOSP module is to ensure the prediction results... As close as possible to its corresponding target representation Therefore, by calculating the cosine similarity between the prediction result and the target representation, an auxiliary loss function for the prediction task is constructed. Parameter updates for the latent space obstacle state prediction module: .

[0066] In this embodiment, step 4, which involves synchronously updating the parameters of the actor network and the critic network, specifically includes: Step 49: Initialize the prediction layer parameters of the latent space obstacle state prediction module. actor network parameters critic network parameters and its target network parameters Soft state-action function Entropy parameter in Batch size and experience replay buffer ; Step 410: Perform multiple training rounds, each training round including: (1) Initialize the current state, the pose of the unmanned system, and the target state from the environment: And generate the corresponding intermediate environment representation. ; (2) Execute multiple time steps within a single training round, each time step including: Sampling actions based on the current strategy: ; Perform actions, interact with the environment, and obtain rewards and the next state: ; Generate intermediate environment representations for the next time step state: ; Store the interaction data in the experience replay buffer: ; Sample batch data from the experience replay buffer: And update the critic network parameters, actor network parameters, prediction layer parameters, or target network parameters based on the sampled batch data; where: The update process for the critic network parameters includes: if the timing for updating the critic is met, then the time-difference loss function is minimized. Calculate the critic network loss And update the critic network parameters; The process of updating the actor network parameters includes: if the timing for updating the actor is met, then maximizing the soft state-action function. Calculate the loss of the actor network. And update the actor network parameters; The update process of the prediction layer parameters includes: if the time for updating LOSP is met, then from the empirical replay buffer... Mid-sample batch data: According to the auxiliary loss function Update the prediction layer parameters; The target network parameter update process includes: if the timing for updating the target network is met, then the target network parameter update is performed. This can be achieved through soft or hard updates, including copying or partially copying the critic network parameters to the target network.

[0067] Step 5: Deploy the trained actor network on the unmanned system, input the navigation task target point and the intermediate environment representation obtained by the dynamic environment perception module in the current dynamic environment into the trained actor network, and output the navigation control actions of the unmanned system. In this embodiment, step 5 specifically includes: Step 51: Deploy the trained actor network on the unmanned system; Step 52: Acquire sensor data and unmanned system motion state in the current dynamic environment, and fuse sensor data and unmanned system motion state into intermediate environment representation through dynamic environment perception module (at this time, intermediate environment representation is used as input data of actor network to obtain current action data of unmanned system). Step 53: Input the intermediate environment representation corresponding to the current dynamic environment and the navigation task target point into the trained actor network; the intermediate environment representation includes continuous Frame observation data The observation data for each time frame consists of the 3D point cloud at the current time. The motion state of unmanned systems and sub-target point location Composition; output is the navigation action command of the unmanned system; the navigation action command includes linear velocity. and angular velocity ,Right now .

[0068] In summary, the innovation of this invention lies in: 1. Intermediate bird's-eye view representing the construction and application of BEV-DEP Improvement: This method explicitly constructs a BEV-DEP intermediate representation in the system, fusing depth images, LiDAR observations, and the robot's own motion state into an intermediate feature representation in the form of a bird's-eye view.

[0069] Functional differences: BEV-DEP is used as input to the DRL strategy, rather than directly using raw sensor data or relying on third-party detection results.

[0070] Beneficial effects: Reduces reliance on external target detection modules and avoids errors introduced by detectors; improves the ability to distinguish between static and dynamic obstacles by filtering noise and retaining navigation-related spatial and motion information; intermediate representations help narrow the gap between simulation and reality, thereby promoting the transfer of simulation training strategies to the real environment.

[0071] 2. The BYOL-based Latent Space Obstacle State Prediction (LOSP) module serves as a self-supervised auxiliary task. Improvements: A contrastive self-supervised learning module (LOSP) based on BYOL is introduced in parallel within the DRL framework to implicitly predict the motion state of obstacles in the latent space.

[0072] Functional differences: The encoder simultaneously receives reinforcement learning signals from the navigation task and self-supervised prediction signals from LOSP, thereby gaining the ability to represent the motion state of obstacles in the latent representation.

[0073] Beneficial effects: By learning obstacle motion prediction, the encoder's sensitivity and robustness to dynamic obstacle motion characteristics are improved.

[0074] 3. Joint optimization training of perception and decision-making Improvements: This invention uses the Latent Space Obstacle State Prediction (LOSP) module as an auxiliary task to share the same feature encoder with the navigation strategy, and performs joint training under a deep reinforcement learning framework, so that the encoder can receive gradient feedback from both the navigation task and the prediction task.

[0075] Functional differences: Unlike traditional methods that train the perception network and decision network independently, this invention achieves collaborative updates of perception and decision through joint training, enabling the encoder to structurally possess both environmental perception and dynamic prediction functions.

[0076] Beneficial effects: This joint optimization mechanism enables the representation learned by the encoder to simultaneously serve current decision-making and future obstacle state prediction, improving the model's ability to distinguish between static and dynamic obstacles and the accuracy of predicting obstacle movement trends, thereby enhancing the navigation and obstacle avoidance performance of unmanned systems. Example 1: Implementation based on BEV-OSP 1. Unmanned System Verification Platform: Agilex Scout Mini unmanned ground vehicle, equipped with Nvidia Jetson Orin NX embedded computing unit, and sensor configuration includes Slamtec S2L 2D LiDAR and Intel RealSense D435i depth camera.

[0077] 2. Simulation Platform: The PEDSIM library is used in the Gazebo simulation platform to conduct 3D simulation environment experiments.

[0078] 3. Experimental setup: (1) Reward parameter settings: , , , , , .

[0079] (2) Motion parameter settings: The linear velocity range is limited to Angular velocity is limited to .

[0080] (3) Main simulation environment: Figure 7 The main simulation environment of this embodiment is shown. Figure 7 (a) The Lobby World scene is a baseline dynamic environment, approximately [size missing]. The maximum speed of pedestrians is ; Figure 7 (b) is an area within a Small City World scene, approximately [size missing]. Includes buildings, vegetation, and other static obstacles; the maximum speed for pedestrians is set at three levels: (aldult), (Children) and (Elderly people). Figure 7 (c) and Figure 7 (d) shows the corresponding evaluation waypoints.

[0081] (4) Navigation strategy training settings: This invention implements the algorithm based on the PyTorch framework, using the Adam optimizer, with an initial learning rate set to The frequency of the training data gradually decreases during training. The batch size for training is set to 128. The training environment is a lobby scene containing 35 dynamic pedestrians. In each training round, the starting and target positions of the unmanned system are randomly selected on the map.

[0082] 4. Comparison Algorithm Explanation: The comparison algorithms in this embodiment include SAC-A and SAC-B for ablation experiments, and baseline methods DWA, TEB, DRL-VO, and BEVNav for comparison experiments. Detailed explanations are as follows: (1) SAC-A: BEV-OSP method without the prediction module LOSP.

[0083] (2) SAC-B: In the BEV-DEP representation, only a single channel BEV-MaxHeight is used to convey environmental spatial information, and prediction is performed only on this channel.

[0084] (3) DWA: Dynamic Window Approach (DWA), a classic sampling-based local obstacle avoidance algorithm.

[0085] (4) TEB: Timed Elastic Band (TEB) is a local path planning method based on optimization.

[0086] (5) DRL-VO: This method uses pedestrian maps as intermediate features and incorporates velocity obstacles into the reward design.

[0087] (6) BEVNav: This method uses a self-supervised spatiotemporal contrastive learning method to learn the BEV representation.

[0088] 5. Performance Indicators: This embodiment uses four types of evaluation indicators commonly used in the navigation field: (1) Success Rate (SR): refers to the proportion of rounds in which navigation is completed without collision.

[0089] (2) Collision Rate (CR): refers to the proportion of navigation rounds in which a collision occurs.

[0090] (3) Path efficiency (Success weighted by Path Length, SPL): This measures the efficiency of the unmanned system's navigation path when the objective is successfully achieved. The calculation formula is as follows: ,in, For path length, This is the optimal trajectory length.

[0091] (4) Average Velocity: refers to the average linear velocity of the unmanned system in each step, which is used to reflect the smoothness and efficiency of the motion.

[0092] 6. Experimental Results and Analysis: (1) Comparison of lobby environment under different crowd densities: First, comparative experiments were conducted in a lobby scenario, with pedestrian densities of 15, 35, and 55 people respectively. As shown in Table 1, the BEV-OSP method achieved or outperformed the existing state-of-the-art (SOTA) baseline method in terms of success rate (SR), collision rate (CR), and path efficiency (SPL). In the three scenarios, compared to the best-performing SOTA method, the navigation success rate of the method of this invention was improved by 15.00%, 15.07%, and 12.86%, respectively.

[0093] (2) Contributions of the ablation experiment verification module: In a lobby scenario, an ablation experiment was designed to test the DEP and LOSP modules. As shown in Table 2, BEV-OSP achieved the best results in terms of SR, CR, and average velocity. Especially in a high-density scenario with 55 pedestrians, the success rate improved by 11.27%, significantly validating the synergistic enhancement effect of DEP perception and LOSP prediction capabilities.

[0094] (3) Cross-scenario generalization performance verification: To further verify the generalization ability of the algorithm, this embodiment was evaluated in a small city world scenario with 45 pedestrians. The results are shown in Table 3. The BEV-OSP method improves the success rate by 16.67% compared to DRL-VO and by 44.83% compared to TEB. It also reduces the collision rate by 42.86% and 57.89% compared to DRL-VO and BEVNav, respectively. It reaches or exceeds the state-of-the-art methods in terms of collision rate, path efficiency, and average speed. The results fully verify the adaptability and advantages of the BEV-OSP framework in complex scenarios.

[0095] Table 1: Comparative Evaluation Results under Different Pedestrian Densities

[0096] Table 2: Ablation Experiment Results of Prediction Task and Observation Coding Module

[0097] Table 3: Generalization performance results under novel complex environments

[0098] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for perception, obstacle avoidance and navigation of an unmanned system based on deep reinforcement learning, characterized in that, The method comprises the following steps: Step 1, constructing a BEV-based obstacle state prediction system, comprising a dynamic environment perception module, a latent space obstacle state prediction module and a deep reinforcement learning framework, wherein the deep reinforcement learning framework comprises an actor network and a critic network; Step 2, the unmanned system interacts with the environment to obtain sensor data and the motion state of the unmanned system, the dynamic environment perception module fuses the sensor data and the motion state of the unmanned system into intermediate environment representation, and inputs the intermediate environment representation and the navigation task target point into the actor network and the critic network, and inputs the intermediate environment representation into the latent space obstacle state prediction module; Step 3, the actor network processes the intermediate environment representation and the navigation task target point to output navigation action data, and the critic network evaluates the value of the navigation action data based on a multi-objective reward function, the intermediate environment representation and the navigation task target point; Step 4, the latent space obstacle state prediction module performs a prediction task by comparing the differences between the front and rear frame observation data of the intermediate environment representation in the latent space, and returns the generated prediction gradient to the latent space obstacle state prediction module and updates the parameters of the latent space obstacle state prediction module to assist in training the dynamic perception capability of the actor network, and synchronously updates the parameters of the actor network and the critic network; Step 5, deploying the trained actor network on the unmanned system, inputting the navigation task target point and the intermediate environment representation obtained in the current dynamic environment through the dynamic environment perception module into the trained actor network, and outputting the navigation control action of the unmanned system; Step 6, performing the navigation and obstacle avoidance task of the unmanned system based on the navigation control action. 2.The method of claim 1, wherein, The step 2 specifically comprises: Step 21, collecting sensor data, including depth images obtained by using a depth camera and environment point cloud data obtained by using a 2D laser radar; Step 22, projecting the depth images and the environment point cloud data to a unified three-dimensional point cloud coordinate system to obtain a three-dimensional point cloud, and simultaneously extracting the spatial attributes of the three-dimensional point cloud; Step 23, discretizing a target region of a preset range into a grid map with the unmanned system as the center; Step 24, projecting the three-dimensional point cloud in the vertical direction onto the grid map; for each grid cell, extracting the maximum height value in the three-dimensional point cloud as the spatial feature of the grid cell; Step 25, combining the motion state of the unmanned system, calculating and filling the relative velocity component of each grid cell in the unmanned system coordinate system to form a velocity feature; Step 26, assigning a default maximum height value to the spatial feature of the grid in the unobservable region, and setting the velocity feature to zero; the unobservable region includes a region beyond the maximum range or field of view of the sensor and an occluded region; Step 27, connecting the spatial feature and the velocity feature to form the intermediate environment representation, and performing normalization processing on the intermediate environment representation; Step 28, input the normalized intermediate environment representation and the navigation task goal point into the actor network and the critic network, and input the normalized intermediate environment representation into the latent space obstacle state prediction module. 3.The method of claim 2, wherein, The step 25 specifically comprises: Step 251, calculate the relative speed in the unmanned system coordinate system at each grid center by rotation matrix transformation: in, This represents the relative velocity of the navigation mission target point in the coordinate system of the unmanned system. express of x Axial components, express of y Axial components; This represents the relative velocity of the navigation mission target point in the world coordinate system relative to the coordinate system of the unmanned system. express of x Axial components, express of y Axial components; It is a rotation matrix. Yaw angle; Step 252, filling and into two velocity features of the grid mesh, respectively; Step 253, generate two types of BEV speed maps: one is unlabeled speed features directly generated based on current observations, which are used as the original input of the actor network; the other is labeled speed features generated in combination with dynamic obstacle pose data from the Gazebo simulator, which are used for global information sequences for obstacle prediction. 4.The method of claim 3, wherein, The labeled speed feature generation process specifically comprises: Refine the unlabeled speed features using dynamic obstacle pose data to generate new speed features; The data corresponding to the grid position is replaced by the relative speed of the pedestrian in the coordinate system of the unmanned system, and for static obstacles or free space, let ; For dynamic pedestrians, let to obtain the velocity component in the unmanned system coordinate system; wherein, represents the unmanned system's own speed vector, represents the speed of the pedestrian; Label the processed new speed features to obtain labeled speed features, while encoding the spatial position and speed information of the dynamic obstacle.

5. The method of claim 2, wherein, The step 27 specifically comprises: Step 271, connect the spatial features with the unlabeled speed features and the labeled speed features respectively to form two three-channel intermediate environment representations; Step 272, using a maximum absolute value scaling procedure, normalize the three channels of observed data to the range [0, 1] using the formula: Step 272, using a maximum absolute value scaling procedure, normalize the three channels of observed data to the range [0, 1] using the formula: wherein denotes moment channel the observation data, denotes the minimum value of the channel observation data, denotes the maximum value of the channel observation data.

6. The method of claim 1, wherein, The step 3 specifically comprises: Step 31, the deep reinforcement learning framework adopts a Soft Actor-Critic algorithm framework, including an actor network and a critic network, the actor network includes a first encoder and an action output head, which is used to process the intermediate environment representation and the navigation task goal point to perceive the environment and output navigation action data; the critic network includes a second encoder and two Q-value networks, which are used to process the intermediate environment representation and the navigation task goal point and evaluate the quality of the navigation action data given under the current observation; Step 32, the actor network receives an intermediate environment representation and a navigation task goal point, the intermediate environment representation includes continuous observation data of each time frame , the observation data of each time frame is composed of a three-dimensional point cloud of the current time , the motion state of the unmanned system and the sub-goal point position ; the output is the navigation action data of the unmanned system; the navigation action data includes linear velocity and angular velocity , that is ; Step 33, the critic network evaluates and optimizes the navigation action data based on a multi-objective reward function, an intermediate environment representation, and a navigation task goal point, the multi-objective reward function includes at least a goal arrival reward, a collision penalty reward, a heuristic distance reward, and an action smoothness reward.

7. The method of claim 6, wherein the method further comprises: The step 33 specifically comprises: The multi-objective reward function is represented as: wherein, is a target arrival reward, i.e., a positive reward given to the unmanned system when it successfully arrives at the navigation task target point; is a collision penalty reward, i.e., a negative reward given to the unmanned system when a collision occurs; is a heuristic distance reward; is an action smoothness reward; The target arrival reward is defined as: wherein, represents the distance between the unmanned system and the navigation task target point at the moment, represents a preset navigation task target point range threshold value, when the unmanned system is considered to have successfully arrived at the navigation task target point; represents the specific value of the reward obtained after the unmanned system successfully arrives at the navigation task target point. The collision penalty reward is defined as: wherein, represents a collision occurs, a specific value of the penalty when a collision occurs; The heuristic distance reward is defined as: wherein, is a weight coefficient, denotes the distance between the moment of the unmanned system and the navigation task target point; denotes the distance change between continuous frames, a heuristic distance reward is based on the distance change between continuous frames is adjusted, when the unmanned system approaches the navigation task target point, that is, a positive reward is obtained, when the unmanned system is far away from the navigation task target point, that is, a negative reward is given; The action smoothness reward is defined as: wherein, and are weight coefficients; action smoothness reward is calculated from the current motion state, linear velocity The higher the positive reward is obtained, angular velocity The lower the negative reward penalty is lower. 8.The method of claim 1, wherein, The latent space obstacle state prediction module in the step 4 performs a prediction task by comparing the differences between the front and rear frame observation data of the intermediate environment representation in the latent space, and returns the generated prediction gradient to the latent space obstacle state prediction module and updates the parameters of the latent space obstacle state prediction module to assist in training the dynamic perception ability of the actor network, specifically comprising: Step 41, the latent space obstacle state prediction module adopts an asymmetric self-supervised contrastive learning architecture, including an original encoder and its corresponding momentum encoder, an original projection layer and its corresponding momentum projection layer, a prediction layer, and a Transformer-based prediction decoder; Step 42, sample the original observation sequence, the action sequence, and the labeled observation sequence composed of the intermediate environment representation from the experience replay buffer; Step 43, map the original observation sequence into the latent space using the original encoder to obtain the latent state sequence; Step 44, use the momentum encoder to map the original observation sequence into the latent space to obtain the momentum state sequence; Step 44, input the potential state sequence and action sequence into the prediction decoder to predict the future potential state sequence to obtain a predicted state sequence; Step 45, process the predicted state sequence through the original projection layer and the prediction layer to obtain a prediction result with semantic segmentation analysis information; Step 46, encode the labeled observation sequence into a momentum potential sequence through the momentum encoder, and then generate a target representation through the momentum projection layer; Step 47, the parameters of the momentum encoder and the parameters of the momentum projection layer are updated through the exponential moving average of the momentum encoder weight and the momentum projection layer weight, and the update formula is as follows: wherein, denotes the parameters of the momentum encoder , is the momentum coefficient, denotes the momentum encoder weights; denotes the parameters of the momentum projection layer , denotes the momentum projection layer weights; Step 48, construct an auxiliary loss function for the auxiliary prediction task by calculating the cosine similarity between the prediction result and the target representation, and use it for parameter updating of the potential space obstacle state prediction module. 9.The method of claim 1, wherein, The step 4 synchronously updates the parameters of the actor network and the critic network, specifically including: Step 49, initialize the prediction layer parameters of the potential space obstacle state prediction module, the actor network parameters, the critic network parameters, the target network parameters, the entropy parameters in the Soft State-Action function, the batch size, and the experience replay buffer; Step 410, perform multiple training rounds, each training round including: (1) initialize the current state, the unmanned system pose, and the target state from the environment, and generate the corresponding intermediate environment representation; (2) execute multiple time steps within a single training round, each time step including: sample an action according to the current policy; execute the action, interact with the environment, and obtain the reward and the next state; generate an intermediate environment representation for the next state; store the interaction data in the experience replay buffer; sample a batch of data from the experience replay buffer, and update the critic network parameters, the actor network parameters, the prediction layer parameters, or the target network parameters based on the sampled batch of data; wherein: the update process of the critic network parameters includes: calculating the critic network loss using the minimum time difference loss function, and updating the critic network parameters; the update process of the actor network parameters includes: calculating the actor network loss by maximizing the soft state-action function, and updating the actor network parameters; the update process of the prediction layer parameters includes: sampling a batch of data from the experience replay buffer, and updating the prediction layer parameters according to the auxiliary loss function; the update process of the target network parameters includes: achieving through soft update or hard update, including copying or partially copying the critic network parameters to the target network. 10.The method of claim 1, wherein, The step 5 has the following steps: Step 51, deploy the trained actor network on the unmanned system; Step 52, obtain sensor data and unmanned system motion state in the current dynamic environment, and fuse the sensor data and unmanned system motion state into an intermediate environment representation through the dynamic environment perception module; Step 53, input the intermediate environment representation corresponding to the current dynamic environment and the navigation task target point into the trained actor network; the intermediate environment representation includes continuous observation data of each frame , the observation data of each frame at each time point is composed of the three-dimensional point cloud , the motion state of the unmanned system and the sub-target point position ; the output is the navigation action instruction of the unmanned system, and the navigation action instruction includes linear velocity and angular velocity.

Images