An unmanned equipment path planning method and system based on hierarchical multi-expert reinforcement learning

Abstract

This invention discloses a path planning method and system for unmanned equipment based on hierarchical multi-expert reinforcement learning, belonging to the field of path planning technology. The method includes: constructing a path planning model to be trained based on an upper-layer deep reinforcement learner and a lower-layer velocity fuzzy logic controller; preprocessing and extracting features from the training environment perception data set of the unmanned equipment to determine the training environment perception features; using the training environment perception data set to determine the training fuzzy input variables; training the path planning model to be trained based on the training environment perception features, the current linear angular velocity of the training unmanned equipment, and the training fuzzy input variables to determine the trained path planning model; and performing path planning on the received environment perception data set of the unmanned equipment based on data preprocessing, feature extraction, fuzzy input variable determination, and the trained path planning model. The method leverages the expert prior knowledge provided by the fuzzy logic controller to improve model training efficiency.

Description

Technical Field

[0001] This invention relates to the field of path planning technology, and in particular to a path planning method and system for unmanned equipment based on hierarchical multi-expert reinforcement learning. Background Technology

[0002] With the development of artificial intelligence technology, autonomous robots and other unmanned equipment are widely used in many fields such as service, logistics, medical care and industry. Among them, path planning is the key part of their autonomous navigation in complex dynamic environments.

[0003] Traditional path planning is typically based on algorithms such as A*, Dijkstra's algorithm, and Rapid Expanding Random Tree (RRT), often relying on a pre-built global map. This approach is significantly inadequate when dealing with unknown environments and dynamic obstacles. With the rapid development of deep learning and reinforcement learning technologies, map-free end-to-end path planning methods based on deep reinforcement learning have gradually become a research hotspot. These methods approximate the path planning problem as a Markov decision process, utilizing deep neural networks to extract features from environmentally perceived data and make autonomous decisions. However, they suffer from low training efficiency. Summary of the Invention

[0004] This invention provides a path planning method and system for unmanned equipment based on hierarchical multi-expert reinforcement learning, which solves the technical problem of low training efficiency in existing path planning methods based on deep reinforcement learning.

[0005] The first aspect of this invention provides a path planning method for unmanned equipment based on hierarchical multi-expert reinforcement learning, comprising:

[0006] Construct a trainable path planning model based on an upper-layer deep reinforcement learner and a lower-layer velocity fuzzy logic controller;

[0007] The training environment perception data set of unmanned equipment is preprocessed and features are extracted to determine the training environment perception features.

[0008] The training environment-aware data set is used to determine the training fuzzy input variables for the lower-level velocity fuzzy logic controller;

[0009] Based on the training environment perception features, the current linear angular velocity of the unmanned equipment, and the training fuzzy input variables, the path planning model to be trained is input. The current state unmanned state linear angular velocity of the unmanned equipment is output through the fusion of upper and lower layer decisions. The reward function value is determined based on the current state unmanned state linear angular velocity to update the upper layer deep reinforcement learner until the training stopping condition is met, and the trained path planning model is determined.

[0010] For the received environmental perception data set of unmanned equipment, path planning is performed based on data preprocessing, feature extraction, fuzzy input variable determination, and a trained path planning model.

[0011] Furthermore, the lower-level velocity fuzzy logic controller includes an adaptive escape navigation fuzzy logic controller, an obstacle avoidance fuzzy logic controller, and a wall-hugging motion fuzzy logic controller;

[0012] The adaptive escape navigation fuzzy logic controller takes the relative target point angular distance of the unmanned equipment with respect to the path target point as input and the escape fuzzy linear angular velocity as output.

[0013] The obstacle avoidance fuzzy logic controller takes the relative obstacle angle distance between the unmanned equipment and the nearest obstacle as input and the obstacle avoidance fuzzy linear angular velocity as output.

[0014] The wall-attaching motion fuzzy logic controller takes the yaw angle of the unmanned equipment relative to the wall's parallel direction as input and the wall-attaching fuzzy linear angular velocity as output.

[0015] Further, the step of inputting the path planning model to be trained based on the training environment perception features, the current linear angular velocity of the unmanned equipment, and the training fuzzy input variables, outputting the current state unmanned linear angular velocity of the unmanned equipment through upper and lower layer decision fusion, and determining the reward function value based on the current state unmanned linear angular velocity to update the upper layer deep reinforcement learner, until the training stopping condition is met, to determine the trained path planning model, includes:

[0016] The action network of the upper-layer deep reinforcement learning machine takes the training environment perception features, the training relative target point angle distance, the training relative obstacle angle distance and the current line angular velocity of the unmanned equipment as inputs, and outputs the fuzzy action weights corresponding to the lower-layer velocity fuzzy logic controller.

[0017] The lower-level velocity fuzzy logic controller outputs the corresponding fuzzy linear angular velocity based on the training relative target point angular distance, the training relative obstacle angular distance, and the training relative wall angle.

[0018] The fuzzy linear angular velocities and fuzzy action weights are used to perform weighted fusion, and the current state unmanned linear angular velocity is output.

[0019] The Q-network of the upper-layer deep reinforcement learner takes the training environment perception features and the current state unmanned state line angular velocity as input, and outputs the action value function.

[0020] The V network, which is an upper-layer deep reinforcement learner, takes the training environment-aware features as input and outputs a state value function.

[0021] The reward function value is determined based on the current state unmanned state line angular velocity, and the Q network, V network and action network are updated in combination with the action value function and the state value function until the training stopping condition is met. The trained upper-layer deep reinforcement learner is determined, and the trained path planning model is output.

[0022] Furthermore, the Q-network of the upper-layer deep reinforcement learning machine includes a first feature extraction branch, a second feature extraction branch, a feature fusion module, and a fully connected neural network; the Q-network of the upper-layer deep reinforcement learning machine takes the training environment-aware features and the current state unmanned state line angular velocity as input, and outputs an action value function, including:

[0023] The training environment perception features are extracted by the first feature extraction branch of the Q network of the upper-layer deep reinforcement learner to determine the deep features of the environment.

[0024] The motion state features are determined by extracting the angular velocity of the current unmanned state line through the second feature extraction branch of the Q network of the upper deep reinforcement learner.

[0025] After fusing the environmental depth features and motion state features through the feature fusion module of the Q-network of the upper-layer deep reinforcement learner, the feature is input into the fully connected neural network for feature processing and outputs the action value function.

[0026] Furthermore, the process of determining the reward function value includes:

[0027] ;

[0028] In the formula, Based on the base reward function value, The distance of the unmanned equipment relative to the target point on the path before executing the action. The distance of the unmanned equipment relative to the target point on the path after performing the action. For the maximum linear velocity, The duration of adjacent time steps. The angle of the unmanned equipment relative to the target point on the path before performing the action. The angle of the unmanned equipment relative to the target point on the path after the action is performed. For the maximum angular velocity, The current linear velocity. The angular velocity is the current state. As a penalty for stagnation, This is the adjustment coefficient.

[0029] Furthermore, the process of determining the path target point includes:

[0030] Analyze the environmental perception data set to determine if obstacles exist;

[0031] If it does not exist, the preset target point will be used as the path target point;

[0032] If present, potential target areas are marked based on the environmental perception data set of the obstacle's vicinity.

[0033] The location of the potential target area is determined based on environmental perception data, the position of the unmanned equipment, and the yaw angle of the unmanned equipment, and path target points are selected from the potential target area.

[0034] Furthermore, the process of determining the reward function value also includes:

[0035] Construct a digital twin that matches the physical parameters of the unmanned equipment;

[0036] Calculate the synchronization deviation between the predicted trajectory state of the digital twin at the current time step in the virtual environment and the actual trajectory state of the unmanned equipment;

[0037] Determine the obstacle density potential field value of the digital twin in the virtual environment for future multiple time steps;

[0038] The base reward function value is corrected based on the synchronization deviation and the obstacle density potential field value to determine the corrected reward function value.

[0039] The process for determining the value of the modified reward function includes:

[0040] ;

[0041] In the formula, This is the corrected final reward value. Based on the base reward function value, For twin confidence weights, The sensitivity coefficient for the Gaussian kernel width. To guide weighting in a forward-looking manner, As a time discount factor, For time step index, For the deduction step index, For the total number of simulation steps, For the first The obstacle density potential field value sensed by the digital twin during the step. It is a natural constant. This refers to the synchronization deviation.

[0042] The second aspect of this invention provides an unmanned equipment path planning system based on hierarchical multi-expert reinforcement learning, comprising:

[0043] The model building module is used to build a trainable path planning model based on an upper-layer deep reinforcement learner and a lower-layer velocity fuzzy logic controller.

[0044] The feature extraction module is used to preprocess and extract features from the training environment perception data set of unmanned equipment to determine the training environment perception features.

[0045] The variable determination module is used to determine the training fuzzy input variables of the lower-level velocity fuzzy logic controller using the training environment-aware data set;

[0046] The model training module is used to input the path planning model to be trained based on the training environment perception features, the current line angular velocity of the training unmanned equipment, and the training fuzzy input variables. It outputs the current state line angular velocity of the unmanned equipment through the fusion of upper and lower layer decisions, and determines the reward function value based on the current state line angular velocity of the unmanned equipment to update the upper layer deep reinforcement learner until the training stopping condition is met, and then determines the trained path planning model.

[0047] The path planning module is used to plan the path of the received environmental perception data set of unmanned equipment based on data preprocessing, feature extraction, fuzzy input variable determination, and a trained path planning model.

[0048] A computer device provided in a third aspect of the present invention includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the path planning method for unmanned equipment based on hierarchical multi-expert reinforcement learning as described in any of the preceding claims.

[0049] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the unmanned equipment path planning method based on hierarchical multi-expert reinforcement learning as described in any of the preceding claims.

[0050] As can be seen from the above technical solutions, the present invention has the following advantages:

[0051] The above-mentioned solution of the present invention provides a path planning method for unmanned equipment based on hierarchical multi-expert reinforcement learning, comprising: constructing a path planning model to be trained based on an upper-layer deep reinforcement learner and a lower-layer velocity fuzzy logic controller; preprocessing and extracting features from the training environment perception data set of the unmanned equipment to determine the training environment perception features; using the training environment perception data set to determine the training fuzzy input variables of the lower-layer velocity fuzzy logic controller; inputting the training environment perception features, the current linear angular velocity of the training unmanned equipment, and the training fuzzy input variables into the path planning model to be trained, outputting the current state unmanned state line angular velocity of the unmanned equipment through upper and lower layer decision fusion, and updating the upper-layer deep reinforcement learner based on the reward function value determined according to the current state unmanned state line angular velocity, until the training stopping condition is met to determine the trained path planning model; and performing path planning on the received unmanned equipment's environment perception data set to be processed based on data preprocessing, feature extraction, fuzzy input variable determination, and the trained path planning model. Based on the above scheme, a path planning model is constructed by combining a fuzzy logic controller suitable for path planning tasks with a deep reinforcement learner. With the help of the expert prior knowledge provided by the fuzzy logic controller, the exploration efficiency in the early stage of model training is improved, thereby making it more efficient in training. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 The flowchart illustrates the steps of a path planning method for unmanned equipment based on hierarchical multi-expert reinforcement learning, as provided in Embodiment 1 of the present invention.

[0054] Figure 2 This is a membership function diagram of the adaptive obstacle avoidance navigation fuzzy logic controller provided in Embodiment 1 of the present invention;

[0055] Figure 3 This is a membership function diagram of the obstacle avoidance fuzzy logic controller provided in Embodiment 1 of the present invention;

[0056] Figure 4 This is a membership function diagram of the wall-attached motion fuzzy logic controller provided in Embodiment 1 of the present invention;

[0057] Figure 5 This is a diagram of a hierarchical multi-expert reinforcement learning framework provided in Embodiment 1 of the present invention;

[0058] Figure 6This is a structural block diagram of an unmanned equipment path planning system based on hierarchical multi-expert reinforcement learning, provided in Embodiment 2 of the present invention. Detailed Implementation

[0059] This invention provides a path planning method and system for unmanned equipment based on hierarchical multi-expert reinforcement learning, which solves the technical problem of low training efficiency in existing path planning methods based on deep reinforcement learning.

[0060] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0061] Please see Figure 1 The present invention provides a path planning method for unmanned equipment based on hierarchical multi-expert reinforcement learning, comprising:

[0062] Step 101: Construct a path planning model to be trained based on an upper-layer deep reinforcement learner and a lower-layer velocity fuzzy logic controller.

[0063] It should be noted that, based on path planning using deep reinforcement learning, this embodiment introduces a combination of various fuzzy logic controllers suitable for path planning tasks to construct a path planning model in order to improve the exploration efficiency in the early stage of model training, thereby making it more efficient and improving the limitations such as insensitivity to initial values ​​of the model.

[0064] Step 102: Perform data preprocessing and feature extraction on the training environment perception data set of unmanned equipment to determine the training environment perception features.

[0065] Unmanned equipment refers to unmanned devices with autonomous environmental perception and path planning capabilities, such as unmanned vehicles and drones.

[0066] Environmental perception data set refers to the combination of environmental state information surrounding the unmanned equipment, i.e., environmental perception data. Environmental perception features refer to the structured feature vectors obtained after feature extraction from the environmental perception data. Training is used to indicate that the system is in the training phase.

[0067] It should be noted that the 360-dimensional environmental state information around the unmanned equipment collected by observation equipment such as radar forms the training environment perception data set for training. Then, data preprocessing is used to enhance data quality to avoid invalid iterations. Next, the preprocessed data is used to extract features through a preset feature extractor to extract effective semantic features, which can reduce dimensionality, reduce computational overhead, and reduce interference from invalid information.

[0068] In one specific implementation of this embodiment, data preprocessing includes maximum value normalization and cross-copying.

[0069] It should be noted that data preprocessing may include data cleaning, data augmentation, and data repair steps such as maximum value normalization, cross-copying, and noise reduction; this embodiment does not limit this. In one implementation, for example, taking a radar array as the environmental perception data group, the measured values ​​(inf) exceeding the radar's maximum detection range are set as its maximum measured value and normalized to the maximum value. Subsequently, the radar array index is... Data within the range is copied to the following array table: At the same time, the radar array index is set to Data within the range is copied to the array index. In its position.

[0070] In one specific embodiment of this example, the feature extractor includes a multi-layer convolutional network.

[0071] It should be noted that a suitable feature extractor can be selected for feature extraction according to the data extraction needs, and this embodiment does not impose any restrictions on this. For example, a multi-layer convolutional network such as CNN can be used. Specifically, the preprocessed 380-dimensional radar data is input into the feature extractor to construct a one-dimensional radar data vector. Local features are extracted through two layers of 1×5 convolution operations, and the number of channels is expanded to 16. High-level features are further extracted through two layers of 1×3 convolution, and the number of channels is expanded to 32. The high-level features are downsampled through max pooling to obtain a 32-channel feature map of size 1×24, i.e., pooled features. The pooled features are flattened to obtain a flattened feature of length 768. The flattened feature is mapped to a 66-dimensional feature space through a fully connected layer as the output of the feature extractor. The overall network structure extracts high-level features from the input data step by step through multi-layer convolution operations, and completes the expression of target features through fully connected layers.

[0072] Step 103: Use the training environment-aware data set to determine the training fuzzy input variables for the lower-level velocity fuzzy logic controller.

[0073] It should be noted that when designing the lower-level speed fuzzy logic controller, a fuzzification method (membership function) is designed to divide each variable into multiple fuzzy subsets. These subsets are then further divided into multiple fuzzy subsets. A rule base is designed and fuzzy rules are applied. The membership degree of the output fuzzy set generated by each triggered rule is calculated. All these output sets are combined into an output fuzzy set. The output fuzzy set is then converted into actual control values ​​using methods such as the centroid method and the maximum membership method. These values ​​serve as the action output of the fuzzy logic controller. When applying the lower-level speed fuzzy logic controller, the designed fuzzy input variables need to be determined based on the environmental perception data set.

[0074] In a more specific embodiment of this example, the lower-level velocity fuzzy logic controller includes an adaptive escape navigation fuzzy logic controller, an obstacle avoidance fuzzy logic controller, and a wall-hugging motion fuzzy logic controller.

[0075] S11, the adaptive escape navigation fuzzy logic controller takes the relative target point angular distance of the unmanned equipment to the target point on the path as input and the escape fuzzy linear angular velocity as output.

[0076] It should be noted that, in order to achieve adaptive obstacle avoidance navigation, two possible scenarios are considered: the absence of potential "trap environments" (such as zigzag obstacles) and the presence of potential "traps" in the environment. Strategies are designed accordingly. In the first scenario, the unmanned equipment is encouraged to move directly toward the target point. In the second scenario, the unmanned equipment is encouraged to search for alternative additional target points in the "trap" environment and reach the target point by bypassing the "trap" environment through segmented navigation.

[0077] The adaptive fuzzy logic controller for escape navigation in this implementation design considers the relative target point distance of the unmanned equipment with respect to the current stage's path target point. and relative target point angle For its membership function, please refer to [reference needed]. Figure 2 As shown, fuzzy input variables Divided into three fuzzy subsets , Indicates short distance, Indicates a medium distance. To represent long distances, fuzzy input variables will be used. Divided into five fuzzy subsets , Indicates the large left corner. Indicates the left corner. Indicates the front angle. Indicates the right corner. Indicates the large right corner, and will fuzzy output the linear velocity of the unmanned equipment. Divided into three fuzzy subsets , Indicates low speed. Indicates medium speed. To indicate high speed, the angular velocity of the unmanned equipment will be displayed as a fuzzy output variable. Divided into five fuzzy subsets , Indicates a sharp right turn. Indicates a slight right turn. Indicates going straight. Indicates a slight left turn. This indicates a sharp left turn, as shown in Table 1:

[0078] Table 1. Fuzzy Logic Rules for Adaptive Escape Navigation Strategy

[0079]

[0080] In one specific embodiment of this example, the process of determining the path target point includes:

[0081] Analyze the environmental perception data set to determine if obstacles exist;

[0082] If it does not exist, the preset target point will be used as the path target point;

[0083] If present, potential target areas are marked based on the environmental perception data set of the obstacle's vicinity.

[0084] The location of the potential target area is determined based on environmental perception data, the position of the unmanned equipment, and the yaw angle of the unmanned equipment, and path target points are selected from the potential target area.

[0085] Understandably, taking a radar array as an example, the radar data is traversed and the difference between its measurements is calculated. If the absolute value of the difference exceeds a certain set threshold and the fluctuation is small in subsequent states (for example, by judging whether the absolute value of the difference between two adjacent radar measurements exceeds the threshold, the radar data within a 20° range before and after it is traversed according to this step; if it does not exceed the threshold, it is considered to have small fluctuation), it can be marked as a potential target area. Using LiDAR data (range and angle) and the yaw angle and position of the unmanned equipment, the approximate location of the potential target area can be calculated. For example, the difference between the radar's current position and the world coordinate system can be calculated using the equipment's yaw angle. Based on the potential target area found in the previous step, the angle between the radar and the equipment's yaw angle can be obtained, and the angle of the potential target area relative to the unmanned equipment (positive direction, not yaw direction) in the world coordinate system can be calculated. Finally, the approximate location of the potential target area is calculated using trigonometric functions based on the coordinates of the unmanned equipment. Finally, a point in this area is randomly selected as an additional target point.

[0086] S12, the obstacle avoidance fuzzy logic controller takes the relative obstacle angle distance between the unmanned equipment and the nearest obstacle as input, and the obstacle avoidance fuzzy linear angular velocity as output.

[0087] It should be noted that the obstacle avoidance fuzzy logic controller in this implementation design considers the relative obstacle distance between the unmanned equipment and the nearest obstacle. and relative obstacle angle For its membership function, please refer to [reference needed]. Figure 3 As shown, fuzzy input variables Divided into three fuzzy subsets , fuzzy input variables Divided into five fuzzy subsets The linear velocity of the unmanned equipment will be fuzzy output variable. Divided into three fuzzy subsets The angular velocity of the unmanned equipment will be fuzzy output variable. Divided into five fuzzy subsets As shown in Table 2:

[0088] Table 2. Fuzzy Logic Rules for Obstacle Avoidance Strategies

[0089]

[0090] S13, the wall-attached motion fuzzy logic controller takes the yaw angle of the unmanned equipment relative to the wall in the parallel direction of the wall as input and the wall-attached fuzzy linear angular velocity as output.

[0091] It should be noted that, for the wall-hugging motion strategy, a fuzzy logic controller for the wall-hugging motion strategy is designed and implemented, taking into account the angle between the yaw angle of the unmanned equipment and the parallel direction of the wall relative to the wall surface. For its membership function, please refer to [reference needed]. Figure 4 As shown, fuzzy input variables Divided into five fuzzy subsets The linear velocity of the unmanned equipment will be fuzzy output variable. Divided into three fuzzy subsets The angular velocity of the unmanned equipment will be fuzzy output variable. Divided into five fuzzy subsets As shown in Table 3:

[0092] Table 3. Fuzzy Logic Rules for Obstacle Avoidance Strategies

[0093]

[0094] Understandably, based on the environmental perception data set, the relative target point angular distance of the unmanned equipment to the target point on the path, the relative obstacle angular distance of the unmanned equipment to the nearest obstacle, and the angle between the unmanned equipment's yaw angle and the wall's parallel direction can be determined. As for determining the wall's parallel direction, taking the radar array as an example, the shortest value (angle) of the radar data can be taken as the parallel reference. The target direction (parallel to the wall) is this shortest angle ±90°. The direction that requires the smallest adjustment degree is selected as the target direction. This can be done by adopting the corresponding data determination method according to different environmental perception data.

[0095] Step 104: Based on the training environment perception features, the current line angular velocity of the unmanned equipment, and the training fuzzy input variables, input the path planning model to be trained, output the current state line angular velocity of the unmanned equipment through the fusion of upper and lower layer decisions, and determine the reward function value based on the current state line angular velocity of the unmanned equipment to update the upper layer deep reinforcement learner until the training stopping condition is met, and then determine the trained path planning model.

[0096] In one specific embodiment of this example, step 103 includes the following sub-steps:

[0097] S21. The action network of the upper-layer deep reinforcement learning machine takes the environmental perception features, the angle distance relative to the target point, the angle distance relative to the obstacle, and the current linear angular velocity of the unmanned equipment as inputs, and outputs the fuzzy action weights corresponding to the lower-layer velocity fuzzy logic controller.

[0098] S22. The lower-level velocity fuzzy logic controller outputs the corresponding fuzzy linear angular velocity based on the training relative target point angular distance, the training relative obstacle angular distance, and the training relative wall angle.

[0099] S23. Weighted fusion is performed using the fuzzy linear angular velocities and the weights of the fuzzy actions to output the linear angular velocity of the current unmanned state.

[0100] S24. The Q-network of the upper-layer deep reinforcement learner is trained with environmental perception features and the current state unmanned state line angular velocity as input, and outputs the action value function.

[0101] S25. The V network of the upper-layer deep reinforcement learner takes the environment-aware features as input and outputs the state value function.

[0102] S26. Determine the reward function value based on the current state unmanned state line angular velocity, and update the Q network, V network and action network in combination with the action value function and state value function until the training stopping condition is met, determine the trained upper-layer deep reinforcement learner, and output the trained path planning model.

[0103] In a more specific embodiment of this example, the Q-network of the upper-layer deep reinforcement learning machine includes a first feature extraction branch, a second feature extraction branch, a feature fusion module, and a fully connected neural network; the Q-network of the upper-layer deep reinforcement learning machine takes the training environment-aware features and the current state unmanned state line angular velocity as input, and outputs an action value function, including:

[0104] The first feature extraction branch of the Q-network of the upper-layer deep reinforcement learner is used to extract features of the training environment perception features to determine the deep features of the environment.

[0105] The second feature extraction branch of the Q-network of the upper deep reinforcement learner is used to extract the angular velocity of the current unmanned state line to determine the motion state features.

[0106] After fusing environmental depth features and motion state features through the feature fusion module of the Q-network of the upper-layer deep reinforcement learner, the feature is input into the fully connected neural network for feature processing and outputs the action value function.

[0107] It should be noted that, for reference Figure 5 The training framework shown consists of: the state space of the upper-layer deep reinforcement learner includes environmental perception features, relative target point angular distance, relative obstacle angular distance, and the current linear angular velocity of the unmanned equipment; the action space of the upper-layer deep reinforcement learner includes fuzzy action weights, based on the three fuzzy logic controllers included in the lower-layer velocity fuzzy logic controller; and the action space contains a three-bit One-Hot code. ;

[0108] The action network, implemented through an upper-layer deep reinforcement learning machine, takes as input environmental perception features, relative target point angular distance, relative obstacle angular distance, and the current linear angular velocity of the unmanned equipment. It outputs fuzzy action weights corresponding to the lower-layer velocity fuzzy logic controller, representing the action weights assigned to each fuzzy logic controller (acting as a bottom-layer expert) in the current environment. In one specific implementation, the action network process includes: concatenating environmental perception features, relative target point angular distance, relative obstacle angular distance, and the current linear angular velocity of the unmanned equipment to obtain joint features; abstracting and extracting the joint features using a multilayer perceptron; normalizing them using a Softmax nonlinear activation function; and outputting fuzzy action weights.

[0109] The lower-level fuzzy logic controller outputs corresponding fuzzy linear angular velocities based on the training relative target point angular distance, training relative obstacle angular distance, and training relative wall angle. These velocities are then weighted and fused with the fuzzy action weights output by the action network. The final output of the current state (unmanned) linear angular velocity is obtained through a linear combination of different actions.

[0110] ;

[0111] In the formula, For unmanned equipment line speed, Equip unmanned vehicles with angular velocity. To escape the fuzzy linear angular velocity, To avoid the obstacle fuzzy line angular velocity, we have: The linear and angular velocities of the fuzzy logic controller attached to the wall represent the recommended linear and angular velocities for different fuzzy logic controllers.

[0112] By reconstructing the Q-network and V-network separately for path planning of unmanned equipment based on the SAC (Soft Actor-Critic, SAC) algorithm, feature processing is performed to output action value function and state value function. In a preferred implementation, the Q-network of the upper-layer deep reinforcement learning machine includes a first feature extraction branch, a second feature extraction branch, a feature fusion module, and a fully connected neural network. The first feature extraction branch extracts features from environmental perception features to determine environmental depth features, while the second feature extraction branch extracts features from the current unmanned state line angular velocity to determine motion state features. The features extracted by the two branches are integrated (e.g., spliced) in the fusion module, and the action value function Q is finally output through the fully connected neural network. The branch structure effectively combines environmental features and motion state information, improving feature representation ability and modeling effect for complex scenes. It is understood that in one feasible implementation, the first feature extraction branch, the second feature extraction branch, and the V-network include a fully connected neural network.

[0113] After an agent interacts with the environment, the interaction information and the single-step reward obtained according to the reward function are encapsulated into a quadruple {current environment state, action, reward, next environment state} and stored in the experience replay pool. When the number of interactions reaches the minimum number of training iterations and the data in the priority experience replay pool reaches the minimum threshold, the agent continues to interact with the environment. The experience replay pool is sampled in batches of size according to priority. The network parameters of the Q network, V network and action network are updated using the SAC algorithm and Adam optimizer. The reinforcement learning training framework is used to maximize the cumulative reward as the optimization objective until the model converges or reaches the maximum number of iterations, etc. The specific settings can be combined with existing technologies as needed to determine the trained upper-layer deep reinforcement learner and output the trained path planning model.

[0114] In a more specific implementation of this embodiment, the process of determining the reward function includes: designing dense targets based on navigation targets, obstacle avoidance targets, smooth trajectory targets, and shortest path targets, and determining the reward function.

[0115] It should be noted that, since path planning is a typical sparse reward problem, reinforcement learners have difficulty obtaining sufficient feedback signals during training, resulting in slow convergence. Therefore, this embodiment adopts a dense reward function, which is designed to consider navigation objectives, obstacle avoidance objectives, smooth trajectory objectives, and shortest path objectives respectively. The navigation objective aims to enable the unmanned equipment to reach the target point, the obstacle avoidance objective aims to enable the unmanned equipment to avoid obstacles, the smooth trajectory objective aims to enable the unmanned equipment to move along a relatively smooth trajectory, and the shortest path objective aims to enable the unmanned equipment to complete the navigation task as quickly as possible.

[0116] Based on the above design, in one implementation, the process of determining the reward function value includes:

[0117] ;

[0118] In the formula, Based on the base reward function value, The distance of the unmanned equipment relative to the target point on the path before executing the action. The distance of the unmanned equipment relative to the target point on the path after performing the action. For the maximum linear velocity, The duration of adjacent time steps. The angle of the unmanned equipment relative to the target point on the path before performing the action. The angle of the unmanned equipment relative to the target point on the path after the action is performed. For the maximum angular velocity, The current linear velocity. The angular velocity is the current state. As a penalty for stagnation, For example, the adjustment coefficient (e.g., , , , According to the above formula, the navigation target is manifested as follows: Reach the target point +100 and The two items that are close to the target point, the obstacle avoidance objective is reflected in: Collision obstacle -100, smooth trajectory target is shown as Items (rewards for increasing linear speed) and The item (rewarding for reduced angular velocity), the shortest path objective is reflected in... (Providing a fixed penalty value at each step of execution, such as -0.1 / -1), this reward function design will encourage the agent to converge to the target point as quickly as possible, while appropriately increasing the linear velocity and decreasing the angular velocity to obtain a smoother trajectory.

[0119] Furthermore, for complex dynamic environments that rely solely on the basic reward function value... To address potential issues such as training convergence oscillations, policy shortsightedness, and the Sim-to-Real gap, this embodiment introduces a dynamic correction mechanism for the reward function based on digital twin co-evolution during the process of determining the trained path planning model.

[0120] Specifically, the process of determining the reward function value also includes: dynamically correcting the basic reward function value through a digital twin co-evolution mechanism; the specific process includes:

[0121] Construct a digital twin that matches the physical parameters of unmanned equipment;

[0122] Calculate the synchronization deviation between the predicted trajectory state of the digital twin at the current time step in the virtual environment and the actual trajectory state of the unmanned equipment;

[0123] Determine the obstacle density potential field value for the digital twin to extrapolate multiple future steps in a virtual environment;

[0124] The base reward function value is corrected based on the synchronization deviation degree and the obstacle density potential field value to determine the corrected reward function value;

[0125] The process for determining the value of the modified reward function includes:

[0126] ;

[0127] In the formula, This is the corrected final reward value. Based on the base reward function value, For twin confidence weights, The sensitivity coefficient for the Gaussian kernel width. To guide weighting in a forward-looking manner, As a time discount factor, For time step index, For the deduction step index, For the total number of simulation steps, For the first The obstacle density potential field value sensed by the digital twin during the step. It is a natural constant. This refers to the synchronization deviation.

[0128] It should be explained in detail that the aforementioned digital twin co-evolution mechanism achieves accurate replication of the physical characteristics of unmanned equipment in the real world by constructing a high-fidelity virtual mapping environment in a physics engine (such as Gazebo, Unity3D, etc.).

[0129] In the specific execution logic, at each training time step Synchronization deviation Used to quantify the difference between theoretical models and actual implementation; specifically, It can be the Euclidean distance between the actual trajectory state vector of the unmanned equipment at the current time step and the predicted trajectory state vector of the current time step derived from the previous time step of the digital twin. The trajectory state includes the position, velocity and yaw angle of the unmanned equipment. It can be understood that the actual trajectory state can be determined by the angular velocity of the unmanned state line when executing the current state in the environment of the deep reinforcement learner.

[0130] Synchronization Deviation Correction Term The term utilizes Gaussian radial basis functions (RBF) to construct a nonlinear gating mechanism; when the true state is highly consistent with the twin predicted state (i.e., When the exponential term approaches 1, the correction coefficient reaches its maximum value. This amplifies the base reward. This means that when the movement of unmanned equipment conforms to physical laws and is highly predictable, it will be given an additional "confidence reward". This forces the reinforcement learning agent to not only learn to "run fast", but also to "run steadily" and "conform to the physical model", which effectively solves the problem that reinforcement learning strategies fail due to differences in dynamics when deployed in practice.

[0131] At every step of the training It receives the current real-time trajectory state (position, velocity, yaw angle) of the unmanned equipment, injects this state into the digital twin as initial conditions, and utilizes the ultra-real-time computing power of the digital twin to quickly pre-simulate the future after adopting the current action strategy. Predicted trajectory state sequence for the next 5-10 time steps;

[0132] obstacle density potential field value The parameters represent the future of digital twins. The potential risks of the location and environment of the step; for example, It can be defined as the intensity of the repulsive force field calculated based on the local lidar point cloud density, or the proportion of grid cells occupied by obstacles in a local map.

[0133] Forward-looking guidance items The term is a long-term early warning mechanism based on a logarithmic potential field; traditional reinforcement learning often suffers from "short-sightedness" (i.e., only focusing on avoiding the current step), while by accumulating future... The environmental complexity of the step is such that even if there are no obstacles around the unmanned equipment at the current moment, if the digital twin calculates that it will enter a dead end or a high-density obstacle area in 3 seconds, this value will change significantly (depending on the design, it can be set as the negative reciprocal of negative penalty or positive guidance; here, the logarithmic form is taken to smooth the gradient in the high-density area). This mechanism gives the algorithm the ability to "predict future risks" and guide the unmanned equipment to avoid paths that seem open at present but actually lead to dangerous situations in the early stages.

[0134] By introducing this corrective formula This achieves a shift from "passive trial and error" to "active prediction," leveraging the forward-looking capabilities of digital twins to compensate for deficiencies in the basic reward function. The lag significantly improves the training efficiency and robustness of path planning models in complex and unknown environments.

[0135] Step 104: For the received environmental perception data set of the unmanned equipment, perform path planning based on data preprocessing, feature extraction, fuzzy input variable determination, and the trained path planning model.

[0136] It should be noted that after training is completed, when the unmanned equipment receives the environmental perception data set to be processed, the training environmental perception data set of the unmanned equipment is preprocessed and features are extracted to determine the planning environmental perception features. The planning fuzzy input variables are determined using the environmental perception data set to be processed. Based on the planning environmental perception features, the planning fuzzy input variables and the current linear angular velocity of the unmanned equipment, the path planning is performed through the trained path planning model, and the planned unmanned state linear angular velocity is output and handed over to the unmanned equipment actuator for execution.

[0137] In this embodiment of the invention, to address the problems of existing path planning such as high requirements for the navigation environment, slow and unstable training process, sparse environmental rewards, and poor robustness (especially in narrow passages and environments with many traps and obstacles), this embodiment adopts a hierarchical reinforcement learning training framework. The lower layer network uses a fuzzy logic controller to implement adaptive escape navigation strategy, obstacle avoidance strategy, and wall-hugging motion strategy respectively. The upper layer network uses a deep reinforcement learning framework to make high-dimensional decisions based on the perception and understanding of the environmental situation using a deep neural network. Finally, it is formalized into a joint action formed by a linear combination of different sub-actions, enabling it to achieve autonomous perception of the environment, decision-making, and execution of action commands for the entire process of path planning. It has basic characteristics such as a certain degree of autonomy and adaptive decision-making in different environments. Because it uses expert prior knowledge such as fuzzy logic controllers, it has higher training efficiency and higher robustness in the training stage. At the same time, the combination of multiple controllers suitable for path planning tasks improves the exploration efficiency in the early stage of model training, thus making it more efficient and less sensitive to the initial value of the model.

[0138] Please see Figure 6 The second embodiment of the present invention provides an unmanned equipment path planning system based on hierarchical multi-expert reinforcement learning, comprising:

[0139] The model building module 601 is used to build a trainable path planning model based on an upper-layer deep reinforcement learner and a lower-layer velocity fuzzy logic controller.

[0140] The feature extraction module 602 is used to preprocess and extract features from the training environment perception data set of unmanned equipment to determine the training environment perception features.

[0141] The variable determination module 603 is used to determine the training fuzzy input variables of the lower-level velocity fuzzy logic controller using the training environment-aware data set;

[0142] The model training module 604 is used to input the path planning model to be trained based on the training environment perception features, the current line angular velocity of the unmanned equipment, and the training fuzzy input variables. It outputs the current state line angular velocity of the unmanned equipment through the fusion of upper and lower layer decisions, and determines the reward function value based on the current state line angular velocity of the unmanned equipment to update the upper layer deep reinforcement learner until the training stopping condition is met, and then determines the trained path planning model.

[0143] The path planning module 605 is used to perform path planning on the received environmental perception data set of unmanned equipment, based on data preprocessing, feature extraction, fuzzy input variable determination, and a trained path planning model.

[0144] Embodiment 3 of the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the path planning method for unmanned equipment based on hierarchical multi-expert reinforcement learning as described in Embodiment 1 of the present invention.

[0145] Embodiment 4 of the present invention also provides a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements the steps of the path planning method for unmanned equipment based on hierarchical multi-expert reinforcement learning as described in Embodiment 1 of the present invention.

[0146] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0147] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between systems or modules may be electrical, mechanical, or other forms.

[0148] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0150] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unmanned equipment path planning method based on hierarchical multi-expert reinforcement learning, characterized in that, include: Construct a trainable path planning model based on an upper-layer deep reinforcement learner and a lower-layer velocity fuzzy logic controller; The training environment perception data set of unmanned equipment is preprocessed and features are extracted to determine the training environment perception features. The training environment-aware data set is used to determine the training fuzzy input variables for the lower-level velocity fuzzy logic controller; Based on the training environment perception features, the current linear angular velocity of the unmanned equipment, and the training fuzzy input variables, the path planning model to be trained is input. The current state linear angular velocity of the unmanned equipment is output through upper and lower layer decision fusion. The reward function value is determined based on the current state linear angular velocity to update the upper-layer deep reinforcement learner until the training stopping condition is met, at which point the trained path planning model is determined. This includes: The action network of the upper-layer deep reinforcement learning machine takes the training environment perception features, the training relative target point angle distance, the training relative obstacle angle distance and the current line angular velocity of the unmanned equipment as inputs, and outputs the fuzzy action weights corresponding to the lower-layer velocity fuzzy logic controller. The lower-level velocity fuzzy logic controller outputs the corresponding fuzzy linear angular velocity based on the training relative target point angular distance, the training relative obstacle angular distance, and the training relative wall angle. The fuzzy linear angular velocities and fuzzy action weights are used to perform weighted fusion, and the current state unmanned linear angular velocity is output. The Q-network of the upper-layer deep reinforcement learner takes the training environment perception features and the current state unmanned state line angular velocity as input, and outputs the action value function. The V network, which uses the training environment-aware features as input, outputs a state value function through the upper-layer deep reinforcement learner. The reward function value is determined based on the current state unmanned state line angular velocity, and the Q network, V network and action network are updated in combination with the action value function and the state value function until the training stopping condition is met. The trained upper-layer deep reinforcement learner is determined and the trained path planning model is output. For the received environmental perception data set of unmanned equipment, path planning is performed based on data preprocessing, feature extraction, fuzzy input variable determination, and a trained path planning model. The lower-level velocity fuzzy logic controller includes an adaptive escape navigation fuzzy logic controller, an obstacle avoidance fuzzy logic controller, and a wall-hugging motion fuzzy logic controller. The adaptive escape navigation fuzzy logic controller takes the relative target point angular distance of the unmanned equipment with respect to the path target point as input and the escape fuzzy linear angular velocity as output. The obstacle avoidance fuzzy logic controller takes the relative obstacle angle distance between the unmanned equipment and the nearest obstacle as input and the obstacle avoidance fuzzy linear angular velocity as output. The wall-attaching motion fuzzy logic controller takes the yaw angle of the unmanned equipment relative to the wall's parallel direction as input and the wall-attaching fuzzy linear angular velocity as output.

2. The path planning method for unmanned equipment based on hierarchical multi-expert reinforcement learning according to claim 1, characterized in that, The Q-network of the upper-layer deep reinforcement learning machine includes a first feature extraction branch, a second feature extraction branch, a feature fusion module, and a fully connected neural network; the Q-network of the upper-layer deep reinforcement learning machine takes the training environment-aware features and the current state unmanned state line angular velocity as input, and outputs an action value function, including: The training environment perception features are extracted by the first feature extraction branch of the Q network of the upper-layer deep reinforcement learner to determine the deep features of the environment. The motion state features are determined by extracting the angular velocity of the current unmanned state line through the second feature extraction branch of the Q network of the upper deep reinforcement learner. After fusing the environmental depth features and motion state features through the feature fusion module of the Q-network of the upper-layer deep reinforcement learner, the feature is input into the fully connected neural network for feature processing and outputs the action value function.

3. The unmanned equipment path planning method based on hierarchical multi-expert reinforcement learning according to claim 1, characterized in that, The process of determining the reward function value includes: ； In the formula, Based on the base reward function value, The distance of the unmanned equipment relative to the target point on the path before executing the action. The distance of the unmanned equipment relative to the target point on the path after performing the action. For the maximum linear velocity, The duration of adjacent time steps. The angle of the unmanned equipment relative to the target point on the path before performing the action. The angle of the unmanned equipment relative to the target point on the path after the action is performed. For the maximum angular velocity, The current linear velocity. The angular velocity is the current state. As a penalty for stagnation, This is the adjustment coefficient.

4. The unmanned equipment path planning method based on hierarchical multi-expert reinforcement learning according to claim 1, characterized in that, The process of determining the target point of the path includes: Analyze the environmental perception data set to determine if obstacles exist; If it does not exist, the preset target point will be used as the path target point; If present, potential target areas are marked based on the environmental perception data set of the obstacle's vicinity. The location of the potential target area is determined based on environmental perception data, the position of the unmanned equipment, and the yaw angle of the unmanned equipment, and a path target point is selected from the potential target area.

5. The unmanned equipment path planning method based on hierarchical multi-expert reinforcement learning according to claim 3, characterized in that, The process of determining the reward function value also includes: Construct a digital twin that matches the physical parameters of the unmanned equipment; Calculate the synchronization deviation between the predicted trajectory state of the digital twin at the current time step in the virtual environment and the actual trajectory state of the unmanned equipment; Determine the obstacle density potential field value of the digital twin in the virtual environment for future multiple time steps; The base reward function value is corrected based on the synchronization deviation and the obstacle density potential field value to determine the corrected reward function value. The process for determining the value of the modified reward function includes: ； In the formula, This is the corrected final reward value. Based on the base reward function value, For twin confidence weights, The sensitivity coefficient for the Gaussian kernel width. To guide weighting in a forward-looking manner, As a time discount factor, For time step index, For the deduction step index, For the total number of simulation steps, For the first The obstacle density potential field value sensed by the digital twin during the step. It is a natural constant. This refers to the synchronization deviation.

6. A path planning system for unmanned equipment based on hierarchical multi-expert reinforcement learning, characterized in that, The method for path planning of unmanned equipment based on hierarchical multi-expert reinforcement learning as described in any one of claims 1-5 includes: The model building module is used to build a trainable path planning model based on an upper-layer deep reinforcement learner and a lower-layer velocity fuzzy logic controller. The feature extraction module is used to preprocess and extract features from the training environment perception data set of unmanned equipment to determine the training environment perception features. The variable determination module is used to determine the training fuzzy input variables of the lower-level velocity fuzzy logic controller using the training environment-aware data set; The model training module is used to input the path planning model to be trained based on the training environment perception features, the current line angular velocity of the training unmanned equipment, and the training fuzzy input variables. It outputs the current state line angular velocity of the unmanned equipment through the fusion of upper and lower layer decisions, and determines the reward function value based on the current state line angular velocity of the unmanned equipment to update the upper layer deep reinforcement learner until the training stopping condition is met, and then determines the trained path planning model. The path planning module is used to plan the path of the received environmental perception data set of unmanned equipment based on data preprocessing, feature extraction, fuzzy input variable determination, and a trained path planning model.

7. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the path planning method for unmanned equipment based on hierarchical multi-expert reinforcement learning as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the unmanned equipment path planning method based on hierarchical multi-expert reinforcement learning as described in any one of claims 1-5.

Images