Tunnel construction robot path planning method and system based on deep reinforcement learning

Abstract

The invention discloses a tunnel construction robot path planning method and system based on deep reinforcement learning, and relates to the technical field of tunnel construction automation and robot navigation, and the method comprises the steps: pre-defining a state space and an action space of robot path planning in a tunnel construction environment, and building a dynamic reward function; constructing a deep reinforcement learning architecture based on a lightweight action network and a comment network, and training an action-comment network based on state-action pair data in an experience pool by taking the sum of an accumulated reward and an entropy regularization item of a maximization strategy as a target; and on the basis of the current state information of the tunnel construction robot, the optimal moving action of the next step is obtained by utilizing the trained action network, and path planning of the robot is realized. According to the method, the deep reinforcement learning technology is applied to the tunnel construction scene, and the autonomous decision-making ability and the path planning precision of the robot in the dynamic construction environment are optimized and improved in combination with real-time data acquisition and processing of the complex tunnel environment.

Description

Technical Field

[0001] This invention relates to the fields of tunnel construction automation and robot navigation technology, and in particular to a tunnel construction robot path planning method and system based on deep reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous expansion of tunnel and underground engineering construction, the complexity and uncertainty of the construction environment have significantly increased, and the requirements for safety and efficiency in engineering construction are rising daily. Against this backdrop, the rapid development of artificial intelligence and robotics has provided key technological support for the automation of tunnel construction, and tunnel construction robots are gradually becoming core operational equipment in modern tunnel engineering.

[0004] However, the unique environment inside tunnels presents challenges to the real-time performance, environmental adaptability, and safety of robot path planning due to weak signals, complex spatial layouts, and the coexistence of static obstacles (such as supports and pipes) and dynamic obstacles (such as construction workers and mobile equipment). Current mainstream path planning methods are difficult to apply to tunnel construction environments and have obvious limitations: the graph search-based A* algorithm has a high computational burden and high response latency in dynamic environments, failing to meet real-time requirements; the random sampling-based RRT (Rapid Expanding Random Tree) algorithm lacks path smoothness and has limited ability to avoid dynamic obstacles; and swarm intelligence-based algorithms such as genetic algorithms and ant colony algorithms heavily rely on manual parameter tuning and have poor robustness.

[0005] Currently, deep reinforcement learning technology has been applied in the field of robot path planning, but existing solutions have not been customized for dynamic, unstructured, and multi-constraint environments such as tunnel construction: on the one hand, the state space representation is difficult to fully characterize the features of the construction environment, and on the other hand, the reward function lacks multi-objective collaborative design, making it difficult to take into account construction needs such as safety, efficiency, and energy saving, resulting in low training efficiency and poor path practicality. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, this invention provides a path planning method and system for tunnel construction robots based on deep reinforcement learning. By applying deep reinforcement learning technology to tunnel construction scenarios and combining it with real-time data acquisition and processing in complex tunnel environments, the invention significantly improves the robot's autonomous decision-making ability and path planning accuracy in dynamic construction environments, thus solving the problems of high reliance on manual labor and low intelligence in the path planning of existing tunnel construction robots.

[0007] In a first aspect, the present invention provides a path planning method for tunnel construction robots based on deep reinforcement learning.

[0008] A path planning method for tunnel construction robots based on deep reinforcement learning, comprising: The state space and action space of the robot path planning in the tunnel construction environment are predefined, and a dynamic reward function is built. A deep reinforcement learning architecture based on a lightweight action network and a comment network is constructed. The goal is to maximize the sum of the cumulative reward and the entropy regularization term of the policy. The action-comment network is trained using state-action pairs from the experience pool. Specifically, the action network generates a motion policy based on the robot's current state information to output the robot's next action command. This action command is then constrained by a differentiable safety constraint module introduced at the action network's output, resulting in a safe action command that is executed. The comment network calculates the objective function and updates the action-comment network parameters based on the input robot state-action pairs. Based on the current state information of the tunnel construction robot, the optimal movement action for the next step is obtained using a trained motion network, thereby realizing the robot's path planning.

[0009] A further technical solution is that the state information of the tunnel construction robot includes positioning state information, environmental perception state information, and task state information; the actions of the tunnel construction robot include a three-dimensional component of movement speed. The positioning status information includes three-dimensional position coordinates and three-axis orientation; the environmental perception status information includes local obstacle distance information, dynamic obstacle coordinates, and relative velocity; and the task status includes the absolute coordinates of the target point.

[0010] In a further technical solution, the dynamic reward function is a total reward function composed of a weighted sum of multiple sub-reward functions, including: target approach reward function, progress reward function, collision penalty function, energy consumption reward function, motion smoothness reward function, and terrain elevation reward function.

[0011] A further technical solution involves training by maximizing an objective function, which is the sum of the standard expected cumulative reward and the policy's entropy regularization term, expressed as: ; in, For strategy The state below -action Access distribution; This is a discount factor used to balance the importance of current rewards and future rewards; Representation Strategy In state Entropy is used to quantify the randomness and exploratory ability of a strategy; This represents the entropy coefficient, used to balance the importance of reward maximization and policy entropy. This is the total reward function.

[0012] In a further technical solution, the safety constraint module is used to convert the original actions output by the action network. Limited to based on the current perception state Calculated safety boundary within, that is The clip function is used to restrict the elements in an array to between the minimum and maximum values.

[0013] A further technical solution involves a lightweight action network used to map high-dimensional states into continuous action commands. This network employs a parallel hierarchical feature extraction and fully connected layer structure. For different state information in the input state vector, the localization state information and task state information are directly input into the fully connected layer. The local obstacle distance information in the environmental perception state information is extracted through a one-dimensional depthwise separable convolution module. The dynamic obstacle information is input into the fully connected layer. The features extracted from each path are concatenated, and then fused and decided by the fully connected layer. Finally, the output layer outputs a normalized action command.

[0014] Further technical solutions include configuring first and second online comment networks and corresponding target comment networks; The first and second online comment networks adopt a parameter-sharing design, sharing the input layer and the first hidden layer, and then branching into two independent lightweight evaluation heads. The lightweight evaluation heads are used to concatenate the shared backbone output features with the action vector and map them to Q-values. The structure of the target comment network is consistent with that of the online comment network, and the network parameters are synchronized through periodic soft updates based on the online comment network.

[0015] Secondly, the present invention provides a path planning system for tunnel construction robots based on deep reinforcement learning.

[0016] A path planning system for tunnel construction robots based on deep reinforcement learning, comprising: A deep reinforcement learning algorithm building module is used to predefine the state space and action space of robot path planning in a tunnel construction environment, and to build a dynamic reward function; The deep reinforcement learning module is used to construct a deep reinforcement learning architecture based on a lightweight action network and a comment network. The goal is to maximize the sum of the cumulative reward and the entropy regularization term of the policy. The action-comment network is trained based on state-action pairs from the experience pool. Specifically, the action network generates a motion policy based on the robot's current state information to output the robot's next action command. This action command is then constrained by a differentiable safety constraint module introduced at the action network's output, resulting in a safe action command that is executed. The comment network calculates the objective function and updates the action-comment network parameters based on the input robot state-action pairs. The path planning module is used to obtain the optimal next movement action based on the current state information of the tunnel construction robot and the trained motion network, thereby realizing the robot's path planning.

[0017] Thirdly, the present invention also provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-described deep reinforcement learning-based path planning method for tunnel construction robots when executing the executable instructions stored in the memory.

[0018] Fourthly, the present invention also provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-described deep reinforcement learning-based tunnel construction robot path planning method.

[0019] Fifthly, the present invention also provides a computer program product comprising executable instructions stored in a computer-readable storage medium; wherein, when the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned deep reinforcement learning-based tunnel construction robot path planning method is implemented.

[0020] The above one or more technical solutions have the following beneficial effects: 1. This invention proposes a path planning method and system for tunnel construction robots based on deep reinforcement learning. By designing a tunnel environment state space including localization state, environmental perception state, and task state, and a corresponding continuous action space, and constructing an adaptive dynamic multi-objective reward function adapted to the tunnel scenario, a framework based on the flexible action-comment (SAC) algorithm is built and trained. A safety constraint layer and a tunnel scenario-based training mechanism are integrated into the action network, and a lightweight neural network is designed to adapt to the robot's embedded computing power. This enables precise navigation with autonomous obstacle avoidance, smooth and efficient path, and optimized energy consumption in dynamic and complex scenarios, suppressing collision risks and meaningless path exploration, while taking into account the real-time responsiveness of the embedded platform and the safety of engineering applications. This solves the problems of existing tunnel construction robot path planning, such as high dependence on manual labor, low level of intelligence, and difficulty in adapting to dynamic and complex tunnel environments.

[0021] 2. This invention utilizes reinforcement learning to autonomously learn tunnel environment patterns and optimal path strategies, eliminating the need for manual parameter tuning. This addresses the issues of high reliance on human intervention and low intelligence in existing tunnel construction robot path planning. During the learning process, a dynamic reward mechanism with weighted summation of multiple sub-reward functions is designed. These dynamic reward functions include sub-functions applicable to tunnel scenarios, such as terrain elevation rewards and collision penalties, taking into account core construction requirements such as safety, efficiency, energy saving, and path smoothness. This guides the robot to adapt to special working conditions such as steep slopes and complex terrain, solving the problem of insufficient path practicality caused by the single-objective orientation of traditional solutions. Furthermore, an entropy regularization term is introduced to balance the exploration and utilization of strategies, enabling the robot to autonomously adjust its path when facing unknown obstacles or environmental changes, reducing the need for human intervention. The final path planning result combines smoothness, energy saving, and terrain adaptability, effectively reducing robot equipment wear and tear, improving construction efficiency, and effectively avoiding both dynamic and static obstacles, ensuring construction safety.

[0022] 3. In this invention, the action network adopts parallel hierarchical feature extraction and depthwise separable convolutional structure, and the comment network adopts parameter sharing design. The number of network parameters is greatly reduced without sacrificing performance, which is suitable for the embedded computing power limitation of tunnel construction robots. Furthermore, a differentiable safety constraint module is introduced at the output end of the action network. Based on real-time perception state, the safety action boundary is dynamically calculated, and the original action is trimmed to the safe range, which significantly reduces the risk of collision and effectively ensures the safety of the construction process.

[0023] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0025] Figure 1 This is an overall architecture diagram of the tunnel construction robot path planning method based on deep reinforcement learning as described in an embodiment of the present invention; Figure 2 This is a schematic diagram of the state space configuration based on the tunnel construction environment in an embodiment of the present invention. Detailed Implementation

[0026] It should be noted that the following detailed descriptions are exemplary and are intended only to describe specific embodiments and to provide further explanation of the invention, and are not intended to limit the scope of exemplary embodiments of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] Example 1 This embodiment proposes a path planning method for tunnel construction robots based on deep reinforcement learning. By constructing a maximum entropy deep reinforcement learning framework, the robot can make autonomous decisions in the continuous action space. With the help of differentiable safety constraints and progressive course learning mechanism, the robot's adaptability and safety in dynamic and complex tunnel environments are significantly enhanced. In addition, the lightweight network design ensures smooth and efficient path operation, while also providing engineering practicality for real-time operation in embedded systems.

[0028] The tunnel construction robot path planning method proposed in this embodiment, such as... Figure 1 As shown, the specific steps include: Step S1: Predefine the state space and action space of the robot path planning in the tunnel construction environment, and build a dynamic reward function.

[0029] In step S1, the state and action space are defined in advance based on the tunnel construction environment. Specifically, for the target tunnel, static and dynamic obstacles are set up inside the tunnel. For the robot body, the following can be assembled: (1) A 16-line lidar is installed on the top of the robot. The scanning frequency is 10Hz. It is used to acquire point cloud data of the surrounding 360° horizontal field of view and 30° vertical field of view to form environmental perception state information E.

[0030] (2) An inertial measurement unit (IMU) and a wheel odometry are used to provide robot pose estimation, which constitutes positioning state information L.

[0031] (3) A global positioning system (which may be a UWB indoor positioning system) is used to provide the robot’s real position and target point position in the simulation for training and evaluation, wherein the target point position constitutes the task state information T.

[0032] The robot's control objective is to autonomously navigate from the tunnel's starting point to the designated target point (i.e., task status information T), while avoiding collisions with all static and dynamic obstacles and maintaining a smooth path and high energy efficiency as much as possible.

[0033] Based on this, the state space and motion space of the robot model are defined. For example... Figure 2 As shown, the state space S represents the set of all information that the robot can perceive and use for decision-making. This state space is defined as consisting of three parts, denoted as S=(L,E,T), where: (1) L represents the positioning state, indicating the robot's own pose information. The robot's pose information is obtained by fusing odometry and IMU data, including three-dimensional position. With three-axis orientation Specifically, this can form a 6-dimensional vector L=[x, y, z, ... Where (x, y, z) represent the position coordinates in the world coordinate system, in meters; The angles () represent the orientation in Euler angles, expressed in radians.

[0034] (2) E represents the environmental perception state, which includes information on the distance to local obstacles. Dynamic obstacle coordinates Relative velocity .

[0035] The above local obstacle distance information This can be obtained by processing the number of LiDAR scan lines, specifically processing the point cloud data of a 16-line LiDAR. This includes: first, dividing the 180° sector area in front of the robot (i.e., the main direction of travel) into N=36 sectors on a horizontal plane, each sector being 5°; then, calculating the shortest distance from all point clouds within each sector to the robot; finally, It is a 36-dimensional vector, where each element represents the distance (in meters) to the nearest obstacle in a direction. If there is no point cloud in a sector, the distance value is set to the maximum detection range, for example, 10m.

[0036] (3) T represents the task state, including the absolute coordinates of the target point in the world coordinate system, which can be expressed as: .

[0037] Finally, L (6-dimensional) and E ( The 36-dimensional state vector plus the 12-dimensional dynamic obstacle equals 48-dimensional state vector, and the T (3-dimensional) state vector are concatenated to obtain a 57-dimensional state vector. This serves as the input for subsequent neural networks.

[0038] Furthermore, to simplify the model and focus on high-level decision-making, the robot's kinematics is abstracted as a point mass capable of omnidirectional movement, and its spatial action space A is defined as the three-dimensional components of the movement velocity, i.e.: .

[0039] Furthermore, a dynamic reward function based on the tunnel construction environment is constructed. Specifically, considering the unstructured and dynamically changing tunnel environment in which the tunnel construction robot operates, this embodiment designs an adaptive dynamic reward function. Through the collaborative guidance of multi-objective rewards, the robot can learn a safe, efficient, and energy-saving walking strategy. The designed total reward function... It consists of a weighted sum of multiple sub-reward functions, and its general form is expressed as: ; in, The weight coefficients for each sub-reward function are dynamically configured based on the specific task stage or environment complexity.

[0040] In this embodiment, the sub-reward function It is specifically designed for tunnel construction scenarios, including but not limited to target proximity rewards. Progress Rewards Collision penalty Energy consumption reward Smooth motion reward Terrain elevation reward Each sub-function focuses on one aspect of the robot, and multiple sub-reward functions are designed to comprehensively address the tunnel environment.

[0041] Then, the total reward function It can be optimized as follows: .

[0042] The sub-reward functions and variables are defined as follows: (1) Goal Achievement Rewards When the robot successfully reaches the target point, it receives a one-time positive reward; otherwise, it receives zero. The mathematical representation of this is: ; (2) Distance Reward : Guide the robot to get closer to the target with each step, in time At that time, the distance between the robot and the target point is (Based on the state space), the distance reward is calculated as follows: ; Among them, when The time indicates that the robot body is moving towards the target direction.

[0043] (3) Collision penalty Used to guide robots in obstacle avoidance. When a collision occurs with a static obstacle (such as a support or pipe) or a dynamic obstacle (such as mobile equipment or people), the penalty force is proportional to the relative velocity at the time of collision. Related, can be expressed as: ; in, , This represents the robot's relative velocity at point A. This represents the robot's relative velocity at point B.

[0044] (4) Energy penalty This is a constant penalty, meaning that before reaching the goal, the robot will be penalized for each step it takes due to energy consumption. This is to encourage efficient and energy-saving paths. The value can be adjusted through energy consumption calibration experiments. This penalty term can be expressed as: .

[0045] (5) Path smoothness reward This reward is designed to encourage robots to produce smooth motion trajectories. Its design comprehensively evaluates motion smoothness from two dimensions: linear velocity and angular velocity. Its basic form is defined as the sum of the total smoothness... A positively correlated linear function is expressed as: ; in, As a reward scaling factor, As the baseline smoothness, in this model, we set... =5, =0.65; Total smoothness The expression for the jitter, which combines linear and angular motion, is as follows: ; ; ; in, Characterizing the change in linear velocity, Characterizing changes in angular velocity, This represents the angular velocity at time t.

[0046] (7) Terrain Adaptability Reward This reward is used to guide the robot to choose an efficient and safe route based on the terrain slope. The reward function is based on the slope value at the current location. (Unit: degrees) and its relationship with the robot's maximum climbing ability The relationship is evaluated in segments, and the specific definitions are as follows: ; in, The maximum climbing angle designed for the robot; exceeding this angle will result in a severe penalty, prohibiting passage. This is a slope threshold used to distinguish between steep and gentle slopes. It can be set according to the actual terrain and robot performance, for example... =15°.

[0047] Step S2: Construct a deep reinforcement learning architecture based on a lightweight action network and a comment network, with the goal of maximizing the sum of the cumulative reward of the policy and the entropy regularization term, and train the action-comment network based on the state-action pair data in the experience pool.

[0048] The motion network generates a motion strategy based on the input robot current state information to output the robot's next action command. The action command is limited by the differentiable safety constraint module introduced at the output end of the motion network to output a safe action command and execute it. The comment network calculates the objective function and updates the parameters of the motion-comment network based on the input robot state-action pair data.

[0049] In step S2, an algorithm framework based on Flexible Action-Comment (SAC) is built and trained. Specifically, considering the high-dimensional, continuous, and partially observable state space of the tunnel construction environment, and the stringent requirements for the continuity and smoothness of motion control for construction robots, traditional deep reinforcement learning algorithms based on discrete action spaces have inherent limitations. To overcome this problem and significantly improve the robot's autonomous navigation and fine control capabilities in dynamic, unstructured tunnels, this embodiment designs a path planning method based on a maximum entropy deep reinforcement learning framework, namely the Flexible Action-Comment algorithm. The core of this method lies in constructing an intelligent decision-making system that includes an action network (or actor network) and a comment network (or critic network), wherein: Action Network Used to determine the current state Generate the robot's motion strategy. This network determines the robot's motion strategy based on the current state. Output a value in the continuous action space The conditional probability distribution on the output, i.e., the parameters of a Gaussian distribution of a random policy, are: the mean vector. Sum of logarithmic standard deviation vector By sampling from this conditional probability distribution, the robot's three-dimensional continuous velocity control commands are obtained and generated. ,Right now Among them, state For the state space defined in step S1 above, actions It is a continuous velocity vector.

[0050] Furthermore, the optimization objective of the above algorithm, based on the standard expected cumulative reward, innovatively introduces an entropy regularization term of the policy, and its maximization objective is: ; in, For strategy The state below -action Access distribution; The discount factor, which can take values ​​such as 0.9 or 0.99, is used to balance the importance of current rewards and future rewards. In problems with infinite time (T→∞), This ensures that the expected value of accumulated rewards is limited, avoiding infinite returns; Representation Strategy In state Entropy is used to quantify the randomness and exploratory ability of a strategy; This represents the entropy coefficient, used to balance the importance of reward maximization and policy entropy. The larger the value, the stronger the randomness of the strategy (exploring more), and The smaller the value, the more the strategy focuses on maximizing rewards (utilizing more resources). This mechanism enables robots to achieve an autonomous and efficient balance between precisely approaching the target and actively exploring the environment to avoid unknown risks.

[0051] from Starting with the goal of maximizing entropy, we configure the first and second online comment networks and their corresponding target comment networks. Specifically, the first and second online comment networks... and Two independent Q-functions are used to collaboratively evaluate the value of state-action pairs, reducing the risk of overestimation through structural redundancy; first and second objective comment networks. and Network parameters Periodicity is derived from online network soft updates and is used to calculate stable training target values, thereby improving training stability.

[0052] In this embodiment, the Q-function estimated by the comment network is defined as: from state Execute action Initially, the expected value of the weighted sum of all future discount rewards and policy entropy is given. The Q function is iteratively updated using the soft Bellman equation (i.e., the core recursive equation under the maximum entropy objective), expressed as: ; In the above formula, Indicates the total reward. Indicates the next state Execute action strategy, Indicates the next state The expected future rewards.

[0053] In actual algorithms, the above equations are not solved directly, but rather a parameterized neural network is used. To approximate reality The network is trained by minimizing the soft Bellman residual, which can be expressed as: ; in, To compute the target value during training, the parameters are used as supervision parameters for training the Q-network. .

[0054] The parameters of the target review network are calculated using an exponential moving average and slowly updated from the online average network, i.e.: ; in, This is the soft update coefficient, which can be 0.005 or 0.001. To maintain the coefficient.

[0055] As one implementation method, to meet the high safety requirements of tunnel construction, this embodiment also makes a key improvement to the basic framework, namely, introducing a differentiable safety constraint module at the output end of the motion network to ensure the original motion output by the network. Dynamically limited to based on the current perception state Calculated safety boundary Within, that is The clip function is used to restrict the elements in an array to between a specified minimum and maximum value, thereby embedding domain knowledge into the learning process and ensuring mobile security.

[0056] Among them, based on real-time lidar data Calculate the safe speed limit for each direction of movement. Specifically, assuming the distance to the nearest obstacle in a certain direction is d, calculate the maximum permissible speed in that direction. ,in =0.3 meters (according to experimental calibration) is the safe distance, and τ=0.5 is the time constant. Finally, the original motion output by the motion network is... Cut to Within the range, a safe action was obtained. .

[0057] The dynamic multi-objective reward function designed above The system employs a course-based learning strategy, starting with training agents in static, simple tunnel environments and gradually transitioning to dynamic scenarios with varying obstacles and complex terrain, in order to improve the robustness and generalization ability of the final strategy.

[0058] As one implementation method, to adapt to the limitations of robot computing power and achieve efficient real-time decision-making on embedded platforms, this embodiment also performs a lightweight construction of the core neural network of the SAC algorithm, specifically including: (1) Lightweight Action Network: This network is responsible for mapping high-dimensional states to continuous action instructions. Its lightweight core lies in the use of hierarchical feature extraction and a simplified fully connected structure. Input processing layer: for the state vector For data of different types (such as location, LiDAR point cloud, target point), design parallel feature extraction paths: for localization status... and task status Structured low-dimensional data is directly fed into the fully connected layer; for environmental perception state... Local obstacle distance information (Typically a one-dimensional sequence from LiDAR), information related to dynamic obstacles , This paper innovatively introduces a one-dimensional convolutional module for initial feature extraction. This module consists of 1-2 layers of one-dimensional convolutions, using depthwise separable convolutions to further reduce the number of parameters and efficiently capture local patterns in the spatial arrangement of obstacles. Based on this, the feature vectors extracted by the parallel pathways are concatenated and then fused and used for decision-making through a minimal fully connected network (e.g., 256 → 128 neurons). The output layer also generates the mean of the actions. With log standard deviation However, the training process for small networks can be stabilized through weight normalization or layer normalization techniques.

[0059] With input 57-dimensional state For example, a lightweight design is implemented for the structure of this action network, specifically including: Input processing layer: For the 57-dimensional state vector, the localization state L (6-dimensional) and task state T (3-dimensional) are directly fed into a fully connected layer (FC, 128 neurons, ReLU activation); for the environmental perception state E... (36-dimensional) data first passes through a one-dimensional convolutional module (1DConv, kernel size 3, depthwise separable convolution, output channels 16, ReLU activation), then flattens it and feeds it into another fully connected layer (FC, 64 neurons, ReLU activation); for the dynamic obstacle information (12-dimensional, including relative coordinates and relative velocity) in the environmental perception state E, since it is structured low-dimensional data, it is directly fed into an independent fully connected layer (FC, 64 neurons, ReLU activation).

[0060] Feature fusion and decision layer: integrating the localization-task pathway (128 dimensions) The output feature vectors of the pathway (64-dimensional) and the dynamic obstacle pathway (64-dimensional) are concatenated (128+64+64=256-dimensional) and passed through a fusion layer (FC, 256→128 neurons, ReLU activation).

[0061] Output layer: Two independent fully connected (FC) layers, outputting the mean μ (3D) and log-standard deviation logσ (3D) of the action, respectively, and the final action. It is sampled from the distribution N(μ,exp(logσ)).

[0062] (2) The two online evaluation networks employ a parameter-sharing and simplified evaluation head design to reduce computational cost while maintaining the advantages of double-Q estimation. The two evaluation networks share the input layer and the first hidden layer. The structure of this shared backbone network is similar to or identical to the part before the fusion decision layer of the action network, achieving reuse of state feature extraction and significantly reducing the number of parameters. After the shared backbone, the network branches into two independent lightweight evaluation branches, each consisting of only 1-2 small fully connected layers (e.g., 128→64 neurons), responsible for mapping the features fused with action information to the final Q-value scalar. Finally, the output features of the shared backbone are compared with the action vector. The data is concatenated and fed into two independent evaluation heads (each fully open: 128→64→1, ReLU activated), outputting two Q values. and action.

[0063] The target review network serves as the anchor point for stable training. Its structure is completely consistent with the corresponding online review network (including shared backbone and independent evaluation heads). Its parameters are synchronized from the online network through a soft update mechanism, which does not introduce additional computational burden.

[0064] During the training process described above, the training parameters used were: discount factor γ = 0.99, and replay buffer size. Soft update coefficient =0.005; the entropy coefficient α adopts an automatic adjustment mechanism, with an initial value of 0.2, and the target entropy is set to -dim(A) = -3. Training uses the Adam optimizer, and the learning rate for both the action network and the comment network is 0.005. The above parameters can be adjusted according to actual needs.

[0065] Step S3: Based on the current state information of the tunnel construction robot, use the trained motion network to obtain the optimal movement action for the next step, thereby realizing the robot's path planning.

[0066] Specifically, after training, the action network (without sampling layers, directly using μ as output) and its safety constraint module are deployed to the robot's embedded computer (such as NVIDIA Jetson Xavier NX) to achieve real-time (10Hz) online path planning and motion control.

[0067] Based on the above methods, for unstructured construction environments in tunnels characterized by weak signals, complex spatial layouts, and dynamically changing obstacles, this approach integrates tunnel-based scenario-based state-action space design, multi-objective adaptive dynamic reward functions, differentiable safety constraint layers, and lightweight neural network architectures to achieve precise navigation with autonomous obstacle avoidance, smooth and efficient paths, and optimized energy consumption in dynamic and complex scenarios. This suppresses collision risks and meaningless path exploration, while also ensuring the real-time responsiveness of embedded platforms and the safety of engineering applications.

[0068] Example 2 This embodiment provides a path planning system for tunnel construction robots based on deep reinforcement learning, including: A deep reinforcement learning algorithm building module is used to predefine the state space and action space of robot path planning in a tunnel construction environment, and to build a dynamic reward function; The deep reinforcement learning module is used to construct a deep reinforcement learning architecture based on a lightweight action network and a comment network. The goal is to maximize the sum of the cumulative reward and the entropy regularization term of the policy. The action-comment network is trained based on state-action pairs from the experience pool. Specifically, the action network generates a motion policy based on the robot's current state information to output the robot's next action command. This action command is then constrained by a differentiable safety constraint module introduced at the action network's output, resulting in a safe action command that is executed. The comment network calculates the objective function and updates the action-comment network parameters based on the input robot state-action pairs. The path planning module is used to obtain the optimal next movement action based on the current state information of the tunnel construction robot and the trained motion network, thereby realizing the robot's path planning.

[0069] Example 3 This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the method provided in this embodiment.

[0070] Example 4 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, will cause the processor to execute the method described above in this embodiment.

[0071] Example 5 This embodiment provides a computer program product including executable instructions, which are computer instructions; the executable instructions are stored in a computer-readable storage medium. When the processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method described in this embodiment.

[0072] The steps and methods involved in Embodiments 2 to 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0073] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0074] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention has been described in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. A path planning method for tunnel construction robots based on deep reinforcement learning, characterized in that, include: The state space and action space of the robot path planning in the tunnel construction environment are predefined, and a dynamic reward function is built. A deep reinforcement learning architecture based on a lightweight action network and a comment network is constructed. The goal is to maximize the sum of the cumulative reward and the entropy regularization term of the policy. The action-comment network is trained using state-action pairs from the experience pool. Specifically, the action network generates a motion policy based on the robot's current state information to output the robot's next action command. This action command is then constrained by a differentiable safety constraint module introduced at the action network's output, resulting in a safe action command that is executed. The comment network calculates the objective function and updates the action-comment network parameters based on the input robot state-action pairs. Based on the current state information of the tunnel construction robot, the optimal movement action for the next step is obtained using a trained motion network, thereby realizing the robot's path planning.

2. The path planning method for tunnel construction robots based on deep reinforcement learning as described in claim 1, characterized in that, The state information of the tunnel construction robot includes positioning state information, environmental perception state information, and task state information; the actions of the tunnel construction robot include the three-dimensional components of its movement speed. The positioning status information includes three-dimensional position coordinates and three-axis orientation; the environmental perception status information includes local obstacle distance information, dynamic obstacle coordinates, and relative velocity; and the task status includes the absolute coordinates of the target point.

3. The path planning method for tunnel construction robots based on deep reinforcement learning as described in claim 1, characterized in that, The dynamic reward function is a total reward function composed of a weighted sum of multiple sub-reward functions, including: target approach reward function, progress reward function, collision penalty function, energy consumption reward function, motion smoothness reward function, and terrain elevation reward function.

4. The path planning method for tunnel construction robots based on deep reinforcement learning as described in claim 1, characterized in that, Training is performed by maximizing an objective function, which is the sum of the standard expected cumulative reward and the policy's entropy regularization term, expressed as: ； in, For strategy The state below -action Access distribution; This is a discount factor used to balance the importance of current rewards and future rewards; Representation Strategy In state Entropy is used to quantify the randomness and exploratory ability of a strategy; This represents the entropy coefficient, used to balance the importance of reward maximization and policy entropy. This is the total reward function.

5. The path planning method for tunnel construction robots based on deep reinforcement learning as described in claim 1, characterized in that, The lightweight action network is used to map high-dimensional states into continuous action commands. The network adopts a parallel hierarchical feature extraction and fully connected layer structure. For different state information in the input state vector, the localization state information and task state information are directly input to the fully connected layer. The local obstacle distance information in the environmental perception state information is extracted through a one-dimensional depthwise separable convolution module. The dynamic obstacle information is input to the fully connected layer. The features extracted from each path are concatenated and then fused and decided by the fully connected layer. Finally, the normalized action command is output through the output layer. The safety constraint module is used to convert the original actions output by the action network. Limited to based on the current perception state Calculated safety boundary within, that is The clip function is used to restrict the elements in an array to between the minimum and maximum values.

6. The path planning method for tunnel construction robots based on deep reinforcement learning as described in claim 1, characterized in that, Configure the first and second online comment networks and their corresponding target comment networks; The first and second online comment networks adopt a parameter-sharing design, sharing the input layer and the first hidden layer, and then branching into two independent lightweight evaluation heads. The lightweight evaluation heads are used to concatenate the shared backbone output features with the action vector and map them to Q-values. The structure of the target comment network is consistent with that of the online comment network, and the network parameters are synchronized through periodic soft updates based on the online comment network.

7. A path planning system for a tunnel construction robot based on deep reinforcement learning, characterized in that, include: A deep reinforcement learning algorithm building module is used to predefine the state space and action space of robot path planning in a tunnel construction environment, and to build a dynamic reward function; The deep reinforcement learning module is used to construct a deep reinforcement learning architecture based on a lightweight action network and a comment network. The goal is to maximize the sum of the cumulative reward and the entropy regularization term of the policy. The action-comment network is trained based on state-action pairs from the experience pool. Specifically, the action network generates a motion policy based on the robot's current state information to output the robot's next action command. This action command is then constrained by a differentiable safety constraint module introduced at the action network's output, resulting in a safe action command that is executed. The comment network calculates the objective function and updates the action-comment network parameters based on the input robot state-action pairs. The path planning module is used to obtain the optimal next movement action based on the current state information of the tunnel construction robot and the trained motion network, thereby realizing the robot's path planning.

8. An electronic device, characterized in that, include: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the path planning method for tunnel construction robots based on deep reinforcement learning as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The device stores executable instructions that, when executed by a processor, implement the path planning method for a tunnel construction robot based on deep reinforcement learning as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the path planning method for tunnel construction robots based on deep reinforcement learning as described in any one of claims 1-6.

Images