A navigation control method, device and medium for an underground inspection robot
By combining a reinforcement learning policy network and an adaptive Gaussian noise navigation control method with Tanh policy regularization and lightweight continuous learning, the problem of unstable navigation of underground inspection robots was solved, achieving stable and efficient navigation in complex environments and improving exploration efficiency and safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 山东浪潮智慧建筑科技有限公司
- Filing Date
- 2025-09-30
- Publication Date
- 2026-07-24
AI Technical Summary
Underground inspection robots suffer from inaccurate positioning, weak communication signals, monotonous environmental structures, and uneven lighting in GPS-denied scenarios, leading to instability of traditional navigation systems. Navigation schemes based on deep learning are also unstable in training and lack policy generalization ability.
A reinforcement learning policy network is adopted, which combines adaptive Gaussian noise and Tanh policy regularization. Navigation control is performed through an end-to-end deep reinforcement learning control architecture, and online incremental updates are performed using a lightweight continuous learning deployment architecture. This solves the problems of poor model generalization and performance degradation after deployment caused by the difference between simulation and reality.
It achieves stability and safety in navigation in complex underground environments, avoids the inaccurate positioning and navigation instability caused by environmental changes in traditional methods, improves exploration efficiency and safety, solves the problem of poor model generalization, and can achieve adaptive optimization without relying on the cloud.
Smart Images
Figure CN121300356B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous navigation and control systems for robots, and in particular to a navigation control method, device and medium for underground inspection robots. Background Technology
[0002] Autonomous navigation of inspection robots in underground spaces (such as underground utility tunnels, mine tunnels, and underground parking lots) faces several unique challenges: First, underground environments are GPS-rejected scenarios, and traditional positioning-based navigation methods are prone to inaccurate positioning; second, communication signals are weak, and robots cannot rely on external remote control or cloud computing; third, the environmental structure is repetitive and monotonous, lacking unique features, and the low texture of walls and floors and uneven lighting can easily interfere with traditional navigation systems based on feature matching or mapping positioning.
[0003] Existing navigation solutions are hierarchical control systems that pre-map, locate, and plan paths. In the above scenarios, their robustness is significantly reduced. Sensor noise, changes in lighting, or environmental symmetry can all cause positioning failures and path planning errors, resulting in unstable navigation and making it difficult to reliably complete inspection tasks.
[0004] In addition, conventional algorithms (such as DDPG and TD3) based on deep learning have problems such as unstable training, large sample requirements, and insufficient policy generalization ability in continuous control tasks such as robot navigation. Furthermore, training often depends on the simulation environment, and the difference between reality and simulation will lead to performance degradation after deployment. Summary of the Invention
[0005] This application provides a navigation control method, device, and medium for underground inspection robots, which solves the problem of unstable navigation control of existing underground inspection robots due to monotonous environment and poor signal.
[0006] The embodiments of this application adopt the following technical solutions: On one hand, this application provides a navigation control method for an underground inspection robot. The method includes: acquiring perception data from a preset simulation environment and inputting the perception data into a preset reinforcement learning policy network to obtain original action commands; constructing the reinforcement learning policy network based on a dual-delay deep deterministic policy gradient algorithm, including an Actor policy network and a dual-Critic evaluation network; adding adaptive Gaussian noise to the original action commands and performing amplitude clipping on the noise-added original action commands to obtain actual action commands; executing the actual action commands, acquiring simulation environment feedback values, and storing the simulation environment feedback values, perception data, and actual action commands in an experience playback buffer; sampling data from the experience playback buffer according to a preset period to update the reinforcement learning policy network, obtaining a converged policy model, and solidifying the policy model parameters; deploying the solidified policy model to an inspection robot in an actual underground space, and incrementally updating the policy model online by collecting perception data from the actual underground space; controlling the robot to perform inspection tasks according to the updated policy model, repeatedly iterating incrementally to achieve continuous optimization of the robot's navigation strategy.
[0007] In one example, data is sampled from the experience replay buffer according to a preset period to update the reinforcement learning policy network and obtain a converged policy model. Specifically, this includes: updating the target network corresponding to the dual Critic evaluation network and the Actor policy network through a soft update mechanism based on the sampled data to obtain a converged policy model; when updating the Actor policy network, a regularization penalty term and weights of the Tanh output are introduced into the policy network loss function, and the restricted output is rescaled through a linear scaling mapping.
[0008] In one example, after deploying the fixed strategy model to an inspection robot in an actual underground space, the method further includes: acquiring real perception data when the robot performs inspection tasks in the actual underground space; reasoning about the real perception data based on the fixed strategy model to obtain real original action commands; performing amplitude clipping on the real original action commands to obtain real actual action commands to drive the robot to perform actions; recording the real environmental feedback value after the robot performs actions, and storing the real environmental feedback value, real perception data, and real actual action commands in an experience playback buffer.
[0009] In one example, the strategy model is updated online incrementally by collecting perception data from the actual underground space. Specifically, this includes: sampling data stored in the experience playback buffer when the robot is charging, waiting for the elevator, and completing a single inspection; updating the dual Critic evaluation network and Actor policy network based on the sampled data, calculating the average parameters during the update process using online random weight averaging technology; and integrating the updated network parameters into the deployed fixed strategy model through a soft update mechanism.
[0010] In one example, the method also includes: the experience replay buffer uses a FIFO storage mechanism, and when the data volume reaches the capacity limit, it automatically deletes the earliest stored preset percentage of data.
[0011] In one example, before executing the actual action command and obtaining the simulation environment feedback value, the method also includes: defining a navigation reward function for training the simulation environment; the navigation reward function includes a positive reward for advancing a preset distance, a negative reward for colliding or deviating from the channel, and an additional reward for completing the inspection route.
[0012] In one example, the amplitude of the original motion command after adding noise is clipped to obtain the actual motion command. Specifically, the clip function is used to clip the original motion command after adding noise to obtain the actual motion command.
[0013] In one example, adaptive Gaussian noise is added to the original action instructions. Specifically, the variance of the adaptive Gaussian noise is adaptively adjusted according to the training process. The adaptive adjustment includes setting the variance to the preset variance when the fluctuation amplitude of the actual action is greater than or equal to a preset threshold from the initial stage of training. When the fluctuation amplitude of the actual action is less than the preset threshold, the preset variance is weakened according to the preset weakening magnitude.
[0014] On the other hand, embodiments of this application provide a navigation control device for an underground inspection robot, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned navigation control methods for an underground inspection robot.
[0015] On the other hand, embodiments of this application provide a non-volatile computer storage medium for navigation control of underground inspection robots, which stores computer-executable instructions that can execute any of the above-mentioned navigation control methods for underground inspection robots.
[0016] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: This application employs an end-to-end deep reinforcement learning control architecture (directly mapping sensor inputs to motion control commands, abandoning the traditional staged processing flow) to avoid error accumulation in intermediate stages and solve the problems of inaccurate positioning and unstable navigation in the face of environmental changes in traditional methods. Through an adaptive exploration mechanism (injecting optimized Gaussian noise into the policy network output, adaptively adjusting the noise variance according to training progress, and pruning the motion amplitude), sufficient and safe exploration is achieved in the early stages of training, preventing the robot from getting stuck in place or losing control, improving exploration efficiency and safety, and laying the foundation for subsequent policy optimization. The Tanh policy regularization mechanism prevents gradient vanishing due to policy output saturation, avoids the Actor network from getting stuck in the dead zone of uncorrectable extreme output values, and ensures motion diversity, overcoming the problem of training divergence in complex underground scenarios. By adopting a lightweight continuous learning deployment architecture (after the robot is deployed, the experience playback mechanism is used to store new environment interaction data, samples are periodically extracted for incremental training, and online random weight averaging and soft update mechanism are combined to smooth parameter updates), the effect of online adaptation can be achieved at the edge without relying on the cloud. This solves the problems of poor model generalization and performance degradation after deployment caused by the difference between simulation and reality, without the need for manual recalibration or frequent software upgrades. Attached Figure Description
[0017] To more clearly illustrate the technical solution of this application, some embodiments of this application will be described in detail below with reference to the accompanying drawings, in which: Figure 1 A flowchart illustrating a navigation control method for an underground inspection robot provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the stage execution steps of a navigation control method for an underground inspection robot provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a navigation and control device for an underground inspection robot, provided as an embodiment of this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0020] Figure 1This is a flowchart illustrating a navigation control method for an underground inspection robot, provided as an embodiment of this application. This method can be applied to various business domains. Certain input parameters or intermediate results in this process can be manually adjusted to help improve accuracy.
[0021] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.
[0022] Based on this Figure 1 The process may include the following steps: S101: Acquire perception data of a preset simulation environment and input the perception data into a preset reinforcement learning policy network to obtain original action instructions; the reinforcement learning policy network is constructed according to the dual-delay deep deterministic policy gradient algorithm and includes an Actor policy network and a dual Critic evaluation network.
[0023] It should be noted beforehand that the underground inspection robot is equipped with a depth camera, LiDAR, and inertial measurement unit. It needs to autonomously inspect underground utility tunnels that stretch for several kilometers, traversing narrow passages and checking the status of pipelines and equipment at fixed points. The internal environment of the utility tunnels has typical characteristics of underground spaces: narrow and winding passages, uneven and persistently dark lighting, simple and repetitive textures on the walls and floors, and numerous communication blind spots with no wireless signal.
[0024] In some embodiments of this application, in order to enable the underground inspection robot to work better, this application first requires offline training of the underground inspection robot, that is, training in a simulation environment.
[0025] First, a simulation environment and navigation reward function tailored to the characteristics of underground space are constructed, including straight and curved passages, different lighting and texture maps, and randomly placed obstacles (simulating falling debris, etc.). A reinforcement learning reward function is designed to encourage the robot to travel along the passages, avoid collisions, and reach predetermined inspection points in a timely manner. For example, a positive reward of 10-20 is given for every meter of progress, a negative reward of -50 to -100 is given for collisions or deviations from the passage (deviation distance > 0.5 meters), and an additional reward of 200-500 is given for reaching the preset inspection point, guiding the robot to prioritize completing the inspection objectives. The robot is repeatedly trained in the simulation environment using the algorithm of this invention: initially, due to the lack of learned effective strategies, the robot may exhibit behaviors such as hitting walls or spinning in place, but through an adaptive exploration mechanism, the algorithm gradually guides the robot to try various actions (turning, decelerating to avoid obstacles, etc.).
[0026] Furthermore, the underground inspection robot is placed in a simulated environment, and its sensors acquire perception data of the simulated environment. Sensing data This corresponds to camera images, lidar point clouds, and inertial measurement data in the simulation environment.
[0027] Furthermore, the sensing data Input a pre-defined reinforcement learning policy network, and output the original action through the Actor policy network. .
[0028] S102: Add adaptive Gaussian noise to the original motion command, and perform amplitude clipping on the original motion command after adding noise to obtain the actual motion command.
[0029] In some embodiments of this application, after obtaining the original action After that, Inject adaptive random perturbation on the basis The actual animation is obtained after clipping using the clip function. ,in These represent the upper and lower limits of robot motion control commands.
[0030] It should be noted that the above Zero-mean Gaussian noise ~N (0, ), Adaptive adjustments based on training progress, during the initial training phase. A value of 0.8-1.2 is used to encourage exploration. When the fluctuation range of the policy network's output action is less than a preset threshold (0.1-0.2), Gradually reduce the value to 0.1-0.3 to avoid the robot spinning in place or going out of control due to unconstrained exploration.
[0031] By employing an adaptive exploration mechanism (injecting optimized Gaussian noise into the policy network output, adaptively adjusting the noise variance based on training progress, and pruning the motion amplitude), sufficient and safe exploration is achieved in the early stages of training, preventing the robot from getting stuck in place or going out of control, improving exploration efficiency and safety, and laying the foundation for subsequent policy optimization.
[0032] S103: Execute the actual action command, obtain the simulation environment feedback value, and store the simulation environment feedback value, perception data, and actual action command into the experience playback buffer.
[0033] In some embodiments of this application, after determining the actual action Then, control the robot to perform actions. Obtain reward values from environmental feedback. It also includes the next sensor status. ,Will( , , , Store in the experience replay buffer.
[0034] It should be noted that the experience playback buffer uses a first-in, first-out (FIFO) storage mechanism, and the storage capacity of a single buffer is set to [value missing]. The system automatically deletes the oldest 10%-20% of the empirical data when the data volume reaches the capacity limit, ensuring the timeliness of the empirical data.
[0035] S104: According to a preset period, data is sampled from the experience replay buffer to update the reinforcement learning policy network, obtain a converged policy model, and solidify the policy model parameters.
[0036] In some embodiments of this application, ( , , , After being stored in the experience replay buffer, data is sampled from the experience replay buffer according to the preset sampling period. The target network corresponding to the dual Critic evaluation network and the Actor policy network is updated using a soft update mechanism to obtain the converged initial policy model and solidify the parameters of the initial policy model.
[0037] It should be noted that updating the Actor policy network involves introducing a Tanh-based output regularization term into the policy network loss function. With weight Simultaneously, the restricted output is rescaled using a linear scaling mapping, and the loss function is... ,in =tanh(u), where u is the original output of the Actor network before Tanh activation.
[0038] Tanh's output regularization terms The specific definition is: when hour, =|u|- ;when hour, =0; where The linear scaling mapping takes values from 1 to 3 and is achieved through a function. This implements mapping of the restricted output to the [-1,1] interval to cover the full range of motion.
[0039] The coefficient τ of the soft update mechanism ranges from 0.001 to 0.01, and the target network parameter update formula is as follows: ,in The new network parameters obtained from the current training, The target network parameters are used to avoid fluctuations in navigation behavior caused by sudden changes in parameters.
[0040] As training progresses, the Actor-Critic network is continuously updated. Combined with Tanh regularization to maintain gradient stability, the robot quickly learns basic steering, obstacle avoidance, and straight-line path following. After numerous iterations, the policy network converges to a reliable navigation strategy: the robot can determine the channel direction end-to-end from sensor data and smoothly control steering and speed within the simulated pipe gallery, avoiding obstacles and covering all inspection points. At this point, the trained neural network model parameters are solidified and deployed as the initial strategy onto the actual robot.
[0041] It should also be noted that the termination condition for offline training is that, within 500-1000 consecutive training cycles, the robot's average inspection completion rate in the simulation environment is ≥95%, the average number of collisions is ≤1, and the variance of the policy network's output actions is stable within the range of 0.05-0.1, ensuring the reliability of the initial policy model.
[0042] By using the Tanh policy regularization mechanism, gradient vanishing caused by policy output saturation is prevented, and the Actor network is prevented from falling into the dead zone where extreme output values cannot be corrected. At the same time, action diversity is guaranteed, and the problem of training divergence in complex underground scenarios is overcome.
[0043] S105: An inspection robot that deploys a fixed strategy model to the actual underground space and performs online incremental updates to the strategy model by collecting perception data from the actual underground space.
[0044] The real-world environment differs from the training environment. For example, the ground friction coefficient, turning slope, and sensor noise caused by humidity in the air in a real utility tunnel can all affect the robot's perception and control accuracy. To address these subtle differences, the system of this invention initiates an online learning mode: while performing inspection tasks, the robot continuously stores sensor readings and action results into an experience playback buffer. When it finds that the strategy performance is unsatisfactory in certain situations (e.g., excessive deceleration in a dark passage or increased braking distance on a slippery surface), the system uses this newly collected environmental data to incrementally train the strategy. Specifically, during idle time slices (e.g., the robot waiting for an elevator or charging), it samples recent experiences from the playback buffer and performs several steps of learning and updating the algorithm of this invention (including Critic evaluation value correction and Actor policy fine-tuning). After each update, the improved parameters are smoothly integrated into the current deployment model through soft updates. At the same time, online random weights ensure that even if a single update is limited by small samples or noise, the averaged strategy remains stable and reliable.
[0045] In some embodiments of this application, after obtaining the policy model through offline training, the policy model is deployed to an inspection robot in an actual underground space. After deployment, when the robot performs inspection tasks in the actual underground space, it acquires real perception data through a depth camera, LiDAR, and inertial measurement unit. Specifically, the images acquired by the depth camera are preprocessed by dehazing and illumination equalization, the LiDAR point cloud is denoised and ground point removed, and the preprocessed image features and point cloud features are fused with the angular velocity and acceleration data of the IMU.
[0046] Furthermore, based on the fixed strategy model, reasoning is performed on the real perceived data to obtain the real original action commands. .
[0047] Furthermore, the original motion commands are clipped using the clip function to obtain the actual motion commands, which drive the robot to perform movement and turning actions.
[0048] Record the real-world feedback values after the robot performs its actions. Including the next real sensor state ,Will( , , , Write to the robot's local edge experience playback buffer.
[0049] Furthermore, without affecting real-time inspections, periodic online incremental updates are performed: during gaps such as robot charging, waiting for elevators, or completing a single inspection, recently stored data is sampled from the experience replay buffer. Then, the dual-Critic evaluation network and Actor policy network are updated based on the sampled data. During the update process, an online random weight averaging technique is used, according to the formula... Calculate the average parameter ,in These are the model parameters after the k-th update. Finally, a soft update mechanism is used to smoothly integrate the updated network parameters into the currently deployed policy model.
[0050] By adopting a lightweight continuous learning deployment architecture, we can achieve online adaptive performance at the edge without relying on the cloud, thus solving the problems of poor model generalization and performance degradation after deployment caused by the difference between simulation and reality.
[0051] S106: Control the robot to perform inspection tasks according to the updated strategy model, and repeatedly iterate incrementally to achieve continuous optimization of the robot navigation strategy.
[0052] In practice, after several days of online operation and adaptation, the robot's navigation strategy was further optimized. For example, the robot gradually learned to adjust its speed according to real-time lighting: it can move steadily forward using LiDAR even in completely dark areas, and it does not excessively decelerate or stop in areas with alternating light and dark conditions due to light interference. In addition, for certain specific corners and ramps (not fully covered in the simulation), the robot found smoother turning and power distribution strategies through multiple trials, eliminating the occasional sudden stops or skidding that occurred during initial deployment. Throughout the process, no human intervention was required; the robot autonomously used data from the inspection process to continuously correct itself. This ability to learn while executing means that even if the internal environment of the utility tunnel changes (such as the addition of new equipment obstacles, terrain changes, or changes in landmarks due to long-term wear and tear), the robot can adapt on its own, maintaining a high level of navigation and inspection efficiency.
[0053] In some embodiments of this application, as the strategy is continuously updated, the robot continues to be controlled to perform inspection tasks based on the updated strategy model. The steps of repeatedly performing online incremental updates to the strategy model are repeated to achieve continuous optimization of the robot navigation strategy.
[0054] It should be noted that, although the embodiments in this application are based on... Figure 1 Steps S101 to S106 will be described sequentially, but this does not mean that steps S101 and S106 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1 The order in which steps S101 to S106 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S101 to S106 can be appropriately adjusted according to actual needs.
[0055] pass Figure 1This application employs an end-to-end deep reinforcement learning control architecture (directly mapping sensor inputs to motion control commands, abandoning the traditional staged processing flow) to avoid error accumulation in intermediate stages and solve the problems of inaccurate positioning and unstable navigation in the face of environmental changes in traditional methods. Through an adaptive exploration mechanism (injecting optimized Gaussian noise into the policy network output, adaptively adjusting the noise variance according to training progress, and pruning the motion amplitude), sufficient and safe exploration is achieved in the early stages of training, preventing the robot from getting stuck in place or losing control, improving exploration efficiency and safety, and laying the foundation for subsequent policy optimization. The Tanh policy regularization mechanism prevents gradient vanishing due to policy output saturation, avoids the Actor network from getting stuck in the dead zone of uncorrectable extreme output values, and ensures motion diversity, overcoming the problem of training divergence in complex underground scenarios. By adopting a lightweight continuous learning deployment architecture (after the robot is deployed, the experience playback mechanism is used to store new environment interaction data, samples are periodically extracted for incremental training, and online random weight averaging and soft update mechanism are combined to smooth parameter updates), the effect of online adaptation can be achieved at the edge without relying on the cloud. This solves the problems of poor model generalization and performance degradation after deployment caused by the difference between simulation and reality, without the need for manual recalibration or frequent software upgrades.
[0056] Figure 2 This is a schematic diagram illustrating the stage execution steps of a navigation control method for an underground inspection robot provided in an embodiment of this application.
[0057] exist Figure 2 The document illustrates two phases of this application: an offline training phase and a deployment and continuous learning phase. The policy model or parameters obtained in the offline training phase are used to deploy in a real underground environment for continuous learning during real-world inspections.
[0058] Figure 3 A schematic diagram of a navigation and control device for an underground inspection robot, provided in an embodiment of this application, includes: At least one processor; and, A memory that is communicatively connected to at least one processor; wherein, A navigation control method for an underground inspection robot, wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable at least one processor to perform any of the above-mentioned actions.
[0059] Some embodiments of this application provide a non-volatile computer storage medium for navigation control of an underground inspection robot, which stores computer-executable instructions that can execute any of the above-mentioned navigation control methods for an underground inspection robot.
[0060] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.
[0061] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.
[0062] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0063] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0066] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0067] Memory may include non-persistent storage in computer-readable media, random access memory (RAM), and non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0068] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0069] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0070] The above are merely embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the technical principles of this application should fall within the protection scope of this application.
Claims
1. A navigation and control method for an underground inspection robot, characterized in that, The method includes: The system acquires perception data from a preset simulation environment and inputs the perception data into a preset reinforcement learning policy network to obtain original action commands. The reinforcement learning policy network is constructed based on a dual-delay deep deterministic policy gradient algorithm and includes an Actor policy network and a dual Critic evaluation network. Adaptive Gaussian noise is added to the original motion command, and the amplitude of the original motion command after adding noise is clipped to obtain the actual motion command. Execute the actual action command, obtain the simulation environment feedback value, and store the simulation environment feedback value, perception data, and actual action command into the experience playback buffer. According to a preset period, data is sampled from the experience replay buffer to update the reinforcement learning policy network, obtain a converged policy model, and solidify the policy model parameters. The fixed strategy model is deployed to the inspection robot in the actual underground space, and the strategy model is updated online incrementally by collecting perception data of the actual underground space. The robot is controlled to perform inspection tasks based on the updated strategy model, and incremental updates are repeatedly iterated to achieve continuous optimization of the robot's navigation strategy. After deploying the solidified strategy model to the inspection robot in the actual underground space, the method further includes: When the robot performs inspection tasks in actual underground spaces, it acquires real-world perception data. Based on the fixed strategy model, reasoning is performed on the real perceived data to obtain the real original action instructions; The amplitude of the original motion command is clipped to obtain the actual motion command, which drives the robot to perform the action. Record the real environmental feedback values after the robot performs the action, and store the real environmental feedback values, real perception data and real actual action commands into the experience playback buffer. The process of collecting sensor data from actual underground space and incrementally updating the strategy model online specifically includes: Data stored in the sampling experience replay buffer is used when the robot is charging, waiting for the elevator, or completing a single inspection. The dual Critic evaluation network and Actor policy network are updated based on the sampled data. During the update process, the average parameters are calculated using an online random weight averaging technique. The updated network parameters are incorporated into the deployed fixed strategy model through a soft update mechanism.
2. The method according to claim 1, characterized in that, The step of sampling data in the experience replay buffer according to a preset period to update the reinforcement learning policy network and obtain a converged policy model specifically includes: Based on the sampled data, the target networks corresponding to the dual Critic evaluation network and the Actor policy network are updated through a soft update mechanism to obtain a converged policy model. When updating the Actor policy network, a regularization penalty term and weights of the Tanh output are introduced into the policy network loss function, and the restricted output is rescaled through a linear scaling mapping.
3. The method according to claim 1, characterized in that, The method further includes: The experience playback buffer adopts a FIFO storage mechanism. When the data volume reaches the capacity limit, the earliest stored preset percentage of data is automatically deleted.
4. The method according to claim 1, characterized in that, Before executing the actual action command and obtaining the simulation environment feedback value, the method further includes: Define a navigation reward function for training in a simulation environment; the navigation reward function includes a positive reward for advancing a preset distance, a negative reward for colliding with or deviating from the channel, and an additional reward for completing the inspection route.
5. The method according to claim 1, characterized in that, The original motion commands with added noise are subjected to amplitude clipping to obtain the actual motion commands, specifically including: The original motion commands with added noise are clipped using the clip function to obtain the actual motion commands.
6. The method according to claim 1, characterized in that, Adding adaptive Gaussian noise to the original action command specifically includes: The variance of the adaptive Gaussian noise is adaptively adjusted according to the training process; the adaptive adjustment includes setting the variance to the preset variance when the fluctuation amplitude of the actual action is greater than or equal to the preset threshold from the initial stage of training, and reducing the preset variance according to the preset reduction amplitude when the fluctuation amplitude of the actual action is less than the preset threshold.
7. A navigation and control device for an underground inspection robot, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a navigation control method for an underground inspection robot as described in any one of claims 1-6.
8. A navigation control storage medium for an underground inspection robot, storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of executing the navigation control method for an underground inspection robot as described in any one of claims 1-6.