A mobile robot gas hazard source autonomous tracing method based on deep reinforcement learning

Abstract

The application belongs to the technical field of reinforcement learning, and discloses a mobile robot gas hazard source autonomous tracing method based on deep reinforcement learning. An intelligent decision-making framework only relying on olfactory information and ranging data is constructed, including a high-fidelity gas diffusion simulation environment construction module, a decision-reward network based on proximal policy optimization, and a multi-level reward guiding mechanism. The high-fidelity gas diffusion simulation environment construction module is constructed based on the convection-diffusion equation to generate training data containing obstacle distribution and dynamic concentration field evolution. The multi-level reward guiding mechanism fuses physical concentration gradient signals, safety obstacle avoidance constraints and exploration incentives to guide the robot to gradually evolve from random walk to a search strategy with "olfactory intelligence". The decision-reward network based on proximal policy optimization adopts a double network architecture to process high-dimensional spatiotemporal state input and output continuous motion control instructions.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to a method for autonomous tracing of gas hazard sources by mobile robots based on deep reinforcement learning. Background Technology

[0002] In complex industrial settings such as modern chemical plants, nuclear power plants, underground mines, and urban pipe networks, leaks of toxic, hazardous, flammable, and explosive gases pose a significant threat to production safety and human lives. If a leak occurs and the source cannot be detected and accurately located immediately, it can easily lead to catastrophic consequences such as fires, explosions, or large-scale poisoning. Therefore, establishing a gas leak monitoring and location system that is capable of rapid response, accurate source tracing, and strong adaptability is crucial for ensuring safe industrial production and reducing accident losses, and it is also a core issue that urgently needs to be addressed in the field of industrial safety.

[0003] However, locating gas leak sources in real-world industrial applications presents significant environmental challenges. These locations are typically highly complex, often featuring a dense network of crisscrossing pipelines, massive storage tanks, and irregular building structures. These physical obstacles not only create intricate navigational mazes, obstructing visibility, but also significantly disrupt airflow, resulting in turbulent, discontinuous, and nonlinear gas concentration distributions. This high dynamism and uncertainty necessitates source tracing systems with exceptional environmental adaptability and flexible obstacle avoidance capabilities.

[0004] Traditional industrial gas monitoring methods primarily rely on manual inspections and fixed sensor networks. Manual inspections require personnel to use portable detectors to conduct thorough, door-to-door searches of hazardous areas. This method is not only labor-intensive and inefficient, but also fails to provide 24 / 7 real-time coverage. More importantly, it exposes personnel directly to hazardous environments, posing a significant risk to their personal safety. While fixed sensor networks can provide continuous monitoring at specific locations, their coverage is limited by the placement and number of sensors, easily creating blind spots in complex equipment environments and making it difficult to respond to sudden and unknown leaks.

[0005] With the development of robotics technology, using mobile robots equipped with gas sensors to replace humans for inspections has become a trend. However, existing automated inspection robot technologies still have limitations. Most mainstream solutions on the market rely on pre-built high-precision maps and preset fixed navigation paths, or employ triggering mechanisms based on simple concentration thresholds. When performing tasks, the robots typically just mechanically travel along the set route, triggering an alarm or stopping when they detect excessive concentration. These robots generally lack intelligent environmental perception and decision-making capabilities, mostly employing a single concentration gradient climbing algorithm, which cannot effectively combine environmental geometric information.

[0006] In summary, existing gas tracing technologies often fall short in the face of complex industrial environments riddled with obstacles. Traditional robots, unlike experienced experts, cannot simultaneously sense changes in gas concentration and use vision or other means to assess the surrounding environment and flexibly plan the optimal path to bypass obstacles and approach the leak source. This current state of technology, characterized by "single perception and rigid decision-making," results in low success rates and long processing times in complex scenarios, failing to meet the urgent needs of industrial sites for proactive early warning and rapid emergency response.

[0007] Despite the progress made by industry and academia over the past few decades in gas sensing technology, wireless sensor networks, and olfactory systems for mobile robots, existing technologies still exhibit systemic defects and shortcomings when facing the triple challenges of unstructured environments, strong dynamic interference, and harsh sensing conditions in real-world industrial scenarios. Traditional monitoring methods are plagued by low spatiotemporal coverage, slow response times, and high false alarm rates, while emerging robotic inspection technologies are limited by the singularity of their sensing modes, the rigidity of their path planning, and the inefficiency of their search algorithms in turbulent environments, making it difficult to truly achieve proactive and accurate source tracing in all weather conditions and all regions.

[0008] (1) The spatiotemporal limitations and maintenance challenges of manual inspection and fixed sensor networks; For a long time, gas leak monitoring in industrial sites has mainly relied on a dual approach combining fixed sensor networks and manual inspections. However, this passive or semi-active monitoring method has inherent defects that are difficult to overcome when facing the complex and ever-changing modern industrial environment. These defects are mainly reflected in blind spots in monitoring coverage, the degradation of sensor performance over time, and the high risk and low efficiency of manual operation.

[0009] First, while fixed sensor networks can continuously monitor specific locations, they are essentially discrete "point" sampling, unable to achieve full coverage in physical space. As analyzed in detail in "Fixed or portable gas detectors: What does your facility need?", fixed gas detectors are typically installed at predetermined key locations, and their number is often limited. This discrete layout easily leads to monitoring blind spots in complex industrial equipment. Second, the long-term reliability of sensors is another major technical bottleneck. "A Review of Gas Measurement Practices and Sensors for Tunnels" discusses in detail the performance degradation mechanisms of various commercial gas sensors in practical applications. The study points out that all types of sensors are inevitably significantly affected by environmental conditions; changes in temperature and humidity, as well as the long-term presence of background interfering gases, can cause severe zero-point drift and decreased sensitivity in sensor readings.

[0010] In the context of the failure of fixed monitoring, manual inspection is seen as a necessary supplementary means, but its limitations cannot be ignored. "Lessons in gas detection: fixed vs. personal monitors," through a retrospective analysis of several major industrial explosion accidents, profoundly reveals the risks of relying solely on portable monitors. This study emphasizes that portable devices are often "reactive" tools, only triggering an alarm when operators have already entered the danger zone and are directly exposed to high concentrations of gas. This lag means that by the time personnel receive a warning, they are often already in extreme danger. "From Manual Inspections to Automation: The Future of Oil and Gas Monitoring" further criticizes the manual inspection model from the perspective of data continuity, pointing out that manual inspection is inherently intermittent and cannot provide a continuous data stream. Furthermore, for the detection of special gases in complex scenarios, "Key Factors Involved in Pipeline Monitoring Techniques Using Robots and WSNs: Comprehensive Survey" points out that manual inspection is not only extremely inefficient, but also completely ineffective with human senses when dealing with colorless, odorless gases or gases that are highly toxic even at low concentrations, necessitating reliance on handheld devices. The report "Minimizing Industrial Gas Safety Risks With Smart Technology" points out from an emergency response perspective that the data silos created by manual inspections and fixed monitoring make it difficult to achieve cross-regional joint early warning and coordinated response in emergency situations. In summary, the spatial dispersion and high maintenance requirements of fixed sensor networks, coupled with the time lag and high personal risks of manual inspections, make existing traditional monitoring systems inadequate in the face of the complex, hidden, and sudden gas leak threats of modern industry.

[0011] (2) The rigidity and lack of adaptability of existing robot path planning in unstructured environments; To overcome the limitations of traditional monitoring methods, mobile robot technology has been introduced into the field of gas source tracing, attempting to expand the sensing range of sensors through the mobility of robots. However, existing robot path planning technologies generally exhibit strong rigidity, and are mostly designed for structured, static environments, making them unsuitable for the typically unstructured, dynamic, and uncertain complex environments of gas leak accident sites.

[0012] To address this issue, "Optimal and Efficient Path Planning for Unknown and Dynamic Environments" points out that traditional replanning strategies often require recalculating the entire optimal path from the current location to the target point based on updated map information when faced with newly encountered obstacles. In scenarios with frequently changing obstacles or extremely complex environments, this high-frequency global replanning consumes enormous computational resources, causing the robot's decision-making frequency to lag behind environmental changes. This computational inefficiency is further amplified in "Path Planning Challenges for Planetary Robots," which indicates that for battery-powered mobile robots with limited computing power, complex replanning algorithms quickly deplete system energy, preventing the robot from sustaining long-term continuous search tasks.

[0013] Furthermore, most existing path planning strategies are based on the geometric shortest path or energy optimality principle, severely lacking a deep understanding of environmental semantics and gas distribution characteristics. "Robot Path Planning in Dynamic Environments: A Review" emphasizes that path planning lacking semantic information can cause robots to mechanically bypass these critical facilities, missing opportunities to detect leaks at close range. Rigidly following preset cruising routes or geometric shortest paths often leads to robots frequently traversing low-concentration or even non-concentration areas, thus losing plume cues.

[0014] Existing algorithms also exhibit significant shortcomings when facing local minima traps. Studies such as "Genetic Algorithm Approach to Resolve the Mobile Robot Path Planning Problem" point out that although local planning algorithms such as the artificial potential field method react quickly, they are prone to getting trapped in local potential energy in obstacle-dense industrial environments. Although evolutionary algorithms attempt to escape local optima through swarm search, their slow convergence speed and high computational cost remain prominent issues in high-dimensional and complex dynamic maps, making it difficult to meet the requirements of real-time obstacle avoidance. "A Comprehensive Review of Coverage Path Planning in Robotics" further points out that when full coverage search of suspected leakage areas is required, most existing coverage path planning algorithms are based on simple reciprocating scanning. This strategy suffers a sharp drop in coverage efficiency when encountering irregular boundaries or dynamic obstacles, and it is prone to leaving blind spots in the search.

[0015] Finally, existing path planning technologies lack the ability to predict and adapt to the movement trends of dynamic obstacles. (See "Improving Path Efficiency and Adaptability in Robot Path Planning with IA") The report points out that in rescue scenarios involving human-machine coexistence or multi-machine collaboration, robots must be able to predict the trajectories of surrounding moving entities and proactively avoid them. In summary, the rigidity of existing robot path planning technologies in their decision-making mechanisms, their lack of adaptability to dynamic environments, and their neglect of task characteristics render them inadequate for the rapid tracing of gas hazard sources in complex industrial scenarios.

[0016] (3) Existing technologies’ over-reliance on visual perception and their failure in extreme environments; In current mainstream mobile robot navigation and environmental perception systems, visual sensors and LiDAR hold an absolute dominant position. This technological approach is based on the ideal assumptions of ample ambient light, high medium transparency, and rich texture features. However, gas leak accident sites often exhibit extreme characteristics such as dense smoke, dust obscuring the view, drastic changes in lighting, or even complete darkness. In these typical environments of visual exclusion or perceptual degradation, the high dependence of existing technologies on visual information becomes a fatal weakness restricting their survival and operational capabilities.

[0017] First, the scattering and absorption of light signals by smoke and aerosol particles drastically degrades the performance of optically based sensors. The paper "4DRT-SLAM: Robust SLAM in Smoke Environments using 4D Radar and Thermal Camera based on Dense Deep Learnt Features" vividly illustrates the consequences of this physical phenomenon: the beams emitted by traditional lidar cannot penetrate dense smoke particles; the beams experience strong diffuse reflection and echo attenuation upon encountering smoke and dust, resulting in sensors receiving a large amount of noisy data. This causes robots to misinterpret floating smoke plumes as hard obstacles, making it extremely difficult for them to move through open passageways.

[0018] Secondly, the vulnerability of visual SLAM systems in feature extraction and matching is glaringly exposed in harsh environments. The paper "Performance Analysis of ORB-SLAM in Foggy Environments" demonstrates through detailed experimental data that the number of key feature points relied upon by classic visual SLAM algorithms decreases sharply under fog or smoke conditions. Once the SLAM system loses localization, the robot cannot determine its position in the environment, ultimately leading to the complete collapse of the navigation system. Furthermore, the paper "Evaluation of Navigation Sensors in Fire Smoke Environments" systematically evaluates the performance of various sensors, including visual cameras, LiDAR, radar, and sonar, under different concentrations of smoke. The results show that when smoke visibility is less than 5 meters, the ranging errors of traditional visible light cameras and near-infrared LiDAR increase exponentially, almost completely losing their environmental perception capabilities.

[0019] (4) The inefficiency and instability of traditional pure olfactory algorithms in complex turbulent environments; At the core algorithm level of gas source tracing, existing technologies mainly rely on search strategies based on pure olfactory information, such as biological chemotaxis and information orientation. However, there is a significant gap between the theoretical assumptions of these algorithms and the physical characteristics of actual industrial environments, resulting in problems such as low search efficiency, severe path oscillations, and a high susceptibility to getting trapped in local optima when dealing with complex airflow environments.

[0020] Traditional chemotaxis algorithms mimic the concentration gradient ascent behavior of bacteria or insects, their core logic assuming that gas concentration exhibits a continuous distribution in space, smoothly decreasing from the source outwards. However, this assumption is almost entirely invalid in real-world fluid dynamic environments. As the classic paper "Tracking scents and locating odor sources is an amazing challenge in robotics" aptly illustrates, in macroscopic industrial environments, gas transport is primarily driven by turbulent advection rather than molecular diffusion. This means that gas plumes are torn apart by turbulent eddies in the environment, interspersed with large areas of odorless, clean air. This highly dynamic and discontinuous signal characteristic renders chemotaxis algorithms based on instantaneous concentration gradients completely ineffective—robots either spin in place due to frequent and drastic changes in the calculated gradient direction, or lose track of their target due to prolonged periods in "no-signal" zones, becoming trapped in endless random walks.

[0021] To address the intermittent signal problem caused by turbulence, researchers have proposed an information-oriented strategy based on Bayesian inference and information entropy, aiming to balance exploration and utilization by maximizing the information gain of each movement. While this algorithm is theoretically elegant, it faces serious robustness issues in practical applications. The paper "Limits on the performance of Infotaxis under inaccurate modelling of the environment" demonstrates through rigorous simulation analysis that the performance of the Infotaxis algorithm is highly dependent on accurate prior modeling of environmental physical parameters. However, in real industrial disaster scenarios, these parameters are unknown and dynamically changing, making accurate prediction extremely difficult and potentially misleading the robot to search in incorrect areas, rendering it even less efficient than a simple random search strategy.

[0022] Furthermore, existing pure olfactory algorithms are inadequate when dealing with multi-source leaks or unsteady leaks. Literature such as "RobotPath Planning in Dynamic Environments: A Review" and "Emergence of active turbulence limits chemotaxis-induced collective condensation" also corroborate this, pointing out that current algorithms lack effective signal separation and source identification mechanisms. When multiple leak points or strong background interference gases exist in the environment, the superimposed concentration fields cause the gradient direction to become completely chaotic, and a single olfactory search logic cannot distinguish the contributions of different sources, causing the robot to fail to lock onto any target. Meanwhile, "Application of anolfactory data preprocessing algorithm to chemotactic robotic navigation" emphasizes the inherent response recovery hysteresis problem of existing gas sensors. This physical limitation further exacerbates the hysteresis effect of algorithms in rapidly changing airflows, making the control loop based on real-time feedback highly prone to oscillation.

[0023] In summary, existing gas hazard source tracing technologies face comprehensive technical bottlenecks in dealing with the complex environments of modern industry, ranging from physical perception and path planning to core search algorithms. Manual inspections and fixed monitoring methods are limited by the discontinuity of spatiotemporal coverage, high maintenance costs, and significant personnel safety risks, making it difficult to build a comprehensive real-time protection network. While existing mobile robot solutions introduce automation capabilities, they lack adaptive mechanisms for unstructured dynamic environments in path planning. Perception systems rely excessively on fragile visual and lidar modes, easily failing in extreme visually limited disaster sites such as smoke and darkness, causing robots to lose basic mobility. Meanwhile, core olfactory search algorithms have long been confined to idealized continuous diffusion models or simple biological reaction rules, resulting in low search efficiency and a high susceptibility to local traps. Therefore, overcoming the limitations of existing technologies and developing a novel tracing technology with robust perception, intelligent path planning capabilities in dynamic environments, and an adaptive anti-turbulence olfactory search strategy has become a critical scientific and engineering problem urgently needing to be solved in this field. Summary of the Invention

[0024] The purpose of this invention is to overcome the over-reliance on visual perception in existing gas tracing robots, and the shortcomings of traditional pure olfactory algorithms, such as low search efficiency and susceptibility to local optima in complex flow fields. This invention provides a mobile robot autonomous gas hazard source tracing method based on deep reinforcement learning, relying solely on olfaction and low-level ranging information. It aims to enable the robot to fully utilize the spatiotemporal characteristics of olfactory signals for intelligent decision-making even in visually limited environments, thereby achieving rapid, robust, and high-precision autonomous odor source tracing.

[0025] The technical solution of the present invention is as follows: a mobile robot autonomous tracing method for gas hazard sources based on deep reinforcement learning, which constructs an intelligent decision-making framework that relies solely on olfactory information and ranging data; the intelligent decision-making framework includes a high-fidelity gas diffusion simulation environment construction module, a decision-reward network based on near-end strategy optimization, and a multi-level reward guidance mechanism; The high-fidelity gas diffusion simulation environment construction module is based on the convection-diffusion equation and generates training data that includes obstacle distribution and dynamic concentration field evolution. The multi-level reward guidance mechanism integrates physical concentration gradient signals, safety obstacle avoidance constraints and exploration incentives to guide the robot to gradually evolve from random walks to a search strategy with "olfactory intelligence". The decision-reward network based on proximal strategy optimization adopts a dual-network architecture to process high-dimensional spatiotemporal state inputs and output continuous motion control commands.

[0026] The high-fidelity gas diffusion simulation environment construction module generates standardized training data containing complex obstacle interference and dynamic wind field effects by numerically solving the two-dimensional convection-diffusion equation, providing olfactory information and a source tracing platform for the intelligent decision-making framework; the two-dimensional convection-diffusion equation is used to describe the spatiotemporal evolution of gas in a confined space, and the continuous form of the governing equation is as follows: (1) in, Represents the gas concentration field; Where is the diffusion coefficient. ; For ambient wind speed vector, ; The injection rate of the leakage source term is used; the finite difference method is used to discretize the control equations in time and space; Diffusion term discretization: The diffusion process is approximated using a second-order central difference scheme with the Laplace operator. in, coordinates The gas concentration at that location; Convection term discretization: The first-order upwind scheme is used to handle the convection term; in, The spatial distance between two environmental points; Time progression: Iterative updates are performed using the explicit Euler method to calculate the next time step. Concentration field: in, That is, the discrete form of the Laplace operator. coordinates Injection rate at the site, This represents the time interval for each iteration; A dynamic hindrance model for gas diffusion by obstacles is introduced; based on the local obstacle density... Dynamically adjust the diffusion coefficient : in, The base diffusion coefficient decreases to [value missing] when the region is an obstacle. Times, simulating gas flow around and retention effects, The environmental medium factor is defined as follows: The environmental medium factor is used to simulate the normal diffusion of gases in free space.

[0027] The decision-reward network based on proximal policy optimization comprises two working networks: a decision network responsible for outputting continuous action policies based on environmental observations, and a reward network responsible for evaluating the expected value of the current state to guide policy updates. The decision network and reward network are both three-layer fully connected neural networks. The parameters of the decision network and reward network are defined as follows: and The forward propagation process is uniformly represented as: in , and , These are the weight matrices and bias vectors for the first two layers of the decision-reward network, respectively. , and , These are the weight matrix and bias vector of the output layer of the decision-reward network, respectively. and These represent the outputs of the decision network and the reward network, respectively. The input layer of the decision network receives a 17-dimensional state vector from the training data generated by the high-fidelity gas diffusion simulation environment building module. Feature extraction is performed through two hidden layers containing 64 neurons each, with hyperbolic tangent activation applied between layers; the output layer is mapped to a two-dimensional vector. This constructs a high-precision continuous motion space, in which the output layer receives the output. The first dimension After activation by the hyperbolic tangent function, it is restricted to... The interval, and used as the mean parameter of the action distribution, for The generation mechanism adopts a "sampling-truncation-mapping" approach: First, random sampling is performed based on a Gaussian distribution strategy to generate normalized actions. : The risk of illegal numerical values ​​caused by the long-tail effect of the Gaussian distribution is eliminated by using a truncation function, and the results are mapped to the physical angle space through a linear transformation: In the formula, The standard deviation is a learnable parameter that decays as training progresses. This is a truncation function; ultimately Mapped as continuous actions , used to control the robot's direction of movement; according to formula (12), the normalized value -1 is mapped to 0 radians, and 0 is mapped to Radians, +1 mapped to 2π radians, eliminates the direction quantization error of traditional discrete motion spaces, enabling the robot to output... An arbitrary floating-point angle within the interval; According to formula (10), the reward function output layer is mapped to a single scalar. The single scalar represents the state value function of the current state, used to calculate the advantage function. This reduces the variance of policy gradient estimation and accelerates training convergence. In the formula, As a discount factor, for The total reward function output at time step 1; the reward network optimizes by minimizing the value prediction error and mean squared error, providing a stable baseline for the decision network.

[0028] The input to the decision-reward network is the 17-dimensional state vector. The training data, derived from the interaction between the robot and the high-fidelity gas diffusion simulation construction module, consists of the following components: 4D position and motion characteristics: including normalized robot physical coordinates and the displacement vector at the previous moment ; 2D olfactory perception features: including the absolute value of gas concentration at the current location. With the change in concentration ; The gradient signal in the total reward function is directly mapped to determine the correctness of the movement direction; 8-dimensional environmental obstacle features: discrete obstacle markers corresponding to 8 directions around the robot; acquired through local look-ahead detection, compressing the complex environmental map into key obstacle avoidance guidelines; 3D meta-information and feedback features: including collision markers And time-related metadata, including collision identifiers. Derived from ranging data during robot movement, it can determine whether the robot has collided with an obstacle based on the ranging data; the time metadata includes the current step count percentage. Ratio of remaining steps .

[0029] The multi-level reward guidance mechanism calculates reward values ​​based on the environmental information obtained from the high-fidelity gas diffusion simulation environment construction module, and guides the training of the decision-reward network based on near-end strategy optimization. The multi-level reward guidance mechanism is a rule-based explicit multi-level reward guidance mechanism that directly encodes the physical characteristics and engineering constraints of gas diffusion into scalar signals, guiding the robot to autonomously evolve a source-tracing strategy in a complex environment. Input of reward guidance mechanism It is a composite vector, representing the training data obtained from the interaction between the robot and the high-fidelity gas diffusion simulation data construction module. It includes: the robot's motion state, olfactory perception information, and collision status indicators. And time metadata; The output of the reward guidance mechanism is a single-step instant reward scalar. The single-step instant reward scalar is composed of five components: concentration gradient reward, safety constraint reward, exploration incentive reward, efficiency penalty and end-game reward, which are linearly superimposed. Total reward function at time step Defined as: Rewards based on concentration gradient; Rewards are imposed to ensure safety; To explore incentive rewards; Punishment for inefficiency; This is the final reward.

[0030] The concentration gradient reward Based on the physical law that the gas concentration increases along the direction of the leak source, the concentration change is converted into a dense navigation signal; in and These represent the gas concentrations in the training data from the two interactions, one before and one after.

[0031] The security constraint reward Imposing significant penalties on dangerous behaviors such as colliding with obstacles; In equations (7) and (8), This is a collision status flag; it takes a value of 1 when a collision occurs and 0 otherwise. The exploration incentive rewards Introduce exploration rewards based on access novelty, which decay over time; In the formula, The maximum allowed number of steps, The number of steps remaining; This is an indicator function that is 1 only when the robot first visits a grid; The efficiency penalty Apply a constant survival penalty per step to suppress the robot's ineffective wandering behavior; The final reward Define the sparse reward at the end of the task, and distinguish different levels of success and failure; The final reward for success includes two additional components: a remaining time reward and a step efficiency reward, designed to guide the robot to complete the task as quickly as possible.

[0032] The decision-reward network based on proximal policy optimization helps it achieve a more stable loss function and training process through pruning. This is done to limit the magnitude of each policy update and prevent new policies from causing issues. Deviating from the old strategy If the distance is too far, the objective function is pruned as follows. : In the formula: The importance ratio of the old and new strategies; For pruning hyperparameters; dominance function Calculated using generalized dominance estimation: in For time-series difference residuals; discount factor Used for those who value long-term cumulative returns; smoothing factor Used to balance bias and variance; Total loss function of decision-reward network Defined as: To reward the network's mean squared error loss, the weight coefficients... ; For policy entropy regularization, the weight coefficients are... =0.01.

[0033] The beneficial effects of this invention are as follows: This invention proposes an autonomous source tracing method for gas hazards using mobile robots based on deep reinforcement learning. First, a high-fidelity gas diffusion simulation environment construction module is used to address the problem of scarce training data, generating high-quality samples containing complex obstacles and dynamic turbulence. Second, by constructing a multi-level reward guidance mechanism, the sparse final objective is successfully decomposed into dense physical navigation signals and engineering constraints. Finally, relying on the powerful policy optimization capability of the PPO reinforcement learning algorithm, an Actor-Critic decision network with spatiotemporal reasoning capabilities is trained. This scheme completely eliminates the reliance on visual perception and overcomes the shortcomings of traditional chemotactic algorithms that are prone to getting trapped in local optima. It enables robots to achieve rapid, safe, and robust autonomous leak source location in complex industrial environments, relying solely on olfactory information and low-level ranging information, providing a new intelligent technological path for chemical safety and emergency rescue. Attached Figure Description

[0034] Figure 1 The diagram shows the decision-reward network framework based on proximal policy optimization. Figure 2 Training process for a mobile robot-based autonomous source tracing method for gas hazard sources using deep reinforcement learning; Figure 3 This is an example of a source tracing scenario. Detailed Implementation

[0035] This invention proposes an autonomous gas hazard source tracing method for mobile robots based on Deep Reinforcement Learning (DRL). For extreme industrial environments with limited vision, this method constructs an intelligent decision-making framework that relies solely on olfactory information and underlying ranging data. This intelligent decision-making framework mainly includes a high-fidelity gas diffusion simulation environment construction module, a multi-level reward guidance mechanism, and a decision-reward network based on Proximal Policy Optimization (PPO). The overall logical architecture is as follows: Figure 1 and Figure 2 As shown. The source tracing effect is as follows. Figure 3 As shown.

[0036] The high-fidelity gas diffusion simulation environment, built upon the convection-diffusion equation, generates training data containing obstacle distribution and dynamic concentration field evolution, solving the problem of obtaining training data in high-risk real-world environments. A multi-level reward guidance mechanism addresses the pain point of sparse rewards in gas source tracing tasks by fusing physical concentration gradient signals, safety obstacle avoidance constraints, and exploration incentives, guiding the robot to gradually evolve from random walks to a search strategy with "olfactory intelligence." The decision-reward network based on proximal policy optimization employs an Actor-Critic architecture, handling high-dimensional spatiotemporal state inputs and outputting continuous motion control commands, thereby achieving rapid and robust source tracing in complex turbulent environments. The specific implementation plan for gas source tracing using this model is as follows: Step 1: Scene preparation; To address the challenges of difficult data acquisition, high risk, and unreproducible scenarios in real gas leaks, this invention constructs a high-fidelity gas diffusion simulation environment building module based on fluid mechanics principles. By numerically solving the Advance-Diffusion Equation (ADE), standardized training data incorporating complex obstacle interference and dynamic wind field effects is generated. This invention employs a two-dimensional Advance-Diffusion equation to describe the spatiotemporal evolution of gas within a confined space. Its continuous-form governing equations are as follows: in, Indicates the gas concentration field (unit: ppm); The diffusion coefficient ( ); For the ambient wind speed vector ( ); The injection rate represents the leakage source term. This equation integrates the effects of molecular diffusion and advection transport mechanisms on gas distribution.

[0037] To achieve efficient solution in a computer, this invention employs the finite difference method (FDM) to discretize the governing equations in both time and space.

[0038] Diffusion term discretization: The diffusion process is approximated using a second-order central difference scheme of the Laplace operator. Convection term discretization: To ensure numerical stability and avoid non-physical oscillations, a first-order upwind scheme is used to handle the convection term. Time progression: Iterative updates are performed using the explicit Euler method to calculate the next time step. Concentration field: To simulate a real industrial environment, this invention introduces a dynamic hindering model of gas diffusion caused by obstacles. Diffusion coefficient It is no longer a constant, but based on the local obstacle density. Dynamic adjustment: Among them, the basic diffusion coefficient Set as When the region is an obstacle, the diffusion coefficient decreases to Times, simulating gas flow around and retention effects, The environmental medium factor is defined as follows: The environmental medium factor is used to simulate the normal diffusion of gases in free space. The simulation physics parameters are set as follows: physical space. Classified as The grid ( Prevailing wind speed crosswind ; Leakage source release rate To satisfy the Courant-Friedrichs-Lewy (CFL) stability condition, the computation time step is... Set as To ensure numerical solution convergence, the simulation lasted 1200 seconds (20 minutes). The system sampled data every 0.5 seconds, or every 50 computational steps, ultimately generating a concentration field tensor with a time dimension of 2400. .

[0039] Step 2: Decision-Reward Network Design; Decision-reward networks are the core guiding mechanism of reinforcement learning systems. This invention abandons the traditional black-box reward prediction and designs a rule-based explicit multi-level reward function. This function directly encodes the physical characteristics of gas diffusion (concentration gradient) and engineering constraints (obstacle avoidance, efficiency) into scalar signals, guiding the agent to autonomously evolve efficient source-tracing strategies in complex environments.

[0040] (1) Inputs and outputs of decision-reward networks; Input to the decision-reward network It is a composite vector containing: the robot's motion state (position coordinates, displacement vector), olfactory perception information (current concentration). Concentration change Visual perception information (obstacle markers in 8 directions around the vehicle), collision status markers And time metadata (number of steps used) Remaining steps ).

[0041] The output of the decision-reward network is a single-step instantaneous reward scalar. The single-step instant reward scalar is composed of five components: concentration gradient reward, safety constraint reward, exploration incentive reward, efficiency penalty, and end-game reward, which are linearly superimposed to balance safety, search efficiency, and positioning accuracy.

[0042] (2) A multi-level reward and guidance mechanism; To address the convergence difficulties caused by sparse rewards, this invention constructs the following five-layer reward system: 1) Concentration gradient reward ( ): Utilizing the physical law that gas concentration increases along the direction of the leak source, the concentration change is converted into a dense navigation signal.

[0043] In the formula, coefficient 50 is used to amplify the weak concentration gradient signal, making it dominate the decision-making process; This ensures that gradient rewards are not calculated when a collision occurs (displacement is hindered).

[0044] 2) Safety constraint rewards ( ): Imposing significant penalties on dangerous behaviors such as colliding with obstacles, prompting robots to learn to utilize free space.

[0045] In the formula, This is a collision indicator function, which takes the value 1 when a collision occurs and 0 otherwise.

[0046] 3) Explore incentive rewards ( To prevent robots from lingering in concentration plateau areas, an exploration reward based on access novelty is introduced, which decays over time.

[0047] In the formula, The maximum allowed number of steps; The indicator function is 1 only when the robot first visits a grid.

[0048] 4) Efficiency-driven punishment ( ): Imposing a constant survival penalty per step to suppress the robot's ineffective wandering behavior.

[0049] 5) Endgame reward ( ): Defines the sparse reward at the end of the task, distinguishing different levels of success and failure.

[0050] The success reward includes two additional components: a remaining time reward and a step efficiency reward, designed to guide the robot to complete the task as quickly as possible.

[0051] In summary, the total reward function Defined as: Step 3: Decision network design; The decision network is the core of this invention for implementing the gas source tracing task, and it adopts an Actor-Critic architecture based on deep neural networks. This architecture consists of two cooperating networks: the Actor network is responsible for outputting continuous action strategies based on environmental observations, and the Critic network is responsible for evaluating the expected value of the current state to guide strategy updates.

[0052] (1) State-space design; To satisfy the Markov property and provide sufficient decision-making basis, this invention designs a 17-dimensional continuous state vector that integrates multimodal sensing information. Its specific components are as follows: 1) Position and motion characteristics (4D): including normalized robot physical coordinates and the displacement vector at the previous moment Normalized coordinates ensure numerical stability, while displacement vectors provide momentum information and collision feedback.

[0053] 2) Olfactory perception features (2D): Includes the absolute value of gas concentration at the current location. With the change in concentration .in Directly mapping the gradient signal in the reward function is the core basis for judging the correctness of the movement direction.

[0054] 3) Environmental obstacle features (8-dimensional): Discrete obstacle markers corresponding to eight directions (up, down, left, right, and four opposite corners) around the robot. This feature is obtained through local look-ahead detection, compressing the complex environmental map into key obstacle avoidance guidance.

[0055] 4) Meta-information and feedback features (3D): Includes collision markers (Indicates whether the previous action caused a collision) and time metadata (current step count percentage) Remaining steps percentage The introduction of time information gives the strategy a time-varying nature, enabling it to dynamically balance between exploration (early stage) and exploitation (late stage).

[0056] (2) Actor network structure and action generation; The Actor network is constructed as a three-layer fully connected neural network. The input layer receives a 17-dimensional state vector. Feature extraction is performed through two hidden layers containing 64 neurons each, with hyperbolic tangent (Tanh) activation applied between layers. The output layer is mapped to a two-dimensional vector. The first dimension Used to generate movement angles. Motion sampling follows a Gaussian distribution strategy: In the formula, The standard deviation parameter is initially set to 0.6 to encourage exploration and decays to 0.1 as training progresses; the final action... Control the robot's direction of movement, with a fixed step length of 0.5 meters.

[0057] (3) Critic network structure and value assessment; The Critic network uses the same hidden layer structure as the Actor network, but the output layer is mapped to a single scalar. This scalar represents the state-value function of the current state, used to calculate the advantage function. This reduces the variance of policy gradient estimation and accelerates training convergence. In the formula, This is the discount factor. The Critic network optimizes by minimizing the value prediction error, i.e., the mean squared error, providing a stable baseline for the Actor network.

[0058] (4) Mathematical representation of decision networks; Define network parameters as and The forward propagation process can be uniformly represented as: in and These are the weight matrix and bias vector for each layer, respectively.

[0059] Step 4: PPO algorithm configuration.

[0060] The Proximal Policy Optimization (PPO) algorithm is the core driving engine for training the decision network in this invention. Compared with traditional policy gradient methods, PPO effectively solves the problem of policy degradation caused by excessively large step sizes by introducing a pruned agent objective function, without the need for complex trust region constraints, thus achieving a balance between training stability and sample efficiency.

[0061] (1) Design of the objective function for pruning To limit the magnitude of each policy update and prevent new policies from being updated... Deviating from the old strategy The objective function is too far away, so this invention adopts the following pruning form. : In the formula: The importance ratio of the old and new strategies is sampled. The clipping hyperparameter is set to 0.2.

[0062] This means that the importance sampling rate is limited to the range of [0.8, 1.2], and the gradient will be truncated when the update magnitude exceeds this range. This is the estimated value of the advantage function. This "pessimistic" lower bound optimization strategy ensures that the strategy update always remains within the safe region, even when the importance ratio fluctuates drastically.

[0063] (2) Generalized dominance estimation; To achieve the optimal balance between bias and variance, this invention employs generalized dominance estimation to calculate the dominance function. : in This represents the time-series difference residuals. Discount factor. Used for those who value long-term cumulative returns; smoothing factor Used to weigh bias against variance.

[0064] (3) Total loss function and optimization strategy To simultaneously optimize strategy and value assessment, and to encourage continuous exploration, the total loss function... Defined as: The mean squared error loss of the Critic network, weight coefficients . This is the policy entropy regularization term, with a weight coefficient c_2=0.01, used to prevent the policy from converging prematurely to a local optimum.

[0065] (4) Training process and hyperparameter settings; The training process adopts a cyclical pattern of "experience collection - batch update": 1) Experience collection: Interactively collect data in the simulation environment using the current strategy. 1) Step-by-step experience data, stored in a buffer. 2) Batch update: Utilizing the collected data for... Gradient updates are performed in rounds, with a batch size of 64. 3) Optimizer: The Adam optimizer is used, and the Actor network learning rate is set to 64. Critic network learning rate 4) Exploratory decay: Standard deviation of motion sampling The initial value is set to 0.6, and it decreases linearly by 0.05 every 250,000 steps as the number of training steps increases, until it reaches a lower limit of 0.1, thus achieving a smooth transition from wide-area exploration to fine-grained utilization.

[0066] This invention utilizes deep reinforcement learning (PPO) technology to construct a method for autonomous tracing of gas hazard sources in mobile robots based on deep reinforcement learning. This process addresses the difficulty of acquiring training data in real high-risk scenarios by constructing a high-fidelity physical simulation environment, and uses a multi-level reward guidance mechanism to guide the robot to evolve a search strategy adapted to complex turbulent environments, thereby obtaining a highly robust intelligent decision-making network. Simultaneously, our design exhibits strong environmental adaptability, suitable for extreme industrial scenarios with limited visibility, such as dense smoke and darkness. Key points are as follows: (1) A decision-reward network based on proximal policy optimization is proposed. By adopting the Actor-Critic architecture and deeply mining multimodal spatiotemporal state features including concentration gradient, motion trend and environmental perception, the traditional chemotaxis algorithm is prone to getting trapped in local optima, and accurate localization without visual assistance is achieved.

[0067] (2) Design a multi-level reward guidance mechanism that integrates physical laws and engineering constraints. This mechanism cleverly decomposes the sparse end goal into dense concentration gradient rewards, safety obstacle avoidance constraints, and exploration incentives based on access novelty, effectively solving the convergence difficulties and training inefficiencies caused by sparse rewards in reinforcement learning in gas source tracing tasks.

[0068] (3) A high-fidelity gas environment physical simulation and trigger-point driven training strategy based on convection-diffusion equations is constructed. By generating standardized training data containing complex obstacle distribution and dynamic concentration field evolution, the physical realism and diversity of the samples and the training startup efficiency of the model are greatly improved.

[0069] This invention proposes an autonomous source tracing method for gas hazards using mobile robots based on deep reinforcement learning. Through the collaborative work of an Actor-Critic architecture and a multi-level reward guidance mechanism, the robot can autonomously plan the optimal path to the leak source based solely on olfactory concentration changes and low-level ranging information. This technology completely solves the problem of existing robots being unable to operate in visually limited scenarios such as dense smoke or complete darkness, while also overcoming the technical bottleneck of traditional algorithms struggling to converge in sparse reward environments. In the verification process, we constructed an experimental scheme with completely isolated training and test sets in a high-fidelity physical simulation environment. The results show that the proposed solution has extremely strong generalization ability; even when facing leak sources in entirely new locations, the robot can still maintain a very high source tracing success rate and obstacle avoidance safety. Furthermore, experimental data demonstrates that the model can effectively and quickly converge from a random exploration state to a stable plume tracking strategy, possessing the potential for practical deployment in complex unstructured industrial environments.

Claims

1. A method for autonomous tracing of gas hazard sources using a mobile robot based on deep reinforcement learning, characterized in that, A smart decision-making framework that relies solely on olfactory information and ranging data is constructed. The smart decision-making framework includes a high-fidelity gas diffusion simulation environment construction module, a decision-reward network based on near-end strategy optimization, and a multi-level reward guidance mechanism. The high-fidelity gas diffusion simulation environment construction module is based on the convection-diffusion equation and generates training data that includes obstacle distribution and dynamic concentration field evolution. The multi-level reward guidance mechanism integrates physical concentration gradient signals, safety obstacle avoidance constraints and exploration incentives to guide the robot to gradually evolve from random walks to a search strategy with "olfactory intelligence". The decision-reward network based on proximal strategy optimization adopts a dual-network architecture to process high-dimensional spatiotemporal state inputs and output continuous motion control commands.

2. The method for autonomous tracing of gas hazard sources in mobile robots based on deep reinforcement learning according to claim 1, characterized in that, The high-fidelity gas diffusion simulation environment construction module generates standardized training data containing complex obstacle interference and dynamic wind field effects by numerically solving the two-dimensional convection-diffusion equation, providing olfactory information and a source tracing platform for the intelligent decision-making framework; the two-dimensional convection-diffusion equation is used to describe the spatiotemporal evolution of gas in a confined space, and the continuous form of the governing equation is as follows: （1） in, Represents the gas concentration field; Where is the diffusion coefficient. ; For ambient wind speed vector, ; The injection rate of the leakage source term is used; the finite difference method is used to discretize the control equations in time and space; Diffusion term discretization: The diffusion process is approximated using a second-order central difference scheme with the Laplace operator. in, coordinates The gas concentration at that location; Convection term discretization: The first-order upwind scheme is used to handle the convection term; in, The spatial distance between two environmental points; Time progression: Iterative updates are performed using the explicit Euler method to calculate the next time step. Concentration field: in, That is, the discrete form of the Laplace operator. coordinates Injection rate at the site, This represents the time interval for each iteration; A dynamic hindrance model for gas diffusion by obstacles is introduced; based on the local obstacle density... Dynamically adjust the diffusion coefficient : in, The base diffusion coefficient decreases to [value missing] when the region is an obstacle. Times, simulating gas flow around and retention effects, The environmental medium factor is defined as follows: The environmental medium factor is used to simulate the normal diffusion of gases in free space.

3. The method for autonomous tracing of gas hazard sources in mobile robots based on deep reinforcement learning according to claim 1, characterized in that, The decision-reward network based on proximal policy optimization comprises two working networks: a decision network responsible for outputting continuous action policies based on environmental observations, and a reward network responsible for evaluating the expected value of the current state to guide policy updates. The decision network and reward network are both three-layer fully connected neural networks. The parameters of the decision network and reward network are defined as follows: and The forward propagation process is uniformly represented as: in , and , These are the weight matrices and bias vectors for the first two layers of the decision-reward network, respectively. , and , These are the weight matrix and bias vector of the output layer of the decision-reward network, respectively. and These represent the outputs of the decision network and the reward network, respectively. The input layer of the decision network receives a 17-dimensional state vector from the training data generated by the high-fidelity gas diffusion simulation environment building module. Feature extraction is performed through two hidden layers containing 64 neurons each, with hyperbolic tangent activation applied between layers; the output layer is mapped to a two-dimensional vector. This constructs a high-precision continuous motion space, in which the output layer receives the output. The first dimension After activation by the hyperbolic tangent function, it is restricted to... The interval, and used as the mean parameter of the action distribution, for The generation mechanism adopts a "sampling-truncation-mapping" approach: First, random sampling is performed based on a Gaussian distribution strategy to generate normalized actions. : The risk of illegal numerical values ​​caused by the long-tail effect of the Gaussian distribution is eliminated by using a truncation function, and the results are mapped to the physical angle space through a linear transformation: In the formula, The standard deviation is a learnable parameter that decays as training progresses. This is a truncation function; ultimately Mapped as continuous actions , used to control the robot's direction of movement; according to formula (12), the normalized value -1 is mapped to 0 radians, and 0 is mapped to Radians, +1 mapped to 2π radians, eliminates the direction quantization error of traditional discrete motion spaces, enabling the robot to output... An arbitrary floating-point angle within the interval; According to formula (10), the reward function output layer is mapped to a single scalar. ; The single scalar represents the state value function of the current state, used to calculate the advantage function. This reduces the variance of policy gradient estimation and accelerates training convergence. In the formula, As a discount factor, for The total reward function output at time step 1; the reward network optimizes by minimizing the value prediction error and mean squared error, providing a stable baseline for the decision network.

4. The method for autonomous tracing of gas hazard sources in mobile robots based on deep reinforcement learning according to claim 3, characterized in that, The input to the decision-reward network is the 17-dimensional state vector. The training data, derived from the interaction between the robot and the high-fidelity gas diffusion simulation construction module, consists of the following components: 4D position and motion characteristics: including normalized robot physical coordinates and the displacement vector at the previous moment ; 2D olfactory perception features: including the absolute value of gas concentration at the current location. With the change in concentration ; The gradient signal in the total reward function is directly mapped to determine the correctness of the movement direction; 8-dimensional environmental obstacle features: discrete obstacle markers corresponding to eight directions around the robot; By acquiring local forward-looking detection, complex environmental maps are compressed into key obstacle avoidance guidelines; 3D meta-information and feedback features: including collision markers And time-related metadata, including collision identifiers. Derived from ranging data during robot movement, it can determine whether the robot has collided with an obstacle based on the ranging data; the time metadata includes the current step count percentage. Ratio of remaining steps .

5. The method for autonomous tracing of gas hazard sources in mobile robots based on deep reinforcement learning according to claim 1, characterized in that, The multi-level reward guidance mechanism calculates reward values ​​based on the environmental information obtained from the high-fidelity gas diffusion simulation environment construction module, and guides the training of the decision-reward network based on near-end strategy optimization. The multi-level reward guidance mechanism is a rule-based explicit multi-level reward guidance mechanism that directly encodes the physical characteristics and engineering constraints of gas diffusion into scalar signals, guiding the robot to autonomously evolve a source-tracing strategy in a complex environment. Input of reward guidance mechanism It is a composite vector, representing the training data obtained from the interaction between the robot and the high-fidelity gas diffusion simulation data construction module. It includes: the robot's motion state, olfactory perception information, and collision status indicators. And time metadata; The output of the reward guidance mechanism is a single-step instant reward scalar. The single-step instant reward scalar is composed of five components: concentration gradient reward, safety constraint reward, exploration incentive reward, efficiency penalty and end-game reward, which are linearly superimposed. Total reward function at time step Defined as: Rewards based on concentration gradient; Rewards are imposed to ensure safety; To explore incentive rewards; Punishment for inefficiency; This is the final reward.

6. The method for autonomous tracing of gas hazard sources in mobile robots based on deep reinforcement learning according to claim 5, characterized in that, The concentration gradient reward Based on the physical law that the gas concentration increases along the direction of the leak source, the concentration change is converted into a dense navigation signal; in and These represent the gas concentrations in the training data from the two interactions, one before and one after. The security constraint reward Imposing significant penalties on dangerous behaviors such as colliding with obstacles; In equations (7) and (8), This is a collision status flag; it takes a value of 1 when a collision occurs and 0 otherwise. The exploration incentive rewards Introduce exploration rewards based on access novelty, which decay over time; In the formula, The maximum allowed number of steps, The number of steps remaining; This is an indicator function that is 1 only when the robot first visits a grid; The efficiency penalty Apply a constant survival penalty per step to suppress the robot's ineffective wandering behavior; The final reward Define the sparse reward at the end of the task, and distinguish different levels of success and failure; The final reward for success includes two additional components: a remaining time reward and a step efficiency reward, designed to guide the robot to complete the task as quickly as possible.

7. The method for autonomous tracing of gas hazard sources in mobile robots based on deep reinforcement learning according to claim 1, characterized in that, The decision-reward network based on proximal policy optimization helps the decision-reward network obtain a more stable loss function and training process through pruning; To limit the magnitude of each policy update and prevent new policies from being updated... Deviating from the old strategy If the distance is too far, the objective function is pruned as follows. : In the formula: The importance ratio of the old and new strategies; For pruning hyperparameters; dominance function Calculated using generalized dominance estimation: in For time-series difference residuals; Discount factor Used for those who value long-term cumulative returns; smoothing factor Used to balance bias and variance; Total loss function of decision-reward network Defined as: To reward the network's mean squared error loss, the weight coefficients... ; For policy entropy regularization, the weight coefficients are... =0.01.

Images