Sensing method and system of rescue robot facing smog environment

Abstract

The invention discloses a sensing method and system of a rescue robot facing a smog environment, and relates to the technical field of robot environment sensing. Original information acquisition and space-time synchronization are carried out, and laser radar and millimeter wave radar original point clouds are unified to the same time reference and space coordinate system; the millimeter wave point cloud is purified through three-stage processing of effective view field constraint, Gaussian intensity separation suppression and density filtering suppression; based on adaptive density clustering, feature extraction and learning type discrimination, outputting a smoke three-dimensional mask, a volume and a scene stage label; and a fusion strategy is adaptively selected according to the scene stage, and a unified fusion point cloud is generated. The system correspondingly comprises an original information acquisition module, a millimeter wave point cloud noise suppression module, a smoke three-dimensional quantitative sensing module and a dynamic fusion weight adjustment and point cloud fusion module. According to the invention, stable sensing under all working conditions of the smog environment can be realized, and the operation safety and reliability of the rescue robot in a low-visibility environment are improved.

Description

Technical Field

[0001] This invention relates to the field of robot environmental perception technology, specifically to a perception method and system for rescue robots in smoke-filled environments. Background Technology

[0002] In high-risk scenarios such as fire rescue and disaster detection, smoke environments severely hinder the environmental perception capabilities of rescue robots. While lidar can provide high-precision, high-resolution three-dimensional spatial structure information, as an optical sensor, its working mechanism is inherently affected by suspended particles in the atmosphere. In smoke environments, it produces strong light scattering phenomena, leading to rapid attenuation of light energy, distortion of echo signals, and point cloud artifacts such as false height, drift, structural loss, or even complete discontinuity. Millimeter-wave radar has strong smoke penetration capabilities and can still stably detect information such as the distance and speed of objects in low-visibility environments. However, millimeter-wave point clouds are inherently sparse and coarse, and are easily affected by factors such as multipath reflection and sidelobe interference, resulting in a large number of false alarm points and insufficient stability.

[0003] To fully utilize the complementary characteristics of the two sensors, the industry generally adopts a multi-source sensing fusion strategy. However, existing fusion schemes still have many shortcomings in smoke environments: First, millimeter-wave point clouds suffer from severe noise pollution, and existing methods lack a dedicated preprocessing mechanism for millimeter-wave radar in smoke environments. Noise points are mixed with real obstacle points, seriously interfering with the accuracy of environmental perception. Second, there is a lack of quantitative perception capability for smoke interference, making it impossible to actively and quantitatively estimate the distribution and volume of smoke, resulting in a lack of foresight and adaptability in fusion decisions. Third, the fusion strategies are rigid, often using fixed fusion logic or weights, and cannot adaptively adjust according to the real-time perceived smoke concentration and the performance degradation of lidar. This can easily lead to perception blind spots at critical moments, threatening rescue safety.

[0004] In existing related technologies, such as the intelligent evacuation guidance robot system for tunnel fires disclosed in patent CN120869118A, the fusion weights of lidar and millimeter-wave radar are adjusted by monitoring the smoke concentration threshold. However, this solution only makes a simple threshold judgment based on the smoke concentration at a single point, and cannot achieve three-dimensional quantitative modeling of smoke. The adjustment of the fusion strategy lacks spatial precision. The environmental adaptive target tracking method proposed in patent CN117522922A directly blocks lidar data when smoke is detected, causing a significant degradation of lidar signal, and relies only on the output results of millimeter-wave radar. This solution does not consider the uneven distribution of smoke and the local effective perception capability of lidar, which can easily lead to waste or loss of perception information in the early and late stages of smoke. Summary of the Invention

[0005] The present invention aims to solve the above-mentioned technical problems by providing a perception method and system for rescue robots in smoke-filled environments.

[0006] To solve the above-mentioned technical problems, the technical solution provided by the present invention is as follows:

[0007] A perception method for a rescue robot in a smoke-filled environment includes the following steps:

[0008] S1. Raw information collection and spatiotemporal synchronization:

[0009] The system acquires raw point cloud data from lidar and millimeter-wave radar, aligns the two types of point clouds on the time axis through a time synchronization mechanism, maps the two types of point clouds to a unified sensor platform coordinate system through a spatial synchronization mechanism, and outputs the synchronized raw point cloud.

[0010] S2, Millimeter-wave point cloud noise suppression:

[0011] The synchronized millimeter-wave raw point cloud is sequentially processed with effective field-of-view suppression, Gaussian intensity separation suppression, and density filtering suppression to output a purified millimeter-wave point cloud.

[0012] S3, Three-dimensional quantitative perception of smoke:

[0013] Using the synchronized laser point cloud as input, candidate point clusters are obtained through adaptive density clustering. The statistical features of each candidate point cluster are extracted and input into a pre-trained classifier for smoke cluster identification. The three-dimensional position, volume and scene statistical indicators of the smoke clusters are calculated, and the smoke stage of the scene is determined based on the scene statistical indicators.

[0014] S4. Dynamic fusion weight adjustment and point cloud fusion:

[0015] Select the corresponding fusion strategy according to the smoke stage, determine the dominant sensor source and weight in different spatial regions, and generate a unified fused point cloud based on the dominant source, weight, purified millimeter-wave point cloud and synchronized original laser point cloud.

[0016] Preferably, the time synchronization mechanism includes time synchronization and spatial synchronization. The spatial synchronization involves establishing rigid external parameters of the lidar coordinate system L and the millimeter-wave radar coordinate system R to the sensor platform coordinate system P, and obtaining translation and rotation parameters through offline calibration. and For any point and As shown in the following formula: ;in, Let L be the set of point clouds represented in the lidar coordinate system L. Let R be the set of point clouds represented in the millimeter-wave radar coordinate system. For the first in the laser point cloud The three-dimensional coordinates of a point in coordinate system L. For the first point cloud in the radar point cloud The three-dimensional coordinates of a point in coordinate system R. To separately The three-dimensional coordinates mapped to the platform coordinate system P. Let L be the rotation matrix from L to P. Let L be the translation vector from L to P. Let R be the rotation matrix from R to P. This is a translation vector from R to P; after time and space synchronization processing, the output is... Aligned on the time axis and belonging to the same coordinate system in the spatial domain .

[0017] Preferably, the effective field of view suppression is based on a preset radial effective range. and highly effective range The filtering criteria are: ; Select millimeter-wave points that satisfy both radial distance and height within the corresponding effective range, and discard invalid points that exceed the effective field of view. The output point set is denoted as . ;

[0018] in, The value range is 0-1.0m. The value range is 10.0-40.0m, both indoors and outdoors. Indoor ,outdoor It is infinitely large.

[0019] Preferably, the Gaussian intensity separation suppression specifically involves fitting the echo intensity distribution using a Gaussian mixture model (GMM), as shown in the following formula: The parameters are estimated using the expectation-maximization algorithm, and the intensity at the intersection of the Gaussian distribution is set as the intensity threshold. The points are classified as follows: The preserved point cloud is recorded as .

[0020] Preferably, the density filtering suppression specifically involves using the DBSCAN algorithm to perform spatial density filtering on the point cloud. ;in The number of points in the neighborhood. To be the minimum number of points, Let be the neighborhood radius, then Points marked as core points are expanded using density reachability relationships to form effective clusters; otherwise, they are marked as noise points. The final output set of effective millimeter-wave point clouds is as follows: .

[0021] Preferably, the point cloud processed by time synchronization and spatial synchronization is used as input, and adaptive clustering, feature extraction and learning-based discrimination are performed on the point cloud for smoke scene, and three types of results are output: ① 3D smoke mask and its cluster volume; ② 3D location and range of smoke region; ③ Smoke stage and statistical indicators for system scheduling and fusion.

[0022] Preferably, the smoke stages include no smoke, initial smoke stage, middle smoke stage, and later smoke stage.

[0023] The present invention also provides a perception system for a rescue robot in a smoke-filled environment, comprising:

[0024] The raw information acquisition module is used to acquire raw point cloud data from lidar and millimeter-wave radar, and to achieve time and spatial synchronization of the two types of point clouds, and output the synchronized raw point cloud.

[0025] The millimeter-wave point cloud noise suppression module is used to perform effective field-of-view suppression, Gaussian intensity separation suppression and density filtering suppression on the synchronized original millimeter-wave point cloud in sequence, and output the purified millimeter-wave point cloud.

[0026] The smoke 3D quantization perception module is used to obtain the 3D position and volume of the smoke mask and smoke clusters through adaptive density clustering, feature extraction and learning-based discrimination, using the synchronized original laser point cloud as input, calculating scene statistical indicators and determining the smoke stage.

[0027] The dynamic fusion weight adjustment module is used to select a fusion strategy based on the smoke stage, determine the dominant sensor sources and weights in different spatial regions, and generate and output a unified fused point cloud.

[0028] Preferably, the raw information acquisition module includes a time synchronization unit and a spatial synchronization unit.

[0029] Preferably, the dynamic fusion weight adjustment module includes a dynamic fusion weight strategy unit and a point cloud fusion construction unit; the dynamic fusion weight strategy unit is used to output the corresponding sector / region-level dominant source and weight according to the smoke stage; the point cloud fusion construction unit is used to receive spatiotemporally aligned two types of point clouds and dominant source weight information, and generate a fused point cloud through point attribution routing and synthesis.

[0030] By employing the above methods and systems, the present invention has the following advantages:

[0031] 1. The heterogeneous sensor spatiotemporal synchronization scheme proposed in this invention uses lidar as the true clock and implements hard-triggered sampling for millimeter-wave radar. Combined with offline calibrated spatial extrinsic parameters, it effectively eliminates cross-sensor time drift and spatial deviation, ensuring that subsequent fusion algorithms work on the same spatiotemporal reference, and significantly improving fusion consistency and perception accuracy.

[0032] 2. The three-stage millimeter-wave denoising link constructed in this invention, consisting of effective field-of-view constraint, intensity adaptive threshold separation, and DBSCAN density filtering, specifically addresses the problem of severe noise pollution in millimeter-wave point clouds under smoggy conditions, significantly improving the quality of millimeter-wave point clouds and providing reliable input for subsequent fusion.

[0033] 3. The smoke 3D quantization and stage determination mechanism proposed in this invention realizes 3D modeling of smoke distribution and quantitative expression of degradation degree through adaptive clustering, feature extraction and learning-based discrimination, providing a precise basis for adaptive adjustment of fusion strategy and enhancing the interpretability of environmental semantics.

[0034] 4. The phased dynamic fusion strategy designed in this invention achieves differentiated sensor weight allocation based on the environmental characteristics of different smoke stages, covering all working conditions from no smoke to dense smoke diffusion, ensuring the integrity and stability of the sensing information, and effectively solving the problem of sensing blind spots caused by the rigidity of existing fusion strategies.

[0035] 5. This invention adopts a lightweight fusion implementation method that decouples the control plane and data plane, directly consumes the upstream sector / sector ring allocation results to perform point routing and synthesis, with low computational overhead, and the source and weight of the fused point cloud can be traced, which is convenient for engineering deployment and long-term maintenance. Attached Figure Description

[0036] Figure 1 This is a system overall framework diagram of the present invention;

[0037] Figure 2 This is a framework diagram of the original information acquisition module of this invention;

[0038] Figure 3 This is a framework diagram of the millimeter-wave point cloud noise suppression module of the present invention;

[0039] Figure 4 This is a framework diagram of the smoke three-dimensional quantization sensing module of the present invention;

[0040] Figure 5 This is a framework diagram of the dynamic fusion weight adjustment and point cloud fusion module of the present invention. Detailed Implementation

[0041] The present invention will now be described in further detail with reference to the full text.

[0042] Combined with appendix Figures 1-5 This invention addresses the stable perception requirements of rescue robots in extreme environments such as smoke and multiple obstructions. It constructs a comprehensive technical system for "multi-source data acquisition—environmental state modeling—dynamic fusion decision-making—point cloud fusion output" based on both lidar and millimeter-wave radar. The overall system framework is as follows: Figure 1 As shown.

[0043] The system can be broadly divided into three main layers: the initial perception layer, the environment modeling layer, and the fusion decision layer. The initial perception layer, composed of LiDAR and millimeter-wave radar, is responsible for real-time acquisition of environmental geometric information and millimeter-wave scattering characteristics, forming two types of initial point cloud inputs. The environment modeling layer includes a millimeter-wave point cloud noise suppression module and a smoke 3D quantization perception module. The former aims to remove false alarms, clutter, and sparse noise points from the millimeter-wave radar's initial point cloud, obtaining clean and reliable millimeter-wave point cloud inputs. The latter utilizes the intensity and spatial characteristics of the LiDAR to perform 3D quantization modeling and stage identification of smoke, providing environmental data for subsequent fusion.

[0044] The fusion decision layer consists of a dynamic fusion weight adjustment module and a point cloud fusion construction module. The dynamic fusion weight adjustment module automatically determines the current environment's stage (no smoke, early smoke, middle smoke, or late smoke) based on smoke concentration, spatial distribution, and perception degradation, and activates the corresponding strategy sub-modules to achieve adaptive weight adjustment of laser and millimeter wave in different spatial regions and environmental scenarios.

[0045] The dynamic fusion weight adjustment module takes smoke concentration, smoke spatial distribution, and perception degradation degree as inputs. When the smoke concentration output by the smoke 3D quantization perception module is 0, the environment is judged to be in the smoke-free stage, and the corresponding fusion strategy submodule is dominated by LiDAR data. When the smoke concentration is not 0 and the spatial distribution volume is less than a set threshold, the environment is judged to be in the early stage of smoke, and the corresponding fusion strategy submodule is radial fusion data (circumferential radial spatial distribution). When the smoke concentration is not 0 and the spatial distribution volume is greater than or equal to the set threshold, the environment is judged to be in the middle stage of smoke, and the corresponding fusion strategy submodule is tangential fusion data (circumferential tangential spatial distribution). When the smoke concentration is 0 (smoke dispersion), and the perception degradation degree meets the criteria of increased invalid points, decreased average reflection intensity, and significantly shortened maximum perception distance, the environment is judged to be in the late stage of smoke, and the corresponding fusion strategy submodule is dominated by millimeter-wave radar data. This enables adaptive weight adjustment of LiDAR and millimeter-wave radar in different spatial regions and environmental scenarios.

[0046] The point cloud fusion module, based on the weighting and partitioning strategies of the previous level, performs spatiotemporal alignment, regional selection, and weighted fusion on laser point clouds and millimeter-wave point clouds, ultimately outputting unified fused point cloud data. This provides stable input for downstream tasks such as environmental mapping, navigation, detection, and tracking. The system can output fused point clouds at 10Hz, with a total processing latency of ≤30ms per frame (denoising ≤10ms, smoke quantization perception ≤10ms, fusion ≤10ms).

[0047] Overall, this invention forms a continuous closed loop of "millimeter-wave denoising → smoke modeling → dynamic fusion → fused point cloud output", enabling the rescue robot to maintain a highly robust environmental perception capability in all working conditions, from no smoke to dense smoke obscuration.

[0048] The raw information acquisition module is used to acquire raw point cloud data from both lidar and millimeter-wave radar during the operation of the rescue robot, and to perform time and spatial synchronization under a unified benchmark, outputting synchronized raw point cloud data that can be directly used for subsequent perception and fusion. The module structure is as follows: Figure 2 As shown.

[0049] For time synchronization, this system uses a lidar as the time reference. The STM32 minimum system board distributes a 1Hz PPS pulse and a corresponding second UTC time message (GPRMC) to the lidar to correct its internal clock and frame timestamp. The lidar does not accept external hard triggering, so its output frame time can be considered the system's "true clock." Simultaneously, the STM32 performs hard-triggered sampling of the millimeter-wave radar using 10Hz PWM to ensure the radar frame's beat is consistent with the unified time base. After the two raw point clouds are transmitted back to the main controller, the main controller writes a timestamp according to the unified clock and performs nearest-neighbor pairing in the buffer to form pairs of data at the same time. This strategy uses a "PPS+UTC time-calibrated lidar" to provide the absolute time stamp, and then uses a "10Hz hard-triggered millimeter-wave radar" to lock the beat, balancing low jitter and traceability, providing a stable input for subsequent spatial synchronization and fusion processing.

[0050] In terms of spatial synchronization, the system uses the sensor platform coordinate system. To unify the expression domain, separate lidar coordinate systems are established. and millimeter-wave radar coordinate system The rigid body parameters of the sensor platform coordinate system are obtained through offline calibration to obtain translation-rotation pairs. and For any point and According to: ;in: A unified coordinate system is established for the sensor platform, serving as a common representation domain; For the lidar coordinate system; For millimeter-wave radar coordinates; This is a set of point clouds (or point sets) represented in the lidar coordinate system L. This is a set of point clouds (or point sets) represented in the millimeter-wave radar coordinate system R. For the first in the laser point cloud A point, its three-dimensional coordinates (column vector) in coordinate system L; For the first point cloud in the radar point cloud A point, its three-dimensional coordinates (column vector) in coordinate system R; To separately The three-dimensional coordinates mapped to the platform coordinate system P; Let L be the rotation matrix from L to P (usually a 3×3 orthogonal matrix). Let L be the translation vector from L to P (usually 3×1). Let R be the rotation matrix from R to P; Let be the translation vector from R to P.

[0051] The above (R,T) is generally obtained from offline extrinsic parameter calibration and represents the rigid body transformation parameters of each sensor relative to the platform coordinate system.

[0052] Mapping to the sensor platform coordinate system enables the co-domain representation of the two types of point clouds in space.

[0053] After time and space synchronization processing, the output is Aligned on the time axis and belonging to the same coordinate system in the spatial domain It can be directly used as a stable input for subsequent millimeter-wave point cloud noise suppression, smoke 3D quantization perception, and cross-modal fusion. Refers to the first Alignment timestamps. After time synchronization, at the same moment The aligned LiDAR / Millimeter-wave radar point set (or point cloud frame) is labeled "sync": indicating that time alignment has been completed (e.g., interpolation, nearest neighbor matching, or triggered synchronization), ensuring that the two modal data correspond to the semantics of the same time.

[0054] The millimeter-wave point cloud noise suppression module addresses the inherent multipath scattering, side-leaf echoes, and sparse discrete points of millimeter-wave radar. Without dedicated denoising processing, directly participating in multi-source fusion would significantly reduce the overall sensing accuracy and stability of the system. Therefore, this invention incorporates a millimeter-wave point cloud noise suppression module after data acquisition, employing a staged filtering technique to perform multi-level quality enhancement on the millimeter-wave point cloud. The module's processing flow includes three sub-steps: effective field-of-view suppression, Gaussian intensity separation suppression, and density filtering suppression. The overall method is as follows: Figure 3 As shown.

[0055] Effective field-of-view suppression: To eliminate invalid points that exceed the physical field of view of the millimeter-wave radar, this step involves processing the millimeter-wave radar point cloud. Perform location-based filtering.

[0056] Each point output by millimeter-wave radar It can be represented by its rectangular coordinates or polar coordinates: The three-dimensional coordinates of a point; Radial distance; Height of the point

[0057] Based on the effective range of millimeter waves set by the system:

[0058] Radial effective range: ;

[0059] High effective range: ;

[0060] The filtering criteria are:

[0061] ;

[0062] in, Determined based on experience with different millimeter-wave radar models , Determined based on the maximum detection range of the millimeter-wave radar model. , and Determined based on the actual application scenario (indoor / outdoor). ,indoor ,outdoor (for infinity)

[0063] Only points that meet the above conditions are retained; the rest are considered distance noise or ground reflection noise and are discarded. The output point set is denoted as... .

[0064] Gaussian intensity separation suppression for millimeter-wave radar echo intensity It can reflect the effectiveness of a point. Low-intensity points are usually background noise or multipath reflections.

[0065] This invention uses a Gaussian mixture model (GMM) to fit the intensity distribution, representing it as a mixture of two Gaussian distributions:

[0066] ;

[0067] Where: N means Gaussian distribution. Indicating targeting The mean is The variance is Gaussian distribution; The echo intensity at the millimeter-wave point; This represents the probability density value when the intensity is 𝐼.

[0068] To represent the "random intensity variable" with a mean of variance is Under a Gaussian distribution, the value is taken as... Density at time, where It is the independent variable. Determine the location of the distribution center. Determines the degree of dispersion of the distribution; The mixture weight / prior proportion of the first Gaussian component represents the proportion of this component's contribution to the overall distribution; The mixing weights for the second Gaussian component, and ; The mean of the first Gaussian component corresponds to the center of the intensity distribution (representing the typical intensity level of this type of sample). The mean of the second Gaussian component; The variance of the first Gaussian component characterizes the range of fluctuation in the intensity of that component (the larger the variance, the more dispersed the component). Let be the variance of the second Gaussian component, and ;

[0069] The above parameters were estimated using the expectation-maximization algorithm, and the intensity threshold was set at the intersection of the Gaussian distributions (where the signal-to-noise ratio is 1). (Intersection threshold adaptive update period 1s), the points are classified as follows:

[0070] ;

[0071] The preserved point cloud is recorded as .

[0072] Density filtering suppresses isolated noise points that still exist in millimeter-wave point clouds after intensity separation. This invention uses DBSCAN (density-based clustering) to perform spatial density filtering on point clouds.

[0073] First, set:

[0074] Neighborhood radius , usually take ;

[0075] Minimum points , usually take ;

[0076] point In the neighborhood radius Number of points ;

[0077] For a point, its neighborhood is defined as:

[0078] ;

[0079] in:

[0080] The first one to be judged Each millimeter-wave point (or point cloud point) has spatial coordinates that are two-dimensional / three-dimensional vectors.

[0081] For the first point cloud Reference points (and) Candidate points for distance comparison are also two-dimensional / three-dimensional coordinate vectors;

[0082] For point of Neighborhood set, that is, in the point set In, with The distance does not exceed The set of all points;

[0083] For point and The spatial distance is usually taken as the Euclidean distance;

[0084] If the following conditions are met:

[0085] ;

[0086] in:

[0087] The number of points in the neighborhood;

[0088] but If a point is marked as a core point, it is expanded to form an effective cluster through density reachability relations; otherwise, it is marked as a noise point.

[0089] The final output set of effective millimeter-wave point clouds is .

[0090] The point cloud structure is stable and has very little noise, providing a reliable input for subsequent multi-source fusion.

[0091] The smoke 3D quantization perception module uses point clouds processed with time and space synchronization. As input, adaptive clustering, feature extraction, and learned discrimination are performed on point clouds in smoke scenes. Three types of results are output: ① a 3D smoke mask and its cluster volume; ② the 3D location and extent of the smoke region; ③ the smoke stage (no / early / mid / late stage) and statistical indicators for system scheduling and fusion. The module structure and data flow are as follows: Figure 4 As shown.

[0092] Adaptive DBSCAN filtering and point clustering are used to construct the synchronized original point cloud. Input the adaptive DBSCAN filter module. Considering that smoke will cause changes in echo intensity and unevenness in point cloud sparsity, for each point... Calculate its strength deviation and with intensity threshold Map it to scaling factor (Constraints) ).

[0093] ;

[0094] The point is then obtained from the scaling factor. Adaptive clustering radius (neighborhood radius):

[0095] ;

[0096] That is, appropriately enlarge when the strength deviation is small. (Maximum magnification) (times), while when the deviation is large, the radius is converged and kept not less than 10 times. DBSCAN is performed at this adaptive radius, ultimately forming a set of candidate point cloud clusters.

[0097] Feature extraction and XGBoost classification for each candidate point cluster The statistical characteristics of smoke are calculated, including: mean intensity, variance intensity, intensity quantile interval (20%–80%), and local point density.

[0098] The calculation details are as follows:

[0099] Mean intensity: This reflects the average reflection intensity of the cluster;

[0100] Strength variance: This describes the degree of dispersion in the intensity distribution;

[0101] Intensity 20%-80% quantiles: - To capture the skewness of the intensity distribution;

[0102] density: , indicating spatial compactness, where Let be the apparent volume of the cluster.

[0103] Features are concatenated into a vector in a fixed order and input to the XGBoost classifier. The classifier is trained offline on labeled data (containing "smoke / non-smoke" labels and samples of different concentrations). In the online phase, it performs only forward inference, outputting the probability that each cluster is a smoke cluster. When the probability exceeds a threshold, the cluster is labeled as a smoke cluster and written into the smoke mask. .

[0104] Simultaneously, the volume of each smoke cluster is calculated based on the cluster's outer boundary. and its three-dimensional position range These quantities will be used for subsequent stage determinations and weight adjustments.

[0105] Intensity distribution pattern recognition and stage determination involve calculating a set of concise distribution statistics after obtaining the smoke clusters:

[0106] Total volume of smoke cluster (The summation can be calculated based on all smoke clusters)

[0107] Maximum sensing distance (MaxDistance): The farthest distance at which a valid echo can still be obtained in the current frame;

[0108] Invalid Count: The number of mismatched or intensity-deficient points counted at the scan point level;

[0109] Average reflection intensity: The average reflection intensity of all points within a frame.

[0110] Based on the thresholds and segmentation rules of the above indicators, the scene is divided into four stages: no smoke, early smoke, mid-smoke, and late smoke. A typical rule is: when there are no smoke point cloud clusters and the average reflectance intensity is greater than the environmental intensity threshold... (Usually 20.0~50.0) is considered the "smoke-free stage"; when smoke point cloud clusters exist and the total volume of the smoke clusters meets the following conditions... hour( The environmental smoke volume threshold is usually taken as ( This is classified as the "initial stage of smoke"; when there is a smoke point cloud cluster and the total volume of the smoke cluster meets the following conditions... When the number of invalid points increases, the average reflection intensity decreases, and the maximum sensing distance shortens significantly, it is determined to be in the "mid-stage of smoke." As the number of invalid points increases, the average reflection intensity decreases, and the maximum sensing distance shortens significantly, it is determined to be in the "late-stage of smoke." All thresholds allow for on-site calibration and adaptive adjustment based on environmental and equipment parameters, preventing the device from being tied to fixed values. The maximum sensing distance is a core indicator for judging the degree of smoke impact; its rate of decrease directly reflects the weakening of sensor detection capabilities due to smoke diffusion. Simultaneously, the absolute value of this distance provides a crucial basis for the timing and weight allocation of introducing millimeter-wave radar for fusion sensing.

[0111] Two other characteristics serve as auxiliary criteria: the point cloud reflectance intensity typically remains stable in the initial stage, decreases significantly after the smoke appears, and remains at a low level during the diffusion period (average intensity below 10, gradually approaching 0); the number of invalid points increases accordingly, further indicating that the smoke's interception and scattering effect on the laser signal is intensified. These phenomena all stem from the scattering and absorption of laser light by smoke particles—especially in the later stages of smoke development, high concentrations of aerosols lead to multiple scattering and rapid energy attenuation, resulting in a significant degradation of point cloud quality and a substantial decrease in detection range, signal strength, and data efficiency. These three characteristics together constitute a landmark indicator system for determining whether smoke has entered the later stages of smoke development.

[0112] The system publishes the following results:

[0113] 1. Smoke mask Provided with point-level / voxel-level identifiers, and as input for dynamic fusion weight adjustment and point cloud fusion, it is used to enhance millimeter-wave weights in high-smoke regions;

[0114] 2. Geometric quantities: the three-dimensional position and volume of each smoke cluster. It also includes a timestamp and source frame number for path planning and human-machine interface display;

[0115] 3. Stage Labels: Four-stage results and their confidence levels, facilitating system strategy switching;

[0116] 4. Quality indicators: such as the percentage of valid points and the average classification probability, which serve as a reliability reference for subsequent modules.

[0117] The dynamic fusion weight adjustment and point cloud fusion module, based on the output environment stage (none / early / mid / late stage) and smoke mask and quality indicators, performs weight adjustment and region selection on two types of sensors according to a strategy, and constructs a fused point cloud under a unified spatiotemporal coordinate system. It is directly invoked by downstream tasks such as obstacle avoidance, mapping, and target recognition. It consists of two parts: a dynamic fusion weight strategy module and a point cloud fusion construction module, such as... Figure 5 As shown.

[0118] The dynamic fusion weight strategy module selects different fusion strategies based on the environmental stage and outputs the dominant source and weight at the sector / region level for downstream fusion use.

[0119] 1. Smoke-free strategy (laser-dominated). The default is to prioritize laser point clouds, with millimeter waves used to compensate for occasional laser distortion on metallic or black objects. Laser points are weighted highly, while millimeter waves are only increased in close-range or low-intensity small areas.

[0120] 2. Initial Smoke Strategy (Radial Fusion). Based on the distribution of the smoke mask in terms of azimuth and distance, the scene is divided into several sectors along the radial direction (preferably 5°-10° / sector, which can be modified as needed). In the sectors marked by the mask, the millimeter-wave weight is gradually increased to dominate, while laser dominance is maintained in the unmarked sectors.

[0121] 3. Mid-Smoke Strategy (Millimeter-Wave Dominance). During the mid-smoke phase, the ambient smoke concentration reaches its maximum, rendering the entire lidar system ineffective. The strategy immediately switches to global millimeter-wave dominance, with only millimeter-wave point cloud data output during the fusion and construction phase, and no further lidar points are introduced. LiDAR point cloud data is used only for temporal status monitoring and does not participate in spatial data fusion; when necessary, it is used to issue speed reduction or speed-limiting suggestions to the upper-level control to ensure continuous visibility and safety under extreme obstruction conditions.

[0122] 4. Post-Smoke Strategy (Tangential Fusion / Degraded Operation). When entering the post-smoke phase, no new smoke is generated in the scene, and the existing smoke gradually diffuses and becomes more uniform. The strategy employs "tangential fusion": equal-width fan rings (preferably 0.5-1.0m) are divided along the radial direction in polar coordinates, and the maximum effective sensing distance of the laser estimated in the current frame is used. As a threshold, all sector rings within the threshold are dominated by laser, with millimeter waves used for point filling and continuity verification; sector rings outside the threshold are dominated by millimeter waves. A narrow buffer band is set near the threshold to smoothly transition the weights and avoid boundary breaks. This strategy utilizes the characteristics of uniform smoke in the later stage and gradual near-field recovery of laser to achieve a stable switching between "laser within the radius and millimeter waves outside the radius".

[0123] The point cloud fusion building module consists of an input submodule, a sector selection module, and a final fusion generation module.

[0124] The function of this module is to align two types of point clouds that have already been spatiotemporally aligned. The module directly synthesizes a fused point cloud frame with the given sector / sector ring allocation results. Specifically, the input consists of laser point clouds and millimeter-wave point clouds in a unified coordinate system and timestamp, as well as the "dominant source and weights / masks" corresponding to each region provided by the upstream. The module does not create new meshes, but only assigns and routes points according to the upstream results: points falling in laser-dominant regions retain laser data, and millimeter-wave data is only supplemented as necessary; points falling in millimeter-wave-dominant regions retain millimeter-wave data, and laser data is only used to supplement close-range details when needed; within the boundary buffer, the transition is smooth according to the upstream weights. If the stage is "mid-smoke," the pure millimeter-wave point cloud is directly output without cross-source synthesis.

[0125] The output after synthesis is a fused point cloud. It also includes brief metadata (timestamp, current mode, dominant source tag / weight summary) for direct use by downstream tasks.

[0126] This invention addresses the data misalignment problem caused by inconsistent time references and different coordinate systems between lidar and millimeter-wave radar. It proposes a time synchronization scheme using lidar as the true clock (PPS+UTC synchronization), millimeter-wave 10Hz hard triggering, and nearest-neighbor pairing, in conjunction with a unified sensor coordinate system. The spatial synchronization mechanism of external parameter calibration can simultaneously achieve frame-level accurate time synchronization and spatial co-domain representation, greatly enhancing fusion consistency and accuracy, and ensuring the stability and reliability of downstream mapping and obstacle avoidance.

[0127] This invention addresses the inherent problems of multipath scattering, side-leaf echoes, and sparse discrete points in millimeter-wave radar. It constructs a three-stage processing link of "effective field-of-view constraint - intensity adaptive threshold separation - DBSCAN density filtering", which can significantly reduce error points and clutter, greatly enhance the stability and usability of millimeter-wave point clouds, and ensure the quality controllability of the input before fusion.

[0128] This invention addresses the problem that smoke is difficult to characterize geometrically directly. It uses statistical features and learning to generate smoke masks, volumes, and stage labels, which can intuitively quantify the degree of degradation, greatly enhance the interpretability of environmental semantics, and ensure that subsequent strategies are adaptively adjusted according to stages.

[0129] This invention addresses the problem of drastic performance fluctuations of a single sensor under different smoke intensities by proposing a phased fusion approach: initial weighting by azimuth sector; mid-term laser failure and global output of pure millimeter-wave point cloud; and final tangential fusion of near-field laser and far-field millimeter-wave signals by radius sector rings. This approach can cover all operating conditions while maintaining information integrity, greatly enhancing system robustness and ensuring safety and perceptibility.

[0130] This invention addresses the problem of difficult maintenance of complex fusion. It decouples the "control plane and data plane" and directly consumes the upstream sector / sector ring allocation results for routing, deduplication and synthesis. It can generate a unified fused point cloud with low overhead and record the source and weight, which greatly enhances maintainability and traceability, and ensures the implementation and long-term operation of the project.

[0131] The present invention and its embodiments have been described above. This description is not restrictive, and the embodiments shown throughout are only one of the embodiments of the present invention. The actual structure is not limited to this. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, they should all fall within the protection scope of the present invention.

Claims

1. A perception method for a rescue robot in a smoke-filled environment, characterized in that, Includes the following steps: S1. Raw information acquisition and spatiotemporal synchronization: The system acquires raw point cloud data from lidar and millimeter-wave radar, aligns the two types of point clouds on the time axis through a time synchronization mechanism, maps the two types of point clouds to a unified sensor platform coordinate system through a spatial synchronization mechanism, and outputs the synchronized raw point cloud. S2, Millimeter-wave point cloud noise suppression: The synchronized millimeter-wave raw point cloud is sequentially processed with effective field-of-view suppression, Gaussian intensity separation suppression, and density filtering suppression to output a purified millimeter-wave point cloud. S3, Three-dimensional quantitative perception of smoke: Using the synchronized laser point cloud as input, candidate point clusters are obtained through adaptive density clustering. The statistical features of each candidate point cluster are extracted and input into a pre-trained classifier for smoke cluster identification. The three-dimensional position, volume and scene statistical indicators of the smoke clusters are calculated, and the smoke stage of the scene is determined based on the scene statistical indicators. S4. Dynamic fusion weight adjustment and point cloud fusion: Select the corresponding fusion strategy according to the smoke stage, determine the dominant sensor source and weight in different spatial regions, and generate a unified fused point cloud based on the dominant source, weight, purified millimeter-wave point cloud and synchronized original laser point cloud.

2. The perception method for a rescue robot in a smoke-filled environment according to claim 1, characterized in that: The time synchronization mechanism includes time synchronization and spatial synchronization. Spatial synchronization involves establishing rigid external parameters from the lidar coordinate system L and the millimeter-wave radar coordinate system R to the sensor platform coordinate system P, and obtaining translation and rotation parameters through offline calibration. and For any point and As shown in the following formula: ;in, Let A be the set of point clouds represented in the lidar coordinate system 𝐿. Let A be the set of point clouds represented in the millimeter-wave radar coordinate system φ. Let be the 3D coordinates of the nth point in the laser point cloud, in coordinate system φ. Let be the 3D coordinates of the nth point in the radar point cloud, in coordinate system . To separately The three-dimensional coordinates mapped to the platform coordinate system P. Let L be the rotation matrix from L to P. Let L be the translation vector from L to P. Let R be the rotation matrix from R to P. This is a translation vector from R to P; after time and space synchronization processing, the output is... Aligned on the time axis and belonging to the same coordinate system in the spatial domain .

3. The perception method for a rescue robot in a smoke-filled environment according to claim 1, characterized in that: The effective field of view suppression is based on a preset radial effective range. and highly effective range The filtering criteria are: ; Select millimeter-wave points that satisfy both radial distance and height within the corresponding effective range, and discard invalid points that exceed the effective field of view. The output point set is denoted as . ; in, The value range is 0-1.0m. The value range is 10.0-40.0m, both indoors and outdoors. , Indoor ,outdoor It is infinitely large.

4. The perception method for a rescue robot in a smoke-filled environment according to claim 1, characterized in that: The Gaussian intensity separation suppression specifically involves fitting the echo intensity distribution using a Gaussian mixture model (GMM), as shown in the following formula: The parameters are estimated using the expectation-maximization algorithm, and the intensity threshold is set based on the intensity at the intersection of the Gaussian distribution. The points are classified as follows: The preserved point cloud is recorded as .

5. The perception method for a rescue robot in a smoke-filled environment according to claim 1, characterized in that: The density filtering suppression specifically involves using the DBSCAN algorithm to perform spatial density filtering on the point cloud. ;in The number of points in the neighborhood. To be the minimum number of points, Let be the neighborhood radius, then Points marked as core points are expanded using density reachability relationships to form effective clusters; otherwise, they are marked as noise points. The final output set of effective millimeter-wave point clouds is as follows: .

6. The perception method for a rescue robot in a smoke-filled environment according to claim 2, characterized in that: Using point clouds processed with time and space synchronization as input, adaptive clustering, feature extraction, and learning-based discrimination are performed on the point clouds in smoke scenes, and three types of results are output: ① 3D smoke mask and its cluster volume; ② 3D location and range of smoke region; ③ Smoke stage and statistical indicators for system scheduling and fusion.

7. The perception method for a rescue robot in a smoke-filled environment according to claim 6, characterized in that: The smoke stages include no smoke, initial smoke stage, middle smoke stage, and late smoke stage.

8. A perception system for a rescue robot in a smoke-filled environment, characterized in that: include: The raw information acquisition module is used to acquire raw point cloud data from lidar and millimeter-wave radar, and to achieve time and spatial synchronization of the two types of point clouds, and output the synchronized raw point cloud. The millimeter-wave point cloud noise suppression module is used to perform effective field-of-view suppression, Gaussian intensity separation suppression and density filtering suppression on the synchronized original millimeter-wave point cloud in sequence, and output the purified millimeter-wave point cloud. The smoke 3D quantization perception module is used to obtain the 3D position and volume of the smoke mask and smoke clusters through adaptive density clustering, feature extraction and learning-based discrimination, using the synchronized original laser point cloud as input, calculating scene statistical indicators and determining the smoke stage. The dynamic fusion weight adjustment module is used to select the fusion strategy according to the smoke stage, determine the dominant sensor sources and weights in different spatial regions, and generate and output a unified fused point cloud. The data output module is used for data output.

9. The perception system for a rescue robot in a smoke-filled environment according to claim 8, characterized in that: The raw information acquisition module includes a time synchronization unit and a spatial synchronization unit.

10. The perception system for a rescue robot in a smoke-filled environment according to claim 8, characterized in that: The dynamic fusion weight adjustment module includes a dynamic fusion weight strategy unit and a point cloud fusion construction unit; the dynamic fusion weight strategy unit is used to output the corresponding sector / region-level dominant source and weight according to the smoke stage. The point cloud fusion construction unit is used to receive two types of point clouds with spatiotemporal alignment and dominant source weight information, and generate a fused point cloud through point attribution routing and synthesis.

Images