Natural gas station-oriented explosion-proof robot multi-modal leakage perception method and system

Abstract

The application discloses a natural gas station-oriented explosion-proof robot multi-modal leakage sensing method and system, relates to the field of sensor fusion, and comprises the following steps: constructing a unified space-time coordinate system with the robot centroid as the origin, and synchronously fusing laser methane, infrared thermal imaging and sound wave sensor data. Abnormal fields of concentration, temperature and sound pressure are obtained through differential processing; the gas diffusion direction is determined based on the concentration gradient, and suspected leakage areas are preliminarily identified through the analysis of the coincidence of sound source positioning and low-temperature abnormal areas; the robot collects local temperature texture and concentration cumulative curves for space-time alignment verification, and accurately locates the real leakage port after excluding false signals; finally, the risk level is determined according to the concentration and sound energy value of the leakage port, and structured alarm information is output. The application has the advantages that through multi-sensor space-time registration and baseline differential noise reduction, multi-field cross verification coarse positioning and near space-time feature review, high-precision, low-false-alarm autonomous inspection and accurate traceability grading of industrial gas leakage are realized.

Description

Technical Field

[0001] This invention relates to the field of sensor fusion, and in particular to a multimodal leak detection method and system for explosion-proof robots in natural gas stations. Background Technology

[0002] Natural gas stations are high-risk areas with flammable and explosive components. Pipelines, valves, pressure regulating equipment, and flange joints are prone to methane leaks due to corrosion, aging, and seal failure. If not handled promptly, these leaks can easily lead to explosions, placing immense pressure on safety management. Currently, leak detection at stations relies mainly on regular manual inspections, fixed single-point gas detectors, and single portable detection equipment, which have significant technical limitations.

[0003] Current gas leak detection robot solutions on the market generally have significant shortcomings. Most rely solely on a single laser methane sensor or infrared thermal imaging for independent detection, lacking a collaborative sensing mechanism using an acoustic array. This prevents unified spatiotemporal registration of multi-source data, as sensors operate in independent coordinate systems with asynchronous sampling times, resulting in chaotic data fusion benchmarks and inherently weak source tracing logic. Conventional equipment relies solely on factory-fixed baseline thresholds to determine leaks, failing to dynamically update environmental concentration, temperature, and sound pressure background baselines during robot movement. This makes it highly susceptible to false alarms due to on-site ventilation disturbances, equipment heat dissipation, mechanical vibrations, and ambient methane levels, and it lacks differential noise reduction capabilities. At the positioning level, it relies solely on concentration gradients to roughly determine the approximate direction, lacking location verification through the intersection of acoustic delay hyperbolic curves and coupling verification of low-temperature anomaly areas. It cannot distinguish interference sources such as low-temperature condensation in pipelines, abnormal fan noises, and valve vibrations, leading to a high proportion of false leak areas. Furthermore, few designs incorporate close-range micro-verification processes by the robot, failing to eliminate false targets through spatiotemporal alignment of temperature textures and concentration accumulation inflection points, resulting in poor accuracy in leak point coordinate positioning. In addition, most existing solutions can only qualitatively determine whether there is a leak, and cannot collect the absolute value of the concentration at the leak point and the characteristic sound energy for hierarchical quantification. They can only simply upload alarm signals and lack standardized traceable data frames with coordinates and robot pose. Overall, they have weak resistance to interference from complex industrial environments, low positioning accuracy, high false alarm rate, and insufficient risk quantification capabilities, making it difficult to meet the needs of refined site inspection and hierarchical handling of hidden dangers. Summary of the Invention

[0004] To improve existing methods and systems, this paper provides a multimodal leak detection method and system for explosion-proof robots in natural gas stations. This method achieves high-precision, low-false-alarm autonomous inspection and accurate source tracing and classification of industrial gas leaks by using multi-sensor spatiotemporal registration and baseline differential noise reduction, multi-field cross-validation coarse positioning and close-range spatiotemporal feature verification.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A multimodal leak detection method for explosion-proof robots in natural gas stations includes:

[0007] Simultaneously activate the laser methane sensor, infrared thermal imaging sensor, and acoustic sensor, establish a spatial coordinate system with the robot's center of mass as the origin, and map the field of view of each sensor to this coordinate system so that the output concentration, temperature, and sound pressure data carry spatial orientation labels and unified time labels.

[0008] During the continuous movement of the robot, the environmental concentration baseline, background temperature baseline, and environmental sound pressure baseline of each sensor are extracted respectively. The current measured signal is differentially canceled with the corresponding baseline to obtain the methane differential concentration field, differential temperature field, and differential sound pressure field.

[0009] The main direction of gas cloud diffusion is determined based on the gradient direction of the methane differential concentration field. The candidate area of ​​the sound source is obtained by hyperbolic intersection based on the time delay difference of the sound wave arriving at each array element. The overlap between the candidate area of ​​the sound source and the low temperature anomaly area in the differential temperature field is determined. If the overlap exceeds the threshold, it is marked as a suspected leakage source area.

[0010] The robot is controlled to approach the suspected area, and the infrared sensor is switched to macro mode to collect the local temperature field. At the same time, the laser sensor accumulates the concentration at a fixed point. The temperature field texture change and the inflection point of the concentration accumulation curve are aligned on the time axis and compared with the spatial coordinates. After filtering out false sound sources, the location of the real leak is determined.

[0011] The absolute concentration value and specific frequency band energy value of the sound wave at the actual leak point are extracted using the actual leak point location as the origin. The risk level is determined by inputting the data into a classification mapping table, and the level, location coordinates and robot pose are encoded into a data frame for output.

[0012] Preferably, the step of mapping the field of view of each sensor to this coordinate system, so that the output concentration, temperature, and sound pressure data carry spatial orientation labels and unified time labels, specifically includes:

[0013] Calibrate the installation offset, pitch angle, and azimuth angle of the laser methane sensor (both the through and open ends), infrared sensor, and acoustic sensor relative to the robot's center of mass.

[0014] A spatial coordinate system is established with the robot's direction of travel as the positive X-axis, the vertical upward direction as the positive Z-axis, and the robot's center of mass as the origin. The three-dimensional coordinates of the detection center point of each sensor in the spatial coordinate system are calculated based on the installation eccentricity and installation attitude angle of each sensor.

[0015] Each frame of data is bound to its corresponding 3D coordinates as a spatial label, and the sampling clocks of each sensor are aligned so that data frames at the same time carry the same global clock count as time labels.

[0016] Preferably, the step of extracting the environmental concentration baseline, background temperature baseline, and environmental sound pressure baseline of each sensor, and performing differential cancellation between the current measured signal and the corresponding baseline specifically includes:

[0017] Under the calibration conditions where the robot is stationary and there is no leakage at the site, the average of multiple concentration values ​​from the laser sensor is taken as the environmental concentration baseline, the statistical average of the temperature values ​​of each pixel from the infrared sensor is taken as the background temperature baseline, and the median value of the sound pressure level of each frequency band from the acoustic sensor is taken as the environmental sound pressure baseline.

[0018] During the robot's movement, the current concentration value, the temperature value of each pixel, and the sound pressure level of each frequency band are subtracted from the corresponding baseline to obtain the methane differential concentration value, differential temperature value, and differential sound pressure value.

[0019] When the methane differential concentration value continuously exceeds the preset noise floor threshold for a duration of one time, the methane differential concentration value, differential temperature value, and differential sound pressure value are output as the corresponding field data and the subsequent coarse localization is triggered.

[0020] Preferably, the step of determining the main diffusion direction of the gas cloud based on the gradient direction of the methane differential concentration field, and obtaining the candidate sound source region by hyperbolic intersection based on the time delay difference of the sound wave arriving at each array element, specifically includes:

[0021] Using the robot's horizontal projection point as the center, the methane differential concentration field is divided into sectors on the horizontal plane in equal circular directions. The radial decay rate of the concentration in each sector is calculated, and the direction of the sector with the smallest decay rate is determined as the main diffusion direction.

[0022] Using any array element as a reference, calculate the cross-correlation function between the received signals of the other array elements and the reference signal, and extract the delay time corresponding to each peak as the arrival delay difference of the sound wave.

[0023] Take any two array elements as a group, calculate the distance difference between the sound source and the two array elements based on the time delay difference and the speed of sound, draw a hyperbola with the two array elements as the focus on the horizontal plane, take the common intersection of at least three hyperbolas as the center of the sound source candidate area, and draw a circle with the center as the center and a preset radius to obtain the sound source candidate area.

[0024] Preferably, the step of determining the overlap between the candidate sound source region and the low-temperature anomaly region in the differential temperature field, and marking it as a suspected leakage source region if the overlap exceeds a threshold, specifically includes:

[0025] Traverse all pixels in the differential temperature field, extract continuous pixel regions with differential temperature values ​​below a preset negative threshold as low-temperature anomaly regions, and calculate the first coordinate of the geometric centroid of the low-temperature anomaly region in the spatial coordinate system.

[0026] Extract the second coordinate of the geometric center of the candidate sound source region in the spatial coordinate system, and calculate the Euclidean distance between the first coordinate and the second coordinate.

[0027] Calculate the overlapping area of ​​the candidate sound source region and the low temperature anomaly region on the horizontal plane projection, as well as the total area of ​​their union. When the Euclidean distance is less than the first threshold and the ratio of the overlapping area to the total area is greater than the second threshold, the overlap is determined to exceed the threshold, and the overlapping area is marked as a suspected leakage source region.

[0028] Preferably, when the controlled robot approaches the suspected area, it switches the infrared sensor to macro mode to collect the local temperature field, while the laser sensor accumulates the concentration at a fixed point. Specifically, this includes:

[0029] When the horizontal distance between the robot's center of mass and the geometric center of the suspected leak source area is less than the preset approach distance, the robot is controlled to hover so that the distance between the front detection surface of the infrared thermal imaging sensor and the surface of the suspected leak source area is not greater than the preset micro detection distance.

[0030] During the robot's hovering period, the laser methane sensor is controlled to continuously perform cumulative concentration measurements. The cumulative measurement time covers three consecutive concentration fluctuation cycles, and the concentration measurement values ​​at each sampling time are arranged in chronological order to form a cumulative concentration rise curve.

[0031] Using the robot's hovering moment as the alignment zero point, the timestamps of each frame of the local temperature field image are matched with the timestamps of each sampling point of the concentration accumulation curve by counting with the same global clock, thus establishing a one-to-one time correlation between each frame of the temperature field image and each concentration sampling value.

[0032] Preferably, the step of aligning the inflection points of the temperature field texture change and the concentration accumulation curve with time axis and comparing spatial coordinates to filter out false sound sources and determine the location of the actual leak specifically includes:

[0033] Extract the boundary pixels of the transition zone from the high temperature region to the low temperature region in the local temperature field, fit the boundary pixels into a closed contour line, and calculate the third coordinate of the geometric center of the closed contour line in the spatial coordinate system.

[0034] Calculate the second derivative of the concentration accumulation curve, mark the time point when the second derivative changes abruptly to positive or negative as the inflection point, extract the spatial coordinates of the concentration sampling value corresponding to the inflection point, and mark them as the coordinates of the concentration change location.

[0035] Calculate the projection deviation between the third coordinate and the coordinate of the concentration change location in the spatial coordinate system. If the projection deviation is less than the third threshold, the geometric center of the closed contour line is determined as the actual leak location and output.

[0036] If the projection deviation value is greater than or equal to the third threshold, the current candidate area of ​​the sound source is determined to be a false sound source, the current suspected leakage source area is abandoned, and other candidate areas are selected again.

[0037] Preferably, the step of extracting the absolute concentration value and the energy value of the sound wave at a specific frequency band at the actual leak point, and inputting them into a classification mapping table to determine the risk level specifically includes:

[0038] After applying a Hanning window to the time-domain sound pressure signal acquired by the acoustic sensor, a fast Fourier transform is performed to obtain the frequency domain spectrum. Frequency components within the characteristic frequency band of natural gas leakage are extracted, and the sum of the squares of the amplitudes at each discrete frequency point is taken as the energy value of a specific frequency band.

[0039] The graded mapping table includes four levels: normal, warning, alarm, and emergency. Each level corresponds to a combination of a concentration threshold range and an energy threshold range, and the ranges of each level are connected end to end.

[0040] The absolute concentration value and the energy value of a specific frequency band at the actual leak point are used as a pair. The concentration threshold range and energy threshold range corresponding to the four levels are traversed. When the pair falls into both ranges of a certain level, the level is determined as the current leakage risk level.

[0041] The risk level, the spatial coordinates of the actual leak location, and the robot's current pose information are packaged into a structured message, and a current global timestamp is appended to the message header before being sent to the site control platform.

[0042] Furthermore, a multimodal leak detection system for explosion-proof robots in natural gas stations is proposed, including:

[0043] Multimodal sensing synchronous acquisition module: integrates laser methane, infrared thermal imaging, and acoustic wave sensors to complete sensor attitude calibration and field of view coordinate system mapping, and unify data spatiotemporal labels;

[0044] Baseline differential preprocessing module: Adaptively extracts the baseline of environmental concentration, temperature and sound pressure at the site, performs differential cancellation on the real-time monitoring data, removes environmental background noise interference, outputs clean differential field data and triggers initial leakage judgment;

[0045] Multi-source information coarse localization module: Combines the concentration field gradient to determine the gas diffusion direction, locks the candidate area of ​​the sound source by the intersection of the hyperbola of the sound wave time delay difference, and completes the overlap determination by linking with the low temperature anomaly area, marking the suspected leak area;

[0046] Precise verification and fake detection module: Controls the robot to hover and detect at close range, matches the inflection point of temperature field texture change and concentration accumulation curve, compares in two dimensions of time and space, filters out false leakage sound sources, and locks the real leakage point;

[0047] Risk classification and assessment module: Extracts the concentration and characteristic frequency band sound wave energy at the leak point, and completes the four-level risk assessment based on the classification mapping table to quantify the leakage hazard level;

[0048] Data transmission output module: integrates risk level, leakage coordinates, robot pose and timestamp, encapsulates them into standardized data frames, and stably uploads them to the site control platform to complete the data closed-loop output.

[0049] Compared with the prior art, the advantages of the present invention are:

[0050] By unifying the coordinate system with the robot's center of mass and using unified timestamp labels for all data, the fusion deviation caused by spatial reference misalignment and asynchronous sampling timing of multiple sensors is completely resolved. Dynamic adaptive baseline differential cancellation during the motion process effectively eliminates environmental interference such as site temperature drift, airflow noise, and background methane levels, greatly reducing false triggering caused by baseline drift under complex working conditions. The system integrates three judgment logics: methane concentration gradient gas mass tracing, acoustic TDOA hyperbola intersection sound source localization, and infrared low-temperature abnormal area overlap verification. It first coarsely screens suspected leak areas, and then uses robot close-range micro-infrared imaging and fixed-point laser concentration accumulation curve spatiotemporal alignment verification to filter false sound sources such as equipment vibration and environmental noise from both temperature texture abrupt changes and concentration inflection points, significantly improving the reliability of positioning. Ultimately, the risk level is determined by the measured concentration at the leak point and the quantitative classification of the sound frequency energy. The system outputs a standardized data frame with coordinates and robot pose, which has the ability to accurately locate the leak point in three dimensions, quantitatively rate the degree of leakage, and trace and archive inspection data. Compared with single sensor detection, the anti-interference ability, positioning accuracy, and control level of missed detection and false alarm are significantly optimized, making it suitable for the all-weather autonomous inspection operation of the station pipeline network. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the multimodal leak detection method for explosion-proof robots in natural gas stations proposed in this invention.

[0052] Figure 2 This is a schematic diagram of baseline extraction and differential cancellation proposed in this invention;

[0053] Figure 3 This is a schematic diagram of obtaining the candidate region of the sound source proposed in this invention. Detailed Implementation

[0054] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0055] A multimodal leak detection system for explosion-proof robots used in natural gas stations includes:

[0056] Multimodal sensing synchronous acquisition module: integrates laser methane, infrared thermal imaging, and acoustic wave sensors to complete sensor attitude calibration and field of view coordinate system mapping, and unify data spatiotemporal labels;

[0057] Baseline differential preprocessing module: Adaptively extracts the baseline of environmental concentration, temperature and sound pressure at the site, performs differential cancellation on the real-time monitoring data, removes environmental background noise interference, outputs clean differential field data and triggers initial leakage judgment;

[0058] Multi-source information coarse localization module: Combines the concentration field gradient to determine the gas diffusion direction, locks the candidate area of ​​the sound source by the intersection of the hyperbola of the sound wave time delay difference, and completes the overlap determination by linking with the low temperature anomaly area, marking the suspected leak area;

[0059] Precise verification and fake detection module: Controls the robot to hover and detect at close range, matches the inflection point of temperature field texture change and concentration accumulation curve, compares in two dimensions of time and space, filters out false leakage sound sources, and locks the real leakage point;

[0060] Risk classification and assessment module: Extracts the concentration and characteristic frequency band sound wave energy at the leak point, and completes the four-level risk assessment based on the classification mapping table to quantify the leakage hazard level;

[0061] Data transmission output module: integrates risk level, leakage coordinates, robot pose and timestamp, encapsulates them into standardized data frames, and stably uploads them to the site control platform to complete the data closed-loop output.

[0062] See Figure 1 As shown, a multimodal leak detection method for explosion-proof robots in natural gas stations includes:

[0063] Simultaneously activate the laser methane sensor, infrared thermal imaging sensor, and acoustic sensor, establish a spatial coordinate system with the robot's center of mass as the origin, and map the field of view of each sensor to this coordinate system so that the output concentration, temperature, and sound pressure data carry spatial orientation labels and unified time labels.

[0064] During the continuous movement of the robot, the environmental concentration baseline, background temperature baseline, and environmental sound pressure baseline of each sensor are extracted respectively. The current measured signal is differentially canceled with the corresponding baseline to obtain the methane differential concentration field, differential temperature field, and differential sound pressure field.

[0065] The main direction of gas cloud diffusion is determined based on the gradient direction of the methane differential concentration field. The candidate area of ​​the sound source is obtained by hyperbolic intersection based on the time delay difference of the sound wave arriving at each array element. The overlap between the candidate area of ​​the sound source and the low temperature anomaly area in the differential temperature field is determined. If the overlap exceeds the threshold, it is marked as a suspected leakage source area.

[0066] The robot is controlled to approach the suspected area, and the infrared sensor is switched to macro mode to collect the local temperature field. At the same time, the laser sensor accumulates the concentration at a fixed point. The temperature field texture change and the inflection point of the concentration accumulation curve are aligned on the time axis and compared with the spatial coordinates. After filtering out false sound sources, the location of the real leak is determined.

[0067] The absolute concentration value and specific frequency band energy value of the sound wave at the actual leak point are extracted using the actual leak point location as the origin. The risk level is determined by inputting the data into a classification mapping table, and the level, location coordinates and robot pose are encoded into a data frame for output.

[0068] Specifically, during the calibration process, a carrier spatial coordinate system is established with the robot's center of mass as the origin, the robot's direction of travel as the positive X-axis, the vertical upward direction as the positive Z-axis, and the positive Y-axis determined according to the right-hand rule. The three-dimensional mounting eccentricity of each sensor relative to the robot's center of mass, as well as the azimuth angle of each sensor's optical axis or detection center axis relative to the X-axis of the carrier coordinate system and its pitch angle relative to the horizontal plane, are measured and recorded.

[0069] For any type of sensor, let the components of its mounting eccentricity along the three axes of the carrier coordinate system be respectively... , , Given an installation azimuth angle of α and an installation elevation angle of β, the three-dimensional coordinates (X, Y, Z) of the sensor's detection center point in the carrier's spatial coordinate system are calculated using the following formula, based on the rigid body coordinate transformation relationship:

[0070]

[0071] in, This represents the rotation matrix about the Z-axis of the carrier coordinate system, corresponding to the rotation transformation of the azimuth angle α; This represents the rotation matrix about the Y-axis, corresponding to the rotation transformation of the pitch angle β; coordinates ( , , The coordinates of the robot's center of mass in the global reference coordinate system are provided by the robot localization module. The coordinates of the center points detected by each sensor are updated in real time as the robot moves.

[0072] For infrared thermal imaging sensors, a pyramid-shaped field-of-view prism is constructed based on the coordinates of its detection center point as the origin, according to its horizontal and vertical field-of-view angles. The spatial region covered by the field-of-view prism is then registered with the carrier's spatial coordinate system, ensuring that each pixel in each frame of temperature image output by the sensor is mapped back to its specific spatial orientation angle in the carrier's spatial coordinate system. For acoustic sensors, the geometric center of its pickup array elements is used as the reference point, and the spatial coordinates of each of the four array elements are bound to the carrier's spatial coordinate system.

[0073] Each frame of concentration measurement output from the laser methane sensor, each frame of temperature image output from the infrared thermal imaging sensor, and each frame of time-domain sound pressure signal output from the acoustic wave sensor are accompanied by corresponding three-dimensional coordinates of the detection center point and a field-of-view direction vector as spatial orientation labels. Simultaneously, the internal clocks of each sensor are synchronized using the IEEE 1588 precise time synchronization protocol, ensuring that data frames acquired by each sensor under the same trigger pulse carry the same global clock count value as a unified time label.

[0074] See Figure 2 As shown, after the explosion-proof robot first enters the site or after each fixed operating period, under normal operating conditions, the robot is controlled to travel to a typical detection position within the site and remain stationary. In this stationary calibration state, the laser methane sensor continuously collects N frames of concentration data, and the concentration of the i-th frame is denoted as... Then the environmental concentration baseline Take the arithmetic mean of N frames of data:

[0075]

[0076] The value of N is set according to the sensor sampling frequency to ensure coverage of at least two complete airflow fluctuation cycles, thus smoothing out instantaneous random disturbances. For infrared thermal imaging sensors, each pixel within its field of view independently establishes a background temperature baseline. Let the pixel coordinates be (u, v), and under calibration conditions, M frames of temperature values ​​for that pixel are continuously acquired. Then the background temperature baseline of that pixel Take the statistical mean of the temperature values ​​of M frames:

[0077]

[0078] The background temperature baseline of all pixels across the entire field of view forms a background temperature baseline map with the same resolution as the thermal imaging. For acoustic sensors, the acquired time-domain sound pressure signal is converted into a frequency-domain sound pressure level spectrum using a fast Fourier transform. For each frequency band, the median value of the sampled sound pressure level is taken as the environmental sound pressure baseline for that frequency band. , where f represents the frequency index.

[0079] During the subsequent continuous motion detection of the robot, the measured concentration value c(t) acquired at each time t is subtracted from the corresponding environmental concentration baseline. The methane differential concentration value Δc(t) was obtained:

[0080]

[0081] Similarly, for each frame of temperature image output by the infrared thermal imaging sensor at the current moment, the base value of the corresponding pixel in the background temperature base map is subtracted pixel by pixel to obtain the differential temperature value. :

[0082]

[0083] The differential temperature values ​​of all pixels form a differential temperature field. The measured sound pressure level S(t,f) of each frequency band at the current moment of the acoustic sensor is subtracted from the corresponding ambient sound pressure baseline. The differential sound pressure value ΔS(t,f) is obtained, and the differential sound pressure values ​​of all frequency bands constitute the differential sound pressure field.

[0084] A sliding window was used to monitor the methane differential concentration value Δc(t), and it was recorded that when it continuously exceeded the preset noise floor threshold. The number of times the threshold is exceeded. When the cumulative time exceeding the threshold reaches a preset first duration... When, i.e., Δc(t) > Duration ≥ If a leak is detected, the entire methane differential concentration field, differential temperature field, and differential sound pressure field calculated at the current moment are output together, triggering the subsequent coarse leak source localization step. If the time limit is exceeded continuously... If the signal is not triggered, it will be considered a transient fluctuation in environmental noise and will not be triggered; the monitoring of data at the next moment will continue.

[0085] See Figure 3 As shown, in the output methane differential concentration field, the horizontal plane is divided into M equal sectors along the circumference, with the projection point of the robot's current position on the horizontal plane as the center. For each sector, the methane differential concentration values ​​at various distances within the sector are extracted radially outward from the center. A linear regression method is used to calculate the slope of the concentration value as a function of radial distance; the absolute value of this slope is the radial decay rate of the concentration in that sector. Let L radial distance points be collected within sector i, and the radial distance of the k-th point be... Concentration value Then the decay rate The formula for calculation is:

[0086]

[0087] in, Let be the radial distance from the k-th sampling point to the center of the circle. This represents the methane differential concentration at that point. and These are the arithmetic mean of the radial distances and concentration values ​​of L sampling points, respectively. The smaller the absolute value of this slope, the slower the concentration decreases with distance in that direction, indicating a stronger diffusion capacity of the air mass. The sector direction that yields the minimum value is determined as the main direction of air mass diffusion. This direction serves as a priori azimuth constraint for subsequent candidate region selection of sound sources.

[0088] The acoustic sensor consists of a rectangular array of four pickup elements, with any one element serving as the reference element R and the other three elements serving as measurement elements. Within the time window, the sound pressure signal x(t) received by the reference array element and the sound pressure signal received by the measurement array element are extracted. Calculate the cross-correlation function between the two. :

[0089]

[0090] Where τ is the time delay variable. The value of τ corresponding to the maximum value of the cross-correlation function is the time difference between the sound wave arriving at the reference element R and the sound wave arriving at the measurement element. The time delay difference between .

[0091] By grouping any two array elements together, and using the position coordinates of the elements in the group and the measured time delay difference, the distance difference from the sound source to the two elements in the group can be calculated based on the speed of sound v. On the horizontal plane, this distance difference defines a hyperbola with the two array elements as foci and a real axis length of d. The intersection of the hyperbolas drawn from at least three different array element combinations is the geometric center coordinate of the candidate sound source region. A circle with this geometric center as its center and a predetermined radius... Draw a circle to obtain the candidate region for the sound source.

[0092] Iterate through all pixels in the differential temperature field, and extract continuous pixel regions with differential temperature values ​​below a preset negative threshold as low-temperature anomaly regions. These regions correspond to localized cooling caused by heat absorption from natural gas leaks. Calculate the first coordinate of the geometric centroid of this low-temperature anomaly region in the spatial coordinate system. Simultaneously, the second coordinate of the geometric center of the candidate sound source region in the spatial coordinate system is extracted. .

[0093] calculate and The Euclidean distance D between them is calculated, and the overlapping area of ​​the candidate sound source region and the low temperature anomaly region on the horizontal plane projection is also calculated. and the total area of ​​their union When D is less than the preset first threshold and If the overlap of the two modes exceeds the preset second threshold, it is determined that the spatial overlap of the localization results exceeds the threshold. At this time, the overlapping area is marked as a suspected leakage source area, and the geometric boundary of the area is output for subsequent fine calibration steps. If the determination fails, the current candidate is abandoned, the sector division or array element combination is readjusted and recalculated.

[0094] The robot plans its path based on the geometric center coordinates of the suspected leak source area. When the Euclidean distance between its own center of mass and this center on the horizontal plane is less than a preset approach distance, the robot brakes and hovers. Then, it controls the extension of the robotic arm equipped with an infrared thermal imaging sensor, so that the sensor's front detection surface is no more than a preset macro distance from the surface of the suspected source area, and switches to macro detection mode to acquire a sequence of local temperature field images around the leak opening with high spatial resolution. Each frame of the image carries a global timestamp.

[0095] During hovering, the open optical path detection end of the laser methane sensor remains aligned with the suspected source region, performing fixed-point concentration accumulation measurements. The accumulation time covers at least three concentration fluctuation cycles, recording the results at each sampling moment. concentration value Arranged chronologically, forming a cumulative concentration increase curve. (Based on hovering time...) To align the zero point in time, the timestamp of each temperature image frame is compared with the timestamp of the concentration sampling point. By matching global clock counts, a one-to-one correspondence is established between temperature images and concentration sample values.

[0096] For each frame of the temperature image in macro mode, calculate the temperature gradient magnitude at each pixel (u,v). :

[0097]

[0098] Where T is the temperature value. and These are the partial derivatives in the horizontal and vertical directions, respectively. A preset gradient threshold is used. Extract the boundary pixels of the transition zone from the high-temperature region to the low-temperature region based on the temperature gradient, fit them into a closed contour line using the least squares ellipse fitting method, and calculate the coordinates of the geometric center of this contour line in the carrier space coordinate system. .

[0099] Concentration accumulation curve Calculate its approximate second derivative value:

[0100]

[0101] Where Δt is the interval between adjacent samplings. The time point at which the second derivative undergoes a sudden change from positive to negative is marked as the inflection point. Extract the horizontal coordinates of the concentration sampling spatial location at that moment. .

[0102] Calculate the projection deviation of the geometric center of the contour line and the location of the concentration abrupt change onto the horizontal plane:

[0103]

[0104] like Less than the preset third threshold If the spatial alignment between the center of the cold zone edge and the location of the concentration abrupt change is determined, C is identified as the actual leak location and output; if ≥ If the current suspected area is identified as a false sound source, the process is abandoned and the coarse localization step is returned to select other candidate areas for a new round of fine-tuning.

[0105] The acoustic sensor continuously acquires the time-domain sound pressure signal near the leak during hovering and fixed-point measurement, and its time-domain sequence is denoted as follows. Where N is the number of sampling points. A Hanning window function w(n) is applied to p(n) to suppress spectral leakage; the window function expression is: Signal after windowing The frequency domain complex spectrum X(k) is obtained by fast Fourier transform, where k is the frequency discrete index, corresponding to the actual frequency. , The sampling rate.

[0106] Based on the acoustic characteristics of natural gas leaks, a lower limit for the characteristic frequency band is pre-set in the storage unit. and upper limit Extracting from the frequency domain spectrum that satisfies... ≤ ≤ For all discrete frequency points, calculate the sum of squared amplitudes at each point as the energy value E for a specific frequency band:

[0107]

[0108] Among them, | | is the frequency The complex amplitude modulus at a given point, E, is in units of amplitude squared and represents the total energy intensity of the leaking sound wave within its characteristic frequency band.

[0109] A risk classification mapping table is pre-stored in the robot's local memory, dividing the risk into four progressive levels: normal, warning, alarm, and emergency. Each level corresponds to a set of concentration threshold intervals and energy threshold intervals, where the concentration and energy intervals are ordered in ascending order and are contiguous, with no overlap or discontinuity between adjacent level intervals. The absolute concentration value output by the laser methane sensor at the actual leak location is denoted as C, and combined with the previously calculated E, forming a binary tuple (C, E). The concentration and energy intervals corresponding to the four levels are compared sequentially in a traversal manner: if C falls into the concentration interval of level i and E falls into the energy interval of the same level, the current leak status is determined to belong to level i, and the level code L=i is denoted (i=1,2,3,4 correspond to normal, warning, alarm, and emergency, respectively).

[0110] After the determination is completed, the level code L and the three-dimensional coordinates of the actual leak point in the carrier space coordinate system are recorded. The robot's current pose is packaged into a structured data frame. A global timestamp is appended to the frame header, and the data is sent to the site's central control platform for monitoring personnel to view in real time and to trigger emergency response systems.

[0111] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0112] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0113] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal leak detection method for explosion-proof robots in natural gas stations, characterized in that, include: Simultaneously activate the laser methane sensor, infrared thermal imaging sensor, and acoustic sensor, establish a spatial coordinate system with the robot's center of mass as the origin, and map the field of view of each sensor to this coordinate system so that the output concentration, temperature, and sound pressure data carry spatial orientation labels and unified time labels. During the continuous movement of the robot, the environmental concentration baseline, background temperature baseline, and environmental sound pressure baseline of each sensor are extracted respectively. The current measured signal is differentially canceled with the corresponding baseline to obtain the methane differential concentration field, differential temperature field, and differential sound pressure field. The main direction of gas cloud diffusion is determined based on the gradient direction of the methane differential concentration field. The candidate area of ​​the sound source is obtained by hyperbolic intersection based on the time delay difference of the sound wave arriving at each array element. The overlap between the candidate area of ​​the sound source and the low temperature anomaly area in the differential temperature field is determined. If the overlap exceeds the threshold, it is marked as a suspected leakage source area. The robot is controlled to approach the suspected area, and the infrared sensor is switched to macro mode to collect the local temperature field. At the same time, the laser sensor accumulates the concentration at a fixed point. The temperature field texture change and the inflection point of the concentration accumulation curve are aligned on the time axis and compared with the spatial coordinates. After filtering out false sound sources, the location of the real leak is determined. The absolute concentration value and specific frequency band energy value of the sound wave at the actual leak point are extracted using the actual leak point location as the origin. The risk level is determined by inputting the data into a classification mapping table, and the level, location coordinates and robot pose are encoded into a data frame for output.

2. The multimodal leak detection method for explosion-proof robots in natural gas stations according to claim 1, characterized in that, The process of mapping the field of view of each sensor to this coordinate system, so that the output concentration, temperature, and sound pressure data carry spatial orientation labels and unified time labels, specifically includes: Calibrate the installation offset, pitch angle, and azimuth angle of the laser methane sensor (both the through and open ends), infrared sensor, and acoustic sensor relative to the robot's center of mass. A spatial coordinate system is established with the robot's direction of travel as the positive X-axis, the vertical upward direction as the positive Z-axis, and the robot's center of mass as the origin. The three-dimensional coordinates of the detection center point of each sensor in the spatial coordinate system are calculated based on the installation eccentricity and installation attitude angle of each sensor. Each frame of data is bound to its corresponding 3D coordinates as a spatial label, and the sampling clocks of each sensor are aligned so that data frames at the same time carry the same global clock count as time labels.

3. The multimodal leak detection method for explosion-proof robots in natural gas stations according to claim 1, characterized in that, The extraction of environmental concentration baseline, background temperature baseline, and environmental sound pressure baseline from each sensor, and the differential cancellation between the current measured signal and the corresponding baseline, specifically includes: Under the calibration conditions where the robot is stationary and there is no leakage at the site, the average of multiple concentration values ​​from the laser sensor is taken as the environmental concentration baseline, the statistical average of the temperature values ​​of each pixel from the infrared sensor is taken as the background temperature baseline, and the median value of the sound pressure level of each frequency band from the acoustic sensor is taken as the environmental sound pressure baseline. During the robot's movement, the current concentration value, the temperature value of each pixel, and the sound pressure level of each frequency band are subtracted from the corresponding baseline to obtain the methane differential concentration value, differential temperature value, and differential sound pressure value. When the methane differential concentration value continuously exceeds the preset noise floor threshold for a duration of one time, the methane differential concentration value, differential temperature value, and differential sound pressure value are output as the corresponding field data and the subsequent coarse localization is triggered.

4. The multimodal leak detection method for explosion-proof robots in natural gas stations according to claim 1, characterized in that, The process of determining the main diffusion direction of the gas cloud based on the gradient direction of the methane differential concentration field, and obtaining the candidate sound source region by hyperbolic intersection based on the time delay difference of sound waves arriving at each array element, specifically includes: Using the robot's horizontal projection point as the center, the methane differential concentration field is divided into sectors on the horizontal plane in equal circular directions. The radial decay rate of the concentration in each sector is calculated, and the direction of the sector with the smallest decay rate is determined as the main diffusion direction. Using any array element as a reference, calculate the cross-correlation function between the received signals of the other array elements and the reference signal, and extract the delay time corresponding to each peak as the arrival delay difference of the sound wave; Take any two array elements as a group, calculate the distance difference between the sound source and the two array elements based on the time delay difference and the speed of sound, draw a hyperbola with the two array elements as the focus on the horizontal plane, take the common intersection of at least three hyperbolas as the center of the sound source candidate area, and draw a circle with the center as the center and a preset radius to obtain the sound source candidate area.

5. The multimodal leak detection method for explosion-proof robots in natural gas stations according to claim 1, characterized in that, The step of determining the overlap between the candidate sound source region and the low-temperature anomaly region in the differential temperature field, and marking it as a suspected leakage source region if it exceeds a threshold, specifically includes: Traverse all pixels in the differential temperature field, extract continuous pixel regions with differential temperature values ​​below a preset negative threshold as low-temperature anomaly regions, and calculate the first coordinate of the geometric centroid of the low-temperature anomaly region in the spatial coordinate system. Extract the second coordinate of the geometric center of the candidate sound source region in the spatial coordinate system, and calculate the Euclidean distance between the first coordinate and the second coordinate. Calculate the overlapping area of ​​the candidate sound source region and the low temperature anomaly region on the horizontal plane projection, as well as the total area of ​​their union. When the Euclidean distance is less than the first threshold and the ratio of the overlapping area to the total area is greater than the second threshold, the overlap is determined to exceed the threshold, and the overlapping area is marked as a suspected leakage source region.

6. The multimodal leak detection method for explosion-proof robots in natural gas stations according to claim 1, characterized in that, When the controlled robot approaches the suspected area, it switches the infrared sensor to macro mode to collect the local temperature field, while the laser sensor accumulates the concentration at a fixed point. Specifically, this includes: When the horizontal distance between the robot's center of mass and the geometric center of the suspected leak source area is less than the preset approach distance, the robot is controlled to hover so that the distance between the front detection surface of the infrared thermal imaging sensor and the surface of the suspected leak source area is not greater than the preset micro detection distance. During the robot's hovering period, the laser methane sensor is controlled to continuously perform cumulative concentration measurements. The cumulative measurement time covers three consecutive concentration fluctuation cycles, and the concentration measurement values ​​at each sampling time are arranged in chronological order to form a cumulative concentration rise curve. Using the robot's hovering moment as the alignment zero point, the timestamps of each frame of the local temperature field image are matched with the timestamps of each sampling point of the concentration accumulation curve by counting with the same global clock, thus establishing a one-to-one time correlation between each frame of the temperature field image and each concentration sampling value.

7. The multimodal leak detection method for explosion-proof robots in natural gas stations according to claim 1, characterized in that, The process of aligning the inflection points of the temperature field texture change with the concentration accumulation curve along the time axis and comparing their spatial coordinates to filter out false sound sources and determine the location of the actual leak specifically includes: Extract the boundary pixels of the transition zone from the high temperature region to the low temperature region in the local temperature field, fit the boundary pixels into a closed contour line, and calculate the third coordinate of the geometric center of the closed contour line in the spatial coordinate system. Calculate the second derivative of the concentration accumulation curve, mark the time point when the second derivative changes abruptly to positive or negative as the inflection point, extract the spatial coordinates of the concentration sampling value corresponding to the inflection point, and mark them as the coordinates of the concentration change location. Calculate the projection deviation between the third coordinate and the coordinate of the concentration change location in the spatial coordinate system. If the projection deviation is less than the third threshold, the geometric center of the closed contour line is determined as the actual leak location and output. If the projection deviation value is greater than or equal to the third threshold, the current candidate area of ​​the sound source is determined to be a false sound source, the current suspected leakage source area is abandoned, and other candidate areas are selected again.

8. The multimodal leak detection method for explosion-proof robots in natural gas stations according to claim 1, characterized in that, The process of extracting the absolute concentration value and the energy value of the sound wave at a specific frequency band at the actual leak point, and inputting them into a risk level mapping table to determine the risk level specifically includes: After applying a Hanning window to the time-domain sound pressure signal acquired by the acoustic sensor, a fast Fourier transform is performed to obtain the frequency domain spectrum. Frequency components within the characteristic frequency band of natural gas leakage are extracted, and the sum of the squares of the amplitudes at each discrete frequency point is taken as the energy value of a specific frequency band. The graded mapping table includes four levels: normal, warning, alarm, and emergency. Each level corresponds to a combination of a concentration threshold range and an energy threshold range, and the ranges of each level are connected end to end. The absolute concentration value and the energy value of a specific frequency band at the actual leak point are used as a pair. The concentration threshold range and energy threshold range corresponding to the four levels are traversed. When the pair falls into both ranges of a certain level, the level is determined as the current leakage risk level. The risk level, the spatial coordinates of the actual leak location, and the robot's current pose information are packaged into a structured message, and a current global timestamp is appended to the message header before being sent to the site control platform.

9. A multimodal leak detection system for explosion-proof robots in natural gas stations, used to implement the multimodal leak detection method for explosion-proof robots in natural gas stations as described in any one of claims 1-8, characterized in that, include: Multimodal sensing synchronous acquisition module: integrates laser methane, infrared thermal imaging, and acoustic wave sensors to complete sensor attitude calibration and field of view coordinate system mapping, and unify data spatiotemporal labels; Baseline differential preprocessing module: Adaptively extracts the baseline of environmental concentration, temperature and sound pressure at the site, performs differential cancellation on the real-time monitoring data, removes environmental background noise interference, outputs clean differential field data and triggers initial leakage judgment; Multi-source information coarse localization module: Combines the concentration field gradient to determine the gas diffusion direction, locks the candidate area of ​​the sound source by the intersection of the hyperbola of the sound wave time delay difference, and completes the overlap determination by linking with the low temperature anomaly area, marking the suspected leak area; Precise verification and fake detection module: Controls the robot to hover and detect at close range, matches the inflection point of temperature field texture change and concentration accumulation curve, compares in two dimensions of time and space, filters out false leakage sound sources, and locks the real leakage point; Risk classification and assessment module: Extracts the concentration and characteristic frequency band sound wave energy at the leak point, and completes the four-level risk assessment based on the classification mapping table to quantify the leakage hazard level; Data transmission output module: integrates risk level, leakage coordinates, robot pose and timestamp, encapsulates them into standardized data frames, and stably uploads them to the site control platform to complete the data closed-loop output.

Images