Inspection device and operating method

Abstract

The inspection device comprises a plurality of sensors (including a visual sensor and one or more additional sensors, e.g., an optical gas imager and an anemometer), a real-time clock, and an inertial measurement unit, wherein the sensors have parallel-aligned central observation axes, and the sensors and real-time clock (and optionally a position module) are configured to generate data points from multiple orientations of the inspection device relative to the object being observed. The operation method of the inspection device includes sweeping the inspection device across a region in a parallel path to acquire data by enclosing the region, timestamping the data and / or assigning coordinate positions, correcting sensor position offsets, collocating visual sensor data with data from other sensors, generating a point cloud and 3D texture of the visual data, and generating a textured 3D model.

Description

Detailed Description of the Invention

[0001] [Cross Reference to Related Applications] This application claims priority based on U.S. Patent Application No. 18 / 603,831 filed on March 13, 2024 and U.S. Provisional Patent Application No. 63 / 529,922 filed on July 31, 2023, respectively, and the entire contents of the above applications are incorporated herein by reference.

[0002] [Technical Field] The present teachings relate to an inspection apparatus and an operating method thereof.

[0003] [Background] Asset inspection is an important responsibility in various industries, as assets degrade during their life cycle. Industry typically adopts inspection and preventive maintenance schedules to ensure assets function properly and even extend their life cycles by detecting defects before they can further degrade or, in some cases, damage assets.

[0004] Direct inspection has conventionally been performed through visual observation, which sometimes uses individual dedicated tools for measuring various features of an asset. In this regard, manual methods of observation, data collection, recording, and archiving have been adopted, but this process is prone to various human errors, can be time-consuming, and limits the robustness of data that can be collected due to time constraints.

[0005] In several industries, such as residential and commercial infrastructure, oil and gas, and chemical processing, gas leaks are of particular concern due to the risks of human exposure, ignition, waste, and environmental damage. Naturally, gas leaks are difficult to detect because gases are not typically detectable by human senses (e.g., by the naked eye), and more advanced methods are needed to collect meaningful data. Leaks can be caused by a variety of factors, including corrosion or other modes of degradation, improper or damaged seals, improperly installed equipment, and similar factors. In this field, the rapid detection and repair of escape gas sources remains a persistent need, and there is a continued need for devices and methods that improve the ease and speed of detection. The above also applies to escape liquids containing volatile components. These challenges are discussed in Brown et al.'s report, "Informing Methane Emissions Inventories Using Facility Aerial Measurements at Midstream Natural Gas Facilities," Environmental Science & Technology, 2023, 57, 14539-14574.

[0006] As the world becomes increasingly aware of environmental damage and climate change, fluid detection has become a critical issue across various industries. One exemplary fluid of interest is methane, produced in the natural gas supply chain. Numerous legal regulations and guidelines have been put in place to mitigate the environmental impact of industries by requiring regular monitoring of pollutants released into the environment and limiting pollutants. In many cases, companies have become more proactive in mitigating their environmental impact by setting internal targets that sometimes exceed legal standards. This trend has become even more pronounced with the introduction of environmental, social, and governance (ESG) scoring. Individual performance in one or more of these categories can influence public perception and may allow or restrict opportunities for partnerships with other companies, government contracts, international business relationships, expansion of existing facilities and promising new properties, or any combination thereof.

[0007] Even with forward-looking environmental targets in the current industrial environment in mind, implementing systems and processes to improve environmental impact can be challenging. Cost is one of the main limitations on addressing these targets. Ideally, more regular and comprehensive inspections of industrial assets using some of the conventional methods discussed in this section could significantly improve environmental impact, although the inspection time must justify the wages paid for it.

[0008] In this regard, Leak Detection and Repair (LDAR) inspectors bring efficiency and accuracy commensurate with their experience, training, and overall competence level. Currently, the number of LDAR inspectors with the high level of competence desired to efficiently and accurately identify leak sources and thereby direct corrective actions is limited in the market. Even when utilizing conventional inspection systems and methods, particularly those employing the advanced sensor technologies discussed below, a significant competence gap exists between less skilled and more skilled LDAR inspectors. This is discussed in Zimmerle et al., "Detection Limits of Optical Gas Imaging for Natural Gas Leak Detection in Realistic Controlled Conditions," Environmental Science & Technology, 2020, 54, 11506-11514.

[0009] In some conventional methods, sensors (e.g., IR gas sensors) acquire only qualitative gas data, such as the shape and size of the gas plume, rather than quantitative (e.g., concentration) measurements. In this regard, detecting the source of a gas leak faces several challenges, as gas plumes can exist in vast spaces, and when viewing images rendered from qualitative gas sensors, any part of an object within the area of ​​the gas plume can be the point of origin. In this regard, typically, when a gas plume is detected, a second inspection (e.g., using the extraction techniques described herein) must be performed, in which the user traverses the area to find the point of origin. As can be recognized, these devices and methods require multiple different types of sensor techniques, and the subsequent detailed inspection can be time-consuming. Furthermore, detected fugitive plumes can be shifted from their point of origin by wind, in which case the user may not be able to pinpoint the point of origin even when searching the area of ​​the plume.

[0010] In some conventional methods, sensors (e.g., tunable semiconductor lasers) can obtain quantitative measurements, but these are not combined with visualization techniques. In this regard, it may be possible to detect the source, but since users are usually only provided with quantitative readings on the sensor device, detecting the source may require multiple sweeps of the sensor from different angles and / or distances to the object. For example, sweeping the sensor from a certain distance to an object may distinguish areas of higher gas concentration, but pipelines, tanks, and other equipment from which the gas may be released usually have complex geometric shapes, and the source may be located at various positions within the peak concentration area. Again, in this case, wind may shift the peak concentration area away from the source, which can mislead the user.

[0011] In some conventional methods, quantitative sensors have been combined with visualization techniques. However, these methods are performed from one or more fixed positions, where the equipment must be installed (e.g., the sensor mounted on a tripod) and then moved to various positions and angles for inspection. Furthermore, these quantitative detection methods require a reliable background reflector (e.g., a reflective sheet placed behind the object so that the object being observed is positioned between the reflector and the sensor) to obtain quantitative gas measurements. In this regard, the gas may be observed against a background that would otherwise be an open sky or a suboptimal reflective surface. However, these methods are still time-consuming, and more comprehensive inspections (e.g., over large areas within oil and gas facilities) are time-constrained by the equipment installation process, so they are usually performed on a small scale for the asset. Moreover, these methods have been unable to determine the gas release rate due to the lack of wind speed and direction information, and even detailed multidimensional spatial information of escaped gas plumes.

[0012] Accurate release rate information is necessary, as it can be used to determine whether acceptable limits are being exceeded and to triage repair work. Release rate provides better insight into the severity of the release compared to local concentration or visualization alone.

[0013] Some conventional visualization techniques may provide basic information, such as peak concentrations within an area, or in some cases, real-time visualization of plumes. However, these types of information are typically interpreted on-site, increasing the overall inspection time. For example, an inspector might set up an optical gas imager in a fixed position and observe the behavior of a gas plume over time (e.g., 10, 15, or even 20 minutes) to determine if a leak is present and identify its estimated origin. Furthermore, current visualization techniques have limitations when large and / or complex assets are being inspected. In some cases, an asset may consist of dozens of conduits and / or containments arranged in a complex geometric configuration, some of which are densely packed and some of which may be obscured from any viewpoint of the inspector by intervening structures.

[0014] Furthermore, while the methods described above are suitable for individual inspection events, comparing various different inspection events over time is difficult due to the nature of the data extracted using these methods. In most cases, robust comparison and visualization of qualitative and quantitative time-lapse data are needed to repair defects quickly and efficiently. For example, inspection events prior to and after a repair may be compared to determine whether the defect was properly repaired and, in some cases, to diagnose further defects. In this regard, more data than conventionally available, as well as unique visualization techniques that enable users to better diagnose defects, are needed.

[0015] Similarly, inspections that acquire other types of data, including thermal and / or acoustic measurements, are necessary in this industry. In some cases, it may be advantageous to combine several types of sensor technologies to enable joint analysis of different types of data to aid in fault diagnosis. For example, excessive heat can contribute to instability in some liquids and gases. In another example, acoustically detectable turbulence in a pipeline can contribute to areas of higher pressure. In these cases, gas concentration measurements alone may not be sufficient for proper fault diagnosis.

[0016] Improved equipment and methods are needed for conducting asset inspections.

[0017] Improved devices and methods are needed for detecting fluid leakage.

[0018] Apparatus and methods are needed to detect both quantitative and qualitative gas measurements.

[0019] Apparatus and methods are needed to estimate the release rate.

[0020] There is a need for equipment and methods that can enable even less experienced inspectors to achieve the same level of efficiency and accuracy as more skilled inspectors.

[0021] Apparatus and methods are needed to perform inspections that involve a single pass through an environment and / or facility, without requiring subsequent passes, such as subsequent passes using different sensors.

[0022] A device and method are needed that allows users to have freedom of movement without the need to install fixed equipment.

[0023] Apparatus and methods are needed that can simultaneously collect thermal and / or acoustic detection values ​​during the collection of chemical data (quantitative and qualitative measurements).

[0024] There is a need for an apparatus and method that enables 3D modeling of an inspected object using non-visual (e.g., chemical, thermal, acoustic, or any combination thereof) data collocated with visual data.

[0025] There is a need for an apparatus and method that enables time-lapse analysis of data and / or joint analysis of different types of data (i.e., data from different types of sensors).

[0026] [Summary] The present teachings provide an inspection apparatus that can solve at least some of the needs identified above. The inspection apparatus may comprise a plurality of sensors. The plurality of sensors may include a visual sensor and one or more of an optical gas imager, at least one anemometer, a thermographic camera, a microphone, or any combination thereof. The plurality of sensors may further comprise a position module. The position module may be configured to assign position coordinates to individual data points or groups of data points generated by the plurality of sensors. The inspection apparatus may comprise a real-time clock. The real-time clock may be configured to timestamp individual data points or groups of data points generated by the plurality of sensors. The inspection apparatus may comprise an inertial measurement unit.

[0027] The plurality of sensors (e.g., visual sensor, optical gas imager, thermographic camera, and microphone) may each be characterized by a central observation axis aligned in parallel.

[0028] The plurality of sensors, the real-time clock, and the position module may operate simultaneously to generate data points from at least two poses of the inspection apparatus relative to an object observed by the inspection apparatus.

[0029] The visual sensor may be an RGB camera or a stereo camera.

[0030] The inspection apparatus does not have to comprise a LIDAR sensor.

[0031] The plurality of sensors may comprise an optical gas imager and at least one anemometer.

[0032] The position module may be a GPS module.

[0033] The at least one anemometer may comprise a hot-wire anemometer.

[0034] The at least one anemometer may comprise a first anemometer, and optionally a second anemometer. The inspection device may have a first groove and a second groove in which the first and second anemometers are respectively positioned. The first groove may be oriented perpendicularly to a central observation axis, and the second groove may be aligned parallel to the central observation axis.

[0035] The anemometer may extend from a front face of the inspection device and may protrude beyond other sensors.

[0036] The optical gas imager may be a tunable semiconductor laser configured to implement tunable diode laser absorption spectroscopy. The optical gas imager may be configured to detect hydrocarbons (e.g., methane), hydrogen sulfide, carbon dioxide, or any combination thereof.

[0037] The inspection device may be hand-held.

[0038] The inspection device may further comprise a visible laser that generates a beam parallel to the central observation axis of the plurality of sensors. The visible laser may be configured to assist a user in tracing a path across an entire region of a target object being observed.

[0039] The inspection device may further include a graphical user interface. The graphical user interface may be a touch-sensor graphical user interface. Data from one or more of the sensors may be displayed on the graphical user interface in real time or substantially in real time.

[0040] The testing device may further include one or more wired or wireless data transmission modules. The wired data transmission module may include an Ethernet port, a USB port, an SD card reader, or the like (preferably an Ethernet port). The wireless data transmission module may include a WiFi module, a Bluetooth module, a cellular module, or the like.

[0041] The inspection device may further include a battery (for example, including a charging port at the bottom of the handle).

[0042] The inspection device may further include a printed circuit board.

[0043] This instruction provides a method that can address at least some of the needs identified above. The method may be for operating the inspection device described above. The method may include acquiring data from two or more of a plurality of sensors, which includes a) sweeping (i.e., sweeping) the inspection device across an area in a substantially parallel path from a fixed position, and b) surrounding the area with the inspection device from the fixed position.

[0044] This method may include timestamping the data and / or assigning position coordinates. The position coordinates may include the position and orientation of the inspection device.

[0045] This method may include correcting the positional offset of the observation axes of two or more sensors.

[0046] This method may include collocating data from a visual sensor with data from one or more of the following: an optical gas imager, at least one anemometer, a thermographic camera, and a microphone.

[0047] This method may include performing photogrammetry on the visual data in order to generate point clouds and 3D textures of the visual data.

[0048] This method may include generating a 3D model using one or more textures from multiple sensors.

[0049] This method may further include discarding duplicate and / or irrelevant data.

[0050] This method may further include verifying the illuminance with a visual sensor to determine whether the illuminance is within a predetermined operating range for the optical gas imager.

[0051] This method may further include performing time-lapse analysis by comparing data from an immediate inspection event and data from a pre-inspection event, both of which contain data on the same object.

[0052] This may further include estimating the release rate of gas leaks that are emitted from the point of origin and released into the atmosphere, thereby forming an escape plume.

[0053] Estimation may include measuring the gas concentration with an open-air optical path gas sensor, measuring the wind speed and / or wind direction with an anemometer, and estimating the release rate based on the gas concentration and wind speed and / or wind direction.

[0054] The estimation may further include generating a visualization of the geometric shape of the escape plume based on multiple gas concentration measurements, providing the visualization as input to a convolutional neural network, and refining the estimate based on the categorical output of the convolutional neural network.

[0055] The convolutional neural network may be trained using multiple visualizations of the geometric shape of the escape plume, where the ground truth of each visualization includes discrete categories of emission models defined by diffusion from the point of origin and positional shifts relative to the point of origin. [Brief explanation of the drawing]

[0056] [Figure 1A] This is a perspective view of the inspection device. [Figure 1B] This is a perspective view of the inspection device. [Figure 2] This is a perspective view of the inspection device. [Figure 3] This shows the inspection equipment currently in use. [Figure 4] This is a perspective view of the inspection device. [Figure 5A] Show the 3D model. [Figure 5B] Show the 3D model. [Figure 6A] Show the 3D model. [Figure 6B] Show the 3D model. [Figure 7A] This is a perspective view of the inspection device. [Figure 7B] This is a perspective view of the inspection device. [Figure 7C] This is a front view of the inspection device. [Figure 7D] This is a perspective view of the inspection device. [Figure 7E] This is a perspective view of the inspection device. [Figure 7F] This is a bottom view of the inspection device. [Figure 7G] This is a top view of the inspection device. [Figure 7H] This is a side view of the inspection device. [Figure 7I] This is a side view of the inspection device. [Figure 8] Exemplary buoyancy, diffusion, and ejection models described herein are shown. [Figure 9] This shows the graphical user interface associated with the digital emission tag. [Modes for carrying out the invention]

[0057] [Detailed explanation] [Inspection equipment] This disclosure describes an inspection apparatus. The inspection apparatus may be configured to acquire visual images, qualitative and quantitative gas measurements, thermal measurements, acoustic measurements, or any combination thereof. The inspection apparatus may be configured to acquire data that ultimately results in a visualization of two or more datasets juxtaposed on a 3D model, according to the methods described herein.

[0058] The inspection device may comprise multiple sensors. These sensors may function to detect physical phenomena and generate output signals (e.g., digital or analog) from them. Physical phenomena may include electromagnetic radiation, mechanical waves, mechanical motion, nuclear radiation, the like, or any combination thereof. As recognized by this disclosure, electromagnetic radiation may be characterized by various points and / or ranges on the electromagnetic spectrum (e.g., visible light, ultraviolet light, infrared light, radio waves, gamma rays, and the like) associated with wavelength and / or frequency; mechanical waves (e.g., detected by accelerometers, vibrometers, microphones, the like, or any combination thereof) may be characterized by amplitude, frequency, wavelength, or any combination thereof; and mechanical motion may be characterized by velocity, direction, or both. Nuclear radiation may be detected by a neutron detector or any other elementary particle detector. Output signals may include images, position coordinates, measurements of physical phenomena, the like, or any combination thereof, as will be discussed in more detail below.

[0059] Multiple sensors may be characterized by a central observation axis. Generally, the central observation axis may extend from the sensor toward the object being observed. For a sensor observing electromagnetic radiation, the central observation axis may be an axis extending through the geometric center of the sensor's field of view (i.e., line of sight). For a sensor observing mechanical waves, the central observation axis may be an axis extending through the geometric center of the acoustic pickup field (e.g., the 0-degree axis of a polar pattern). Unlike electromagnetic radiation and mechanical waves, mechanical motion (e.g., sound, wind, or inertia) may be characterized by the directionality of the physical phenomenon located on a continuum of angles in a polar coordinate system. In some embodiments, one, two, or more discrete observation axes may be defined by a tube, groove, dish, or similar housing one or more sensors (e.g., directional microphones or wind sensors), where the orientation of the tube, groove, dish, or similar guiding sound or receiving wind may define the observation axis.

[0060] Multiple sensors may be arranged on the inspection device such that the photodetectors, microphones, or similar devices (i.e., components that thereby emit / receive electromagnetic radiation and / or sound) are positioned in close proximity to one another. The multiple sensors may have positional offsets of about 5 cm or less, 4 cm or less, 3 cm or less, 2 cm or less, or even 1 cm or less. It can be recognized from this teaching that wind speed and / or wind direction can be determined without relating them to the observation axes of other sensors described herein, so that an anemometer does not need to be limited by its positional offset relative to other sensors. Preferably, the positional offset can be minimized to support the correction of positional offsets and even collocation described herein.

[0061] Multiple sensors may include one or more vision sensors. A vision sensor may function to transmit electromagnetic radiation in the visible spectrum (e.g., approximately 400 nm to 700 nm) into an image. The image may be encoded in a digital medium (e.g., a non-temporary storage medium), transmitted visually to a graphical user interface (e.g., an LCD or OLED display), or both. The image may have an array of pixels arranged in an XY coordinate system.

[0062] In some embodiments, photogrammetry may be performed to add a Z coordinate (depth) to the XY coordinate system. In this regard, multiple images may be stitched together. A point cloud may be generated from the stitched images. A texture may be generated from the point cloud. Multiple images may be stitched together based on contextual information. Contextual information may include overlapping features in the images, overlapping features in the images generated from non-visual data such as those described herein (e.g., data from open-air optical path gas sensors, thermographic cameras, microphones, and similar devices), data from real-time clocks as described herein, data from position models as described herein (e.g., position coordinates), data from gyroscopes and / or accelerometers as described herein, or any combination thereof. A visual sensor may collect individual images of 100 or more, 500 or more, 1,000 or more, 2,000 or even 3,000 or more objects (otherwise referred to herein as assets, industrial assets, or equivalents) during an inspection event. In this regard, scanning an object and / or region along a path described herein may provide multiple images having overlapping features that can be mapped and stitched together.

[0063] In some embodiments, the visual sensor may be a stereo camera, which may add a Z coordinate to the XY coordinate system. The stereo camera may be employed in conjunction with the photogrammetry described above, or it may be employed without photogrammetry.

[0064] In some embodiments, the visual sensor may be coupled with laser image detection and ranging ("LIDAR"), which may add a Z coordinate to the XY coordinate system. LIDAR may be advantageous for providing high-density point clouds. While high-density point clouds may be advantageous for detail in some situations, data size, the downstream data processing pipeline of data acquisition by multiple sensors, and / or the overall size of the sensor device may be more important, in which case the photogrammetry and stereo camera embodiments may be advantageous.

[0065] A 3D coordinate system can provide navigation for a 3D model (e.g., pitch, roll, yaw). That is, a user can view different orientations of a 3D model on a 2D medium (i.e., a graphical user interface for a laptop, desktop computer, mobile phone, and similar devices). In this regard, different surfaces and / or sides of an object can be viewed on the graphical user interface. The Z coordinate may be communicated visually as a heatmap (e.g., on the graphical user interface). The data visualization may be based on any suitable color model (e.g., the RGB color model).

[0066] The vision sensor may include one or more complementary metal-oxide-semiconductor ("CMOS") image sensors, charge-coupled device ("CCD") sensors, similar devices, or any combination thereof. The vision sensor may produce high-resolution images. As referred to herein, high resolution may mean from about 10 megapixels ("MP") to 50 MP (e.g., about 12 MP or more, 15 MP or more, 20 MP or more, 30 MP or more, or even 40 MP or more). An example of a suitable vision sensor may include the Raspberry Pi High Quality Camera, commercially available from Raspberry Pi Ltd.

[0067] Multiple sensors may include one or more position modules. A position module may define the position of the inspection device and function to correlate data acquired at that position with that position. A position module may function in conjunction with one or more satellite-based positioning services (e.g., the Global Positioning System ("GPS")). A position module may comprise a receiver (e.g., an antenna), a microcontroller, or both. A position module may receive signals (e.g., radio signals) to triangulate the position module with respect to three or more satellites (or cell phone base stations for cell phone navigation within the scope of this teaching). A position module may indicate its position in a coordinate system, e.g., latitude and longitude, or optionally, altitude. Altitude may be advantageous in situations where the facility being inspected has different altitudes, levels, or similar. The introduction of a position module in the apparatus and methods described herein may be particularly advantageous for collecting large datasets in space. In this regard, the data may be organized at the location where the data was collected, and the position coordinates assigned to the data may be used for a digital reconstruction of the object in 3D space.

[0068] Multiple sensors may include one or more gyro sensors. A gyro sensor may function to determine the rotational acceleration of the inspection device and to estimate the angular velocity, pitch, roll, and / or yaw of the inspection device. A gyro sensor may provide contextual information for photogrammetry. A gyro sensor may work in conjunction with one or more anemometers to determine wind direction and one or more position modules to provide contextual information to data collected by multiple sensors, or both.

[0069] Multiple sensors may include one or more accelerometers. Accelerometers may function to determine linear acceleration and estimate linear velocity and direction. Accelerometers may work in conjunction with one or more anemometers to determine wind direction, and with one or more position modules, or both, to provide contextual information to data collected by multiple sensors. Accelerometers may provide contextual information for photogrammetry.

[0070] Multiple sensors may include one or more inertial measurement units ("IMUs"). An inertial measurement unit may comprise one or more gyroscopes, accelerometers, or both. An inertial measurement unit may function to determine the position and / or orientation ("attitude") of the inspection device.

[0071] One or more contextual data sources described herein (e.g., position modules, gyroscopes, accelerometers, inertial measurement units, and real-time clocks) are understood to be usable in the photogrammetry methods described herein. That is, the above contextual data can be used to accurately correlate images, measurements, or both with objects observed by the sensors described herein.

[0072] Multiple sensors may include one or more open-air optical path gas sensors (open-path gas sensors). An open-path gas sensor may function to transmit electromagnetic radiation to an image, to identify the presence or absence of a target gas, and optionally to determine the concentration of the target gas. An open-path gas sensor may radiate an electromagnetic radiation beam into the environment (as opposed to a surrounded measurement cell). An open-path gas sensor may comprise a radiator that emits electromagnetic radiation and a receiver that receives the reflected electromagnetic radiation. The electromagnetic radiation may travel through a target gas (e.g., a escaping plume) and be reflected by solid objects, such as objects from which the target gas escapes. The emitted electromagnetic radiation may be at least partially absorbed by the molecules of the target gas in a narrow band associated with specific wavelengths, and may generally not show absorption outside these bands. The target gas may absorb electromagnetic radiation in characteristic wavelength bands. In this regard, the receiver may acquire electromagnetic radiation attenuated according to the Lambert-Beer relationship, thereby identifying the target gas and / or its concentration using characteristic absorption patterns.

[0073] The open-path gas sensor may comprise one or more mirrors, lenses, optical filters, or any combination thereof. The received electromagnetic radiation may ultimately be directed to a photodetector. The photodetector may include an infrared photodetector or a quantum well infrared photodetector ("QWIP"). The photodetector may comprise a semiconductor material (e.g., InSb, InAs, Hg-CdTe, or similar). The semiconductor may be selected based on sensitivity, spectral selectivity, operating temperature, peak wavelength, or any combination thereof. With respect to spectral selectivity, the wavelength may be selected to target the absorption band of the target gas and avoid the absorption band of the non-target gas. With respect to operating temperature, the semiconductor may or may not require cooling and associated cooling mechanisms in the inspection apparatus.

[0074] The open-path gas sensor may perform infrared absorption spectroscopy. The open-path gas sensor may also be an optical gas imager.

[0075] An open-path gas sensor may employ wavelength-modulated laser absorption spectroscopy (preferably, tunable semiconductor laser absorption spectroscopy). An open-path gas sensor may use a tunable wavelength-modulated diode laser as its light source. The laser wavelength may be swept between the non-absorption band of the target gas and one or more specific absorption bands. When the wavelength is tuned outside a characteristic narrow absorption band ("offline"), the received light is equal to or stronger than when it is within the narrow absorption band ("online"). Measuring the relative amplitude of the offline reception to the online reception yields a measurement of the methane gas concentration along the path the laser beam travels. The focused light is converted into an electrical signal, which is processed so that the fluid (e.g., methane) column density (fluid concentration integrated over the beam length) can typically be reported in ppm·m. Tuned semiconductor laser absorption spectroscopy may be advantageous (over infrared radiation) with respect to its ability to acquire concentration data using this teaching.

[0076] An example of a suitable tunable semiconductor laser that can be used in this instruction is the Model S350-W2, commercially available from Henan Zhongan Electronic Detection Technology Co., Ltd.

[0077] The open-path gas sensor may perform measurements at speeds of approximately 5 Hz to 20 Hz (for example, 10 Hz).

[0078] The target gas may include hydrocarbons (e.g., methane), hydrogen sulfide, carbon dioxide, or any combination thereof. These are merely examples of possible target gases, and this teaching assumes that the inspection device can detect and / or characterize the concentration of any gas. In general, detection may rely on the characteristic absorption of electromagnetic radiation by the gas. In this regard, the frequency of the electromagnetic radiation emitted from the inspection device may be selected to detect different types of gases.

[0079] The target gas may be selected from a well-known mixture of gases. For example, methane is a component of natural gas. Therefore, in this context, a leak of a gas mixture may be identified by detecting one of the components of the mixture.

[0080] The optical gas imagers described herein may be advantageous over conventional extraction detection techniques that rely on drawing air through a measuring cell and performing techniques such as spectroscopy, electrochemical methods, solid-state methods, piezoelectric methods, gas chromatography, flame ionization, calorimetry, or any combination thereof. In the extraction method, the device including the measuring cell draws in only the gas within its immediate vicinity, and therefore, the readings must be acquired within or in close proximity to the escape plume.

[0081] An open-air optical path gas sensor can sense the reflectivity of an object's surface to which electromagnetic radiation is reflected, light from an external light source (e.g., a light bulb or the sun) reflected off the object, or both. The open-air optical path gas sensor may operate within a range of light intensity, outside which data acquisition may be paused and / or the user may be warned. This range may be approximately 1 lux to 5 lux, more preferably approximately 2 lux to 3 lux. External light sources and / or highly reflective background objects may cause the intensity to exceed the upper limit of the range. Poorly reflective background objects may cause the intensity to fall below the lower limit of the range. One or more visual sensors may be employed to measure light intensity. The measured light intensity may be compared against a predetermined operating range. Outside the predetermined operating range, data acquisition may be stopped and / or the user may be warned. In this regard, time can be saved by avoiding observations under suboptimal conditions, instructing the user to move on to inspecting downstream objects, and then later returning to the object that was previously under suboptimal conditions.

[0082] Multiple sensors may include one or more anemometers. An anemometer may function to transmit its interaction with the wind in a signal. In some embodiments, the signal may be analog. The inspection device may include an analog-to-digital converter for converting the analog signal to a digital format. An anemometer may identify wind speed, wind direction, or both. An anemometer may be any suitable type of anemometer, including a hot-wire anemometer, an ultrasonic anemometer, an acoustic resonant anemometer, or any combination thereof. Preferably, multiple sensors include hot-wire anemometers (e.g., constant-current anemometers, constant-voltage anemometers, constant-temperature anemometers, and pulse-width modulation anemometers, preferably constant-temperature anemometers).

[0083] An anemometer may be placed inside a tube or groove. The tube or groove may be oriented in a direction related to the observation axes of multiple sensors. Therefore, wind with a direction generally coaxial with the tube or groove may enter the tube or groove and interact with the anemometer. The tube or groove may be oriented perpendicular, parallel, or at any angle to the observation axes of the sensors. The inspection device may include two, three, or even four anemometers oriented at different angles to the observation axes of multiple sensors.

[0084] The wind direction can be determined to be coaxial with the anemometer and the pipe / groove assembly, which interact with the wind.

[0085] The wind direction may be estimated from measurements of two or more anemometers. The estimated wind direction may be off-axis from the anemometer. Wind speed may be represented as a vector in this regard. For example, a wind direction between 0 and 90 degrees relative to an anemometer may be determined based on multiple wind speed vectors measured by the same anemometer.

[0086] When a single anemometer and tube / groove assembly is employed, the direction of the tube is preferably perpendicular to the observation axis. In this regard, the perpendicular direction of the wind can be important for understanding and characterizing the diffusion characteristics of the escape plume. For example, wind directions perpendicular to the observation axis may result in lower concentration measurements compared to the absence of wind. Real-time feedback of concentration and / or wind direction may be communicated to the user, prompting them to scan a wider area to detect the leak source and / or capture a larger portion of the escape plume.

[0087] In another example, wind direction parallel to the observation axis may, under certain circumstances, provide less information than wind direction perpendicular to it. That is, if the escape plume moves toward or away from the inspection device, the concentration measured by light passing through the escape plume may not identify where along the observation axis the leak occurred.

[0088] An anemometer may be employed to estimate the leakage rate (i.e., volume per unit time). Algorithms and / or models may be used to determine the leakage rate from the concentration (such as that identified by an open-air optical path gas sensor) and the wind speed correlated with the concentration measurement. Generally, wind speed can identify the migration behavior (or lack thereof) of a escaping plume. Algorithms and / or models may incorporate one or more other variables, including, but are not limited to, ambient temperature, relative humidity, altitude, similar, or any combination thereof.

[0089] Surprisingly, it has been found that, under certain circumstances, wind direction is not required to obtain accurate leak velocity estimates (e.g., with a coordination of 95% or more, more preferably 97% or more, or even more preferably 99% or more, with the actual leak velocity). Due to the handheld nature of this inspection device and its method of use, measurements are typically taken at a distance from the object at which leak velocity estimates can be obtained without wind direction. Furthermore, by continuous data collection and 3D modeling as described herein, the movement of wind plumes can be tracked, and leak velocity can be estimated from there.

[0090] Multiple sensors may include one or more thermographic cameras. A thermographic camera may function to transmit electromagnetic radiation in the infrared spectrum (e.g., approximately 700 nm to 1 mm) into an image. The image may be encoded in a digital medium (e.g., a non-temporary storage medium), transmitted visually on a graphical user interface (e.g., an LCD or OLED display), or both. The image may have an array of pixels arranged in an XY coordinate system. In some embodiments, the pixels may be mapped to the XY coordinate system of an image obtained, for example, from a visual sensor. Thermal measurements may be transmitted visually as a heatmap (e.g., on a graphical user interface). The image may be displayed in false color.

[0091] Given the typical resolution of conventional thermal images that can define surfaces, edges, and other structural features, the photogrammetry methods described herein may be employed in a manner similar to that of visual images to stitch together different thermal images and render thermal textures for 3D models.

[0092] Multiple sensors may include one or more microphones. The microphones may function to transmit mechanical wave characteristics to an analog signal (for example, through the interaction of mechanical waves with a diaphragm). The inspection device may also include an analog-to-digital converter for converting the analog signal to a digital format.

[0093] The microphone may include a directional microphone. Suitable examples of directional microphones may include a parabolic microphone, a shotgun microphone, a boundary microphone, a phased array microphone, or any combination thereof.

[0094] The microphone may include any other type of microphone, such as an omnidirectional microphone. In this regard, the directivity of the signal may be determined by post-processing. Post-processing may include phased array processing.

[0095] Acoustic data may be collocated with visual data in order to generate textures that can be mapped onto a 3D model, similar to or in a similar manner to the gas data described herein.

[0096] The inspection device may include one or more real-time clocks ("RTCs"). The RTCs may function to measure the passage of time (for example, in terms of Coordinated Universal Time, or as timers started during inspection events). The RTCs may work in conjunction with multiple sensors to timestamp output signals from those sensors (e.g., images, position coordinates, measurements of physical phenomena, etc.). The output signals may be synchronized based on the timestamps.

[0097] In one example, the inspection device may include a visual sensor, an optical gas imager, an anemometer, a position module, an inertial measurement unit, and a real-time clock.

[0098] In one example, the inspection device may include a visual sensor, an optical gas imager, a thermographic camera, an anemometer, a position module, an inertial measurement unit, and a real-time clock.

[0099] In one example, the inspection device may include a visual sensor, a thermographic camera, a position module, an inertial measurement unit, and a real-time clock.

[0100] In one example, the inspection device may include a visual sensor, a microphone, a position module, an inertial measurement unit, and a real-time clock.

[0101] In one example, the visual sensor may be a stereo camera, the optical gas imager may be a tunable semiconductor laser, the anemometer may be a hot-wire anemometer, or they may be combined in any way.

[0102] This instruction assumes that a combination of the following features of an inspection device may provide a unique and unconventional solution that surpasses the performance of conventional devices used for asset inspection (e.g., inspection of fluids, such as methane). These features include a handheld structure, an optical gas imager (e.g., a tunable semiconductor laser), a visual sensor (e.g., a stereo camera, preferably a high-resolution stereo camera), and an anemometer (e.g., a hot-wire anemometer).

[0103] Conventional devices lacking one or more of these features lack the performance of the inspection apparatus described herein. For example, some conventional devices are stationary (rather than handheld), which limits their ability to visualize data such as 3D models as described herein. Some conventional devices do not have an anemometer, which limits their ability to measure discharge velocity and / or detect the point of origin of leakage.

[0104] This disclosure is not intended to be limited by the examples described above, and the inspection device may include one or any combination of the multiple sensors described herein, and optionally, a real-time clock. These examples are selected considering typical end-user requirements and cost considerations for providing various functions in a single device.

[0105] In a typical inspection apparatus configuration, it can be recognized that one or more visual sensors may work in conjunction with one or more optical gas imagers, thermographic cameras, microphones, or any combination thereof to collate data from one or more optical gas imagers, thermographic cameras, microphones, or any combination thereof with coordinates in a 2D image and / or 3D model. This teaching may find particularly advantageous applications using 3D modeling of the various types of data described herein to provide inspectors with further details.

[0106] The test device may include one or more data transmission modules / ports. A data transmission module may function to transmit data between the test device and one or more computer devices. A data transmission module may be wired or wireless. The test device may include a combination of wired and wireless data transmission modules. An exemplary wired data transmission module may include an Ethernet port, a USB port, or similar. A wired data transmission module may accept an external memory device (e.g., an SD card, a USB flash drive, or similar) and perform read and / or write operations. An exemplary wireless data transmission module may include a WiFi module, a Bluetooth module, a cellular module, or similar. The test device may accept a SIM card.

[0107] The inspection device may include one or more graphical user interfaces. The graphical user interfaces may communicate information to the user in real time. The information may include gas concentration, gas release rate, wind speed, name of the object being observed, time, date, inspection duration, number of files collected (e.g., image frames, sensor measurements, etc.), data download / upload status, live feeds from one or more sensors, and one or more indicators (e.g., visual or audible indicators of exceeding concentration limits).

[0108] The testing device may comprise one or more printed circuit boards. The printed circuit board (PCB) may comprise one or more processors, memory storage devices (e.g., non-temporary memory storage media), data connections, power connections, or any combination thereof. The testing device may comprise a computing PCB, peripheral devices (i.e., sensors, a real-time clock PCB, a data transmission module PCB, or any combination thereof), or a single PCB integrating one or more of the above.

[0109] The inspection device may comprise one or more processors. The processors may be configured to communicate with multiple sensors. The processors may perform the methods described herein in accordance with computer-readable instructions. One or any combination of sensors may comprise one or more processors performing dedicated processes related to the operation of the sensor (e.g., for a vision sensor, converting the interaction with the photodetector's electromagnetic radiation into a digital signal), and one or more processors performing post-processing (e.g., data correlation, position offset correction, data collocation, and similar, by the methods described herein). The processors may perform some or all of the methods described herein. Furthermore, this teaching assumes a system having one or more computer devices that communicate with the inspection device and are capable of performing some or all of the methods described herein.

[0110] The inspection device may be equipped with one or more heat sinks. The heat sinks may function to absorb heat from the components of the inspection device (e.g., sensors, processors, and similar).

[0111] The inspection device may be equipped with one or more fans. The fans may function to generate airflow to remove heat from the components of the inspection device and / or heatsinks.

[0112] The inspection device may be equipped with one or more temperature and / or humidity sensors. The temperature and / or humidity sensors may function to monitor the temperature and / or humidity within the inspection device. The temperature and / or humidity sensors may be used to operate / stop one or more fans, operate / stop one or more sensors, warn the user, or any combination thereof. In this regard, one or more of the sensors may have an operating temperature range in which reliable measurements may not be obtainable outside of them, or in some cases the sensor may be damaged. In particular, a tunable semiconductor laser may have a maximum operating temperature of about 50°C or less, 45°C or less, or even 40°C or less, and / or a maximum operating humidity of about 80% RH or less.

[0113] The inspection device may be equipped with one or more batteries. The batteries (e.g., lithium-ion batteries) may be mounted on the inspection device to power the multiple sensors and other hardware described herein, so that the inspection device may be portable.

[0114] [method] This disclosure describes the operation method of the inspection apparatus described herein.

[0115] This method may involve acquiring data from multiple sensors. The multiple sensors may include at least one vision sensor. The multiple sensors may further include one or more of the following: an optical gas imager, at least one anemometer, a thermographic camera, and a microphone.

[0116] The user can acquire data by sweeping the inspection device across a region of interest concerning an object. Within the region, the sensor's observation axis can be swept along a nearly parallel path (e.g., horizontally, vertically, or diagonally). Optionally, after sweeping along a nearly parallel path, the sensor's observation axis can be swept around the periphery of the region. Sweeping along a nearly parallel path may be performed primarily to collect data to be used for visualization and analysis. Sweeping around the periphery may be performed to collect contextual data to help stitch the images together.

[0117] A sweep may be performed to obtain measurements within a region. The region referred to herein in relation to a sweep may be understood as a pyramidal or conical region depicted by the field of view boundary of the sweep described above. In this regard, the user is positioned at the apex of the pyramidal or conical region. The user may remain still or move (e.g., walk) during the sweep.

[0118] A visible light laser can help the user orient the inspection device in the correct direction relative to the object and / or region of interest. The visible light laser may be shone approximately parallel to the observation axes of multiple sensors. The visible light laser can be reflected from surfaces, and therefore can visually indicate approximately where the observation axes of multiple sensors extend.

[0119] A sweep may include a first sweep, a second sweep, and optionally one or more additional sweeps.

[0120] The first sweep may traverse a wider area than the second sweep. The first sweep may be performed while the user is exercising (e.g., walking). The first sweep may be performed while the user walks around the periphery of the object and / or area of ​​interest, preferably around substantially the entire periphery of the object and / or area of ​​interest. The first sweep may be performed to obtain measurements of gas concentration and / or release rate to determine if a leak is present, to locate the approximate area of ​​the leak source, to obtain qualitative data on the leak, to obtain data for constructing a 3D model, or to combine them as desired.

[0121] During the first sweep, the user may be alerted to the presence of a leak. The alert may be an audible and / or tactile (e.g., vibration) alert. The alert may be triggered by meeting or exceeding a maximum gas concentration threshold of approximately 150 ppm·m or less, 100 ppm·m or less, or even 50 ppm·m or less. Preferably, the maximum gas concentration threshold is above the noise threshold.

[0122] The second sweep, and optionally one or more additional sweeps, may traverse a narrower area than the first sweep. The second sweep may be performed while the user is stationary, although the user may move to different positions around the object during the second sweep, or remain stationary in those positions. The second sweep may be performed to obtain higher-resolution data of gas concentration, release rate, and qualitative data.

[0123] At least the visual sensor can result in unique and unconventional data visualizations. In this regard, the handheld nature of the inspection device described herein and its method of operation are understood to result in the above data visualizations by acquiring a large number of images from different orientations (e.g., 100 or more, 500 or more, or even 1,000 or more images). The inspection device may be held at a distance of about 15 meters or less, more preferably about 10 meters or less, more preferably about 5 meters or less, or even more preferably about 3 meters or less from the object being inspected. With respect to the 3D model described herein, an accuracy of less than 1 inch can be obtained at 3 meters or less, and an accuracy of 2 to 3 inches can be obtained at distances greater than 3 meters but less than 10 meters. The above accuracy refers to the dimension of at least one feature that can be visually conveyed on the 3D model within these dimensional ranges.

[0124] This method may include timestamping data with a real-time clock and / or assigning position coordinates to the data via a position module. This method may also include associating the corresponding timestamps with the data having position coordinates.

[0125] This method may include correcting the positional offset of the observation axes of multiple sensors. The observation axis may be corrected to the observation axis of any one of the multiple sensors, or to any other predetermined reference axis (e.g., the geometric center of the inspection device).

[0126] This method may include performing photogrammetry on visual data as described herein, and optionally on thermal and / or acoustic data. The coarse poses of each data point, including images / frames, may be estimated based on the above photogrammetry. That is, the position and orientation of the inspection device when each data point was acquired can be estimated.

[0127] This method may include collocating data from a visual sensor with data from one or more of the following: an optical gas imager, at least one anemometer, a thermographic camera, and a directional microphone. By collocating, each individual data point from each sensor may have a 3D coordinate assigned to it for downstream 3D modeling purposes. The 3D coordinate may lie on the surface of the object being observed.

[0128] This method may include discarding duplicate and / or irrelevant data. Examples of irrelevant data may include background objects outside the first observed object, detailed data on the ground or other surfaces around the object (e.g., color, illuminance, or non-visual sensor measurements), error data that cannot be properly mapped to a 3D model of the object, or any combination thereof.

[0129] This method may include displaying the data on a graphical user interface. The data may be displayed as a 3D model with one or more textures corresponding to different sensors. One or any combination of textures may be selectively applied to the 3D model for individual or joint analysis of the data. The 3D model may be exploreable by visualizing the model in digital space from different orientations. In this regard, various angles of acquired data of an object may result in visualization of various angles, surfaces, features, and similar aspects of the object via the 3D model. One or more quantitative measurements, such as gas concentration and / or release rate within a region, temperature within a region, and similar aspects, may be extracted from the 3D model as discussed herein.

[0130] Images collected by a visual sensor may be processed according to the methods described herein (including, for example, photogrammetry) to generate a 3D model. The 3D model may have a point cloud and / or one or more textures including surfaces and colors. Photogrammetry may be employed to generate a point cloud from images acquired from a visual sensor, including an image created by stitching together images from different orientations.

[0131] Photogrammetry may include homography estimation. Homography estimation may include determining linear and / or pivotal movement between one or more features in a series of images, and image stitching may be performed based on such movement.

[0132] One or more features include boundaries, corners, and similar features. For example, a boundary in an image is the boundary between an object (e.g., a pipe) and the background scenery.

[0133] Based on the homography estimation described above, multiple images may be stitched together to generate a 3D model. By incorporating non-visual data (e.g., chemical, thermal, and acoustic) simultaneously with or nearly simultaneously with visual data, the visual representations of the non-visual data can be stitched together in a corresponding manner.

[0134] As described herein, some gaps in images of one or more surfaces of an object may be reproduced, for example, by a neural network trained with training data containing multiple images of the object. In this regard, initial comprehensive digital modeling may allow the neural network to fill in the gaps in the images of the inspection set.

[0135] Non-visual data may be juxtaposed on the point cloud. Vision may be used to determine the position and orientation of any non-visual sensors (e.g., optical gas imagers, thermographic cameras, and microphones). As a result, this will allow us to determine where the non-visual sensors are pointing on the surface of the object being observed. Therefore, all data may be collocated (i.e., assigned 3D coordinates, e.g., X, Y, Z coordinates on the surface of the object). Photogrammetry may be employed in addition to the above. The photogrammetry method used to generate the 3D model may differ from collocation photogrammetry.

[0136] From the above perspective, the geometric shape of the fluid plume may be identified. The geometric shape can be projected onto the surface of the 3D model. In this regard, the visualization of the fluid plume does not have to appear as a plume around the 3D model of the object. Such visualization may be advantageous for efficiently identifying the point of leakage without the plume being reproduced in 3D space. Also, numerous sweeps at different distances through the plume may not be necessary. For example, some conventional methods may employ a series of sweeps through numerous "slices" of the plume to characterize meaningful information about the point of leakage.

[0137] The 3D modeling described herein can avoid marginal inspection analysis. In this regard, data can be collected at the inspection event, and the analysis can be performed after the inspection without the need to analyze, judge, and / or adjust the marginal inspection.

[0138] The inspection device may perform a method for analyzing data collected by one or more of the sensors described herein. This method may be performed on the inspection device, off-site, or both. This method may be embodied by computer-readable instructions stored in one or more non-temporary storage media. This method may be executed by one or more processors.

[0139] The method may include determining the position and orientation ("pose") of one or more sensors within an environment having one or more objects placed therein. The sensors may include visual sensors and, optionally, one or more other types of sensors as described herein. Typically, the pose of at least one visual sensor may be determined, and the poses of one or more other sensors may be determined based on the pose of the visual sensor, based on the understanding that visual data can provide relatively more detail to aid in the accuracy of the output of the neural network discussed herein.

[0140] This method may include acquiring inspection data. During an inspection event, one or more of the sensors described herein may observe the object and / or the environment in which the object is located. Each part of the observation data may be obtained from the orientation of the object being observed. The data may include, for each pixel, RGB data (however, other color models may also be assumed by this teaching), depth data, single-spectrum electromagnetic data, multispectral electromagnetic data, thermal data, acoustic data, chemical data, or any combination thereof. Electromagnetic data may include radiation, reflection, absorption, or any combination thereof. Acoustic data may include amplitude. Chemical data may include concentration.

[0141] At least visual data acquired by a visual sensor may be used to generate one or more point clouds for one or more images and / or 3D models. In this regard, any other type of data (e.g., thermal data, single-spectral electromagnetic data, multispectral electromagnetic data, chemical data, acoustic data, or any combination thereof) may be mapped onto the image or point cloud. In other words, any type of data discussed herein may be associated with coordinates in Euclidean space. Such coordinates allow the user to visualize different types of data on 2D images and / or 3D models. Furthermore, as discussed herein, the method seeks to provide texture data acquired from different types of sensors (e.g., thermal sensors) that are associated with the visual data pixel by pixel.

[0142] This method may include estimating one or more poses of corresponding one or more input images from a check set of visual data and optionally a check set of thermal data. The poses may be estimated by a CNN, which may include one or more layers that function to estimate the poses of semantically segmented images. The output of the CNN (estimated pose) may be referred to herein as the rough pose, in contrast to the fine pose discussed below.

[0143] With respect to thermal data, or other types of data discussed herein, semantic segmentation can assist in organizing inspection data. In one embodiment, a human operator may visually view a thermal image mapped onto a visual image to determine the temperature of an object and / or its dependent parts. In another embodiment, all temperature measurements of a single object may be averaged (e.g., mean, median, mode) or otherwise analyzed (e.g., maximum, minimum, etc.), and such values ​​may be assigned to the object and / or its accessories, so that the object and / or accessories can be identified by their characteristics.

[0144] This method may include generating one or more composite images for one or more corresponding coarse poses. The composite images may be generated by an interpolating neural network. The interpolating neural network may receive coarse poses from a CNN and output a composite image corresponding to the coarse pose. The composite images are predicted by the interpolating neural network based on the training dataset and the coarse poses estimated by the CNN.

[0145] When an interpolating neural network predicts a composite image for non-visual types of data discussed herein, it is assumed that the coarse orientation of the image is equal to the coarse orientation of the corresponding visual image. This assumption may be based on the assumption that the sensors are located on the same robot as the visual sensors. In this regard, the sensors may be placed in close proximity to each other (e.g., at distances of approximately 60 cm or less, 50 cm or less, 40 cm or less, 30 cm or less, 20 cm or less, or even 10 cm or less). This assumption may be adjusted in the refinement steps discussed below.

[0146] The rough pose estimated by the CNN can facilitate processing in the refinement step. That is, the refinement step attempts to adjust the rough pose so that the synthesized image is in harmony with the input image, so refinement may require further adjustments and thus more processing time. This method attempts to employ a CNN that estimates a rough pose close to the actual pose of the sensor. The rough pose may deviate from the actual sensor pose by less than 5%, less than 2%, less than 1%, or even less than 0.1%.

[0147] This method may include refining one or more coarse poses. The coarse poses may be refined to obtain a precise pose. The coarse poses may be refined by minimizing the difference between the composite image and the input image. This may be applied to visual images and images generated from any other type of sensor (e.g., thermal, acoustic, chemical, etc.) as optionally described herein. In this regard, the composite image may be shifted so that individual pixels of the composite image match individual pixels of the input image. Such a shift of the composite image may be characterized by a corresponding shift applied to the coarse pose. For example, shifting the pose of a visual sensor by 10 cm in the X direction results in a shift corresponding to the pixels of the image.

[0148] The refinement of visual and thermal images may be performed sequentially or simultaneously. The visual images may be refined first, followed by the thermal images.

[0149] A form of interpolation neural network may be employed to refine the rough pose. For example, iNeRF (Inverting Neural Radiance Field) may be employed. iNeRF may have an additional head for refining the rough pose of non-visual images. Therefore, refining the rough pose of visual images and their corresponding non-visual images may be performed simultaneously.

[0150] A head as referred to herein may be a mean module of a neural network specialized in identifying a desired output. Each head may receive input from the backbone of the neural network and generate a desired output. The input from the backbone may be common to all heads. The output of each head may be unique to the other heads. For example, a first head may be configured to predict a synthetic visual image (e.g., to reconstruct an image containing data that would otherwise be acquired from a camera), and a second head may be configured to predict a synthetic non-visual image (e.g., to reconstruct an image containing data that would otherwise be acquired from non-visual sensors described herein, e.g., chemical sensors, thermal sensors, acoustic sensors, or any combination thereof). Discrete non-visual data may be processed by a separate head.

[0151] The precise pose can be approximately 99% or more, 99.5% or even 99.9% or more accurate to the actual pose of the sensor acquired from the input image. This instruction assumes that, in some situations, the rough pose may be as accurate to the actual pose as the intended accuracy of the precise pose. In this regard, refinement may not be performed on a given image. However, the rough pose is usually less accurate to the actual pose than the precise pose.

[0152] The neural network described above may be trained on true 2D images, one or more 3D models derived from true 2D images, one or more 3D models constructed by humans via CAD software, one or more 3D models constructed with photogrammetry software, one or more 3D models constructed with point cloud software, or any combination thereof.

[0153] The method described in this instruction may not require position tracking technology embedded in the sensor. Inspection data may include images where posture is unknown. However, posture can be determined using this method.

[0154] The precise posture identified by this method may be used in downstream processes including generating a 3D model, stitching together 2D images, performing time-lapse analysis, performing joint analysis, or any combination thereof. Time-lapse analysis may refer to comparing the output of the above method, generated from the inspection dataset, with historical data. Joint analysis may refer to data ultimately obtained from different types of sensors to identify anomalies in the object and / or environment being inspected.

[0155] One example of the preferred methods described above may include those described in U.S. Provisional Application No. 63 / 422,043, which is incorporated herein by reference for all purposes.

[0156] In another embodiment, an inertial measurement unit ("IMU") may determine the rough orientation of the inspection device as each measurement, including images / frames, is acquired. The IMU may optionally work in conjunction with GPS and / or a real-time clock to estimate the rough orientation instead of photogrammetry. In this regard, the processing time and power associated with photogrammetry may be eliminated.

[0157] The use of inertial measurement units can offer some advantages over photogrammetry. Specifically, complex transformations, such as combinations of linear and rotational movements, or relatively large-scale movements (e.g., a user rapidly moving an inspection device along a large angle), can increase processing time and reduce the fidelity of photogrammetry. However, inertial measurement units can determine the orientation of the inspection device when data points are acquired, thus eliminating the need for photogrammetry processing.

[0158] This method may include estimating the release velocity. The release velocity may be estimated based on one or any combination of the release models described herein. The flow rate may generally be estimated based on measurements obtained from an open-air optical path gas sensor and an anemometer. Surprisingly, it has been found that the geometric shape of the escape plume can improve the accuracy of the release velocity estimation. In this regard, a release model that takes into account the geometric shape of the escape plume is proposed by this teaching for estimating the release velocity.

[0159] While not intended to be limited by theory, open-air optical path gas sensors may determine the integral of the gas concentration along the entire path of the laser, and therefore, the measured gas concentration of a escaping plume diffusing over a wide area may, under certain circumstances, be approximately equal to the measured gas concentration of an escaping plume concentrated in a narrow area. Consequently, their respective estimated emission velocities may be approximately equal. However, independent of wind effects, escaping plumes diffusing over a wide area can usually be perceived as exhibiting a higher emission velocity due to the outflow velocity that pushes the gas farther from their point of origin. The geometric shape of the escaping plume may provide a more complete assessment of the emission velocity.

[0160] The release model may include a buoyancy model. The buoyancy model may be characterized by the geometric shape of the escaping plume, which is concentrated near the point of origin and shifts away from the point of origin depending on the wind direction and wind speed. The buoyancy model exhibits a low release velocity due to a low outflow velocity, which does not push the escaping plume far away from the point of origin and / or in a direction opposite to the wind direction.

[0161] The release model may include a diffusion model. The diffusion model may be characterized by a wider diffusion compared to the buoyancy model and / or by a geometric shape of the escape plume that is shifted relatively more from the point of origin than in the buoyancy model. The diffusion model exhibits a moderate release velocity due to a moderate outflow velocity that pushes the escape plume from the point of origin and / or in a direction opposite to the wind direction.

[0162] The release model may include an ejection model. The ejection model may be characterized by the geometric shape of the escaping plume having greater diffusion compared to the diffusion model and / or being shifted relatively more from the point of origin than in the buoyancy model. The ejection model exhibits a large release velocity due to the large outflow velocity that pushes the escaping plume away from the point of origin and / or in a direction different from the wind direction.

[0163] This instruction assumes that a combination of multiple release models may be used to estimate the release velocity. That is, a escaping plume may have features of two or more release models. Each release model may be weighted based on their interaction with the geometric shape of the escaping plume, the estimated release velocity, wind direction, wind speed, or any combination thereof. For example, for a given escaping plume, 40% of the weight may be applied to the buoyancy model and 60% to the diffusion model.

[0164] The emission rate may be estimated, at least partially, by a neural network (e.g., a convolutional neural network). A visualization of the geometric shape of the escape plume may be provided as input to the neural network. A categorization of one or more emission models described herein may be provided as output from the neural network.

[0165] The neural network may be trained on a dataset that includes visualizations of escape plumes. The leaks that generate these escape plumes may be controlled, and therefore the emission rate may be known. In this way, ground truth may be established. The above may be called staged training data. The neural network may then be trained on unstaged data, i.e., actual leak data where the emission rate is unknown.

[0166] The emission rate may be determined based on one or more gas concentration measurements from an open-air optical path gas sensor, as well as one or more measurements of wind speed and / or wind direction from an anemometer. Surprisingly, it has been found that visualization of the geometric shape of the escape plume can improve the accuracy of the emission rate. In this regard, the emission rate estimated from the gas concentration measurements, as well as the wind speed and / or wind direction, may be corrected by incorporating the output of the neural network described above, i.e., the categorization of the geometric shape under one or more emission models.

[0167] This method may include creating a Digital Emissions Tag ("DET"). A DET may function to identify the presence of an leak, confirm the absence of a leak, locate the leak so that technicians can later find and repair it, provide evidence of regulatory compliance, or a combination of these.

[0168] Conventionally, technicians go to the site with gas sensors, such as optical gas imagers, and when a leak is detected, they indicate the source by various means, such as tying a ribbon around the source, driving a stake into the ground near the source, or even marking the source (with paint, chalk, etc.). This allows the source to be later found by the technician assigned to repair the leak.

[0169] The digital release tag may include at least one visualization of a gas leak (optionally overlaid in real time on an image or video in the case of a video), the highest concentration, and the release rate. The digital release tag may further include one or more of the following: GPS coordinates, orientation of the inspection device, time / date, temperature, and source ID. As used herein, “source ID” may refer to identification information of the leak source, such as a pipe, wellhead, compressor, or similar.

[0170] The source ID may be identified in various ways. In some embodiments, a neural network (e.g., a convolutional neural network) may be trained using the source image and therefore may output the source ID based on the image input. In some embodiments, GPS coordinates, orientation, and / or time may be used by referring to a predetermined map in which the source ID is located. In some embodiments, the user may manually input the source ID, for example, when the DET is generated or thereafter. This instruction assumes that one or any combination of the above methods may be used.

[0171] Typically, most leaks (for example, over 90%, or even over 95%) occur at the flange where two pipes are joined. However, this instruction also considers other leak sources, such as corrosion or mechanical damage along the length of the pipe. In this regard, the training data supplied to the neural network described above may be biased towards flange detection.

[0172] DET may be created by the user manually instructing the inspection device to "start" and "stop" recording. During recording, the user may sweep the inspection device over areas with the highest gas concentration above a threshold. The user may be alerted when this threshold is met or exceeded, as described herein. The user may remain stationary while sweeping the inspection device. DET may be created during a second sweep, as described herein. The sweep may result in visualization of escape plumes.

[0173] If there are multiple leaks in the same equipment, multiple DETs may be created for the same source ID.

[0174] This method may include generating a digital compliance record. The digital compliance record may function to certify that there was no leak, or that the leak was below a threshold release rate (e.g., in accordance with internal policy or government regulations).

[0175] The digital qualification record may include one or more of the following: GPS coordinates, orientation of the inspection device, time / date, temperature, and source ID. The digital qualification record may further include, for example, the highest concentration and release rate, even if the value is 0.

[0176] Digital emission tags and / or digital qualification records may be generated on or off the inspection device. Digital emission tags and / or digital qualification records may be stored in a database. Digital emission tags and / or digital qualification records may be communicated to stakeholders, such as facility operators, regulatory bodies, or third parties.

[0177] [Example 1] The sensor device was tested on-site to inspect for methane leaks emitted from the equipment. The sensor device consisted of a tunable semiconductor laser, a high-resolution camera, and an anemometer. The sensor device was handheld and carried by an operator through the inspection site. The operator may be considered a “low-experienced leak inspection and repair inspector” as described herein. Approaching the equipment being inspected, i.e., within approximately 10 meters or less, the operator walked around the equipment, sweeping the sensor device across the entire equipment as they walked.

[0178] Controlled leaks were induced during equipment selection, and the discharge rate of each controlled leak was known in a single-blind manner. Operators did not know which equipment the induced leaks were being released from, nor did they know the discharge rate of each induced leak. Furthermore, multiple false positives (false detections) were introduced into the field.

[0179] A 90% confidence level was achieved for a discharge rate detection level of 4.0 liters / minute. This means that confidence decreases below this level. For comparison, experienced leak detection and repair (LDAR) inspectors using optical gas imaging (OGI) devices (i.e., those with experience in 700 to 4,000 individual inspection events) have a 90% confidence level for discharge rate detection levels between 2.6 liters / minute and 7.7 liters / minute. Less experienced leak detection and repair inspectors using OGI devices (i.e., those with experience in 25 to 250 individual inspection events) have a 90% confidence level for discharge rate detection levels above 20 liters / minute.

[0180] Thus, this test demonstrated that a low-experience LDAR inspector using the sensor device of this application can achieve a 90% confidence level within the emission velocity detection level of a high-experience LDAR inspector.

[0181] The sensor device demonstrated a true positive detection rate of 92% and a false positive detection rate of 2%, which was higher than all other test participants. The next three best participants reported, a) a true positive detection rate of 59% and a false positive detection rate of 3%, b) a true positive detection rate of 70% and a false positive detection rate of 15%, and c) a true positive detection rate of 66% and a false positive detection rate of 13%, respectively.

[0182] [Example 2] The inspection device described in this instruction was validated for detection range at Lawrence Berkeley National Laboratory. The sensor device consisted of a tunable semiconductor laser, a high-resolution camera, and an anemometer. During validation, controlled leaks of 1 gram / hour, 2 grams / hour (0.05 liters / minute), 5 grams / hour, 10 grams / hour, 20 grams / hour, and 40 grams / hour were measured by the inspection device described in this instruction, and accurate results were obtained at all of the aforementioned leak levels. The performance of the inspection device was validated using measurements from a Semtech Hi-Flow extraction analyzer with an accuracy of 5% or more at a lower detection threshold of approximately 0.03 liters / minute.

[0183] Figures 1A and 1B show the inspection device 10 according to this teaching. The inspection device 10 is a handheld device comprising a handle 12 and a sensor unit 14 attached to the handle 12. The sensor unit 14 comprises a plurality of sensors 16, including a visual sensor 18, an open-air optical path gas sensor 20 having a radiator 22 and a receiver 24 (however, this teaching assumes a sensor technology in which the radiator and receiver are integrated into separate units), and a microphone 26. The plurality of sensors have observation axes 28 that are substantially parallel to each other and directed toward the object and / or region of interest for data acquisition.

[0184] The sensor unit 14 also includes a visible light laser 30 to assist the user in aiming the inspection device 10. In this regard, the user can visually determine where the multiple sensors 16 are pointed. The path of the visible light laser 30 is preferably substantially parallel to the observation axes 28 of the multiple sensors 16. The power button 32 is located on the top surface of the sensor unit 14, but this teaching assumes that the power button 32 may be located at any feasible location on the sensor unit 14 and / or the handle 12.

[0185] As shown in Figure 1B, the sensor unit 14 also includes a data transmission port 34 (an Ethernet port, as shown) and a power connection 36 for recharging the onboard battery (for powering the sensor, processor, graphical user interface, and similar). A modular access point 38 is located at the bottom of the sensor unit 14 and near the handle 12, where other sensors may be mounted on the inspection device 10, where the wires of other sensors may pass through, and where other sensors may be connected to the data transmission means and power supply. In Figure 4, the anemometer 52 is mounted on the modular access point 38, but in this teaching, any other sensors (including redundant sensors such as a second vision sensor) may be mounted on the modular access point 38. Since wind speed and wind direction in a typical area do not depend on the orientation of the sensor 16, it can be recognized that the position of the anemometer 52 does not necessarily need to be considered with respect to the observation axis 28 of the multiple sensors 16. However, it is generally recommended that the inspection device 10 be held within 50 meters, more preferably within 40 meters, more preferably within 30 meters, more preferably within 20 meters, or even more preferably within 10 meters, of the object being observed.

[0186] Figure 2 shows a graphical user interface 40 on the surface of the inspection device 10 opposite to the surface where the multiple sensors 16 are located. The graphical user interface 40 displays the methane concentration 42 measured by the open-air optical path gas sensor 20, the wind speed 44 measured by the anemometer 52 (shown in Figure 4), and the luminance 46 measured by the visual sensor 18 (luminance is related to the accuracy of the sensor measurement, which depends on the reflection of electromagnetic radiation).

[0187] This instruction assumes that the graphical user interface 40 can display any of the measurements discussed herein, and optionally a live feed from the visual sensor 18 (optionally, along with visualizations of quantitative and / or qualitative gas, thermal, acoustic measurements, or any combination thereof, juxtaposed on the live feed from the visual sensor 18). The graphical user interface 40 is touch-panel operable, allowing the user to interact with it, for example, by pressing a “start” button to begin data acquisition and an associated “stop” button to stop data acquisition. This instruction assumes that the inspection device 10 can operate in an observation-only mode, where instantaneous measurements are displayed to the user, when data does not need to be collected / recorded (i.e., stored in a non-temporary storage medium).

[0188] Instantaneous wind speed and / or wind direction, brightness, or any combination thereof may be advantageous to the user in orienting the inspection device 10 in the appropriate direction. In some embodiments, the displayed wind speed and wind direction prompt the user to orient the device upstream of the wind when the point of origin of the leak is located upstream. In some embodiments, the displayed brightness prompts the user to perform subsequent passes of the inspection device 10 or wait for the ambient light conditions to change in order to obtain optimal measurements. In this regard, excessive reflected light (e.g., from the sun) may interfere with the gas measurements discussed herein. It is also assumed by this teaching that various visual and / or acoustic indicators may be presented to the user via a graphical user interface.

[0189] Figure 3 shows the user 48 pointing the inspection device 10 towards the object 50 to be inspected. In a typical inspection event described herein, the user 48 may walk along a path and stop at various intervals to allow the inspection device 10 to sweep through areas of the object 50. After sweeping one area, the user 48 may move to other areas of the object 50. It can be recognized from this teaching that the inspection device 10 allows for the free movement of the user 48 without the need to install stationary sensors, background reflectors, or other types of equipment.

[0190] Figure 4 shows an anemometer 52 mounted on the inspection device 10. The anemometer 52 is mounted on a modular access point 38, and this teaching assumes that other sensors, including redundant sensors, may also be mounted in this position. The anemometer 52 extends from the front of the inspection device 10 to minimize or eliminate any influence of hand movements or other objects on the wind interacting with the anemometer. In this example, the hot-wire anemometer is located in a tube or groove formed in a protruding tab.

[0191] Figures 5A to 6B show an exemplary 3D model 54 of the object 50 and its surroundings (i.e., the ground on which it is located and other nearby structures / objects). One orientation of the 3D model is shown, but it is understood that the 3D model may be pitched, rolled, and yawed, or otherwise explored, in 3D space defined by the XYZ coordinate system, in order to view different orientations of the 3D model (e.g., the opposite side of the object 50). The 3D model 54 is constructed from data collected by multiple sensors 16 shown in Figures 1A and 1B.

[0192] In a typical inspection event, a visual sensor acquires multiple photographic frames (e.g., 500 or more, 1,000 or more, or even 2,000 or more) from various angles relative to one or more user positions, and a point cloud of the object 50 is constructed using the photogrammetry methods described herein. In this example, photogrammetry may be used to stitch together the various frames to form a visual representation of the object in digital 3D space. Furthermore, multiple photographic frames may be employed to construct a texture, which may be applied to the point cloud, and the texture may include features such as color and illuminance (including shadows 56). The texture may also be understood as including one or more surfaces of the observed object that include details less than 3 inches, less than 2 inches, or even less than 1 inch (e.g., dividing lines, grooves, ridges, hardware, and similar). The detection and visualization of shadows 56 may be particularly advantageous for the thermal measurements discussed herein, as areas of shadow that are generally cooled can be distinguished (via joint analysis) from those that might otherwise be considered abnormal thermal measurements.

[0193] An open-air optical path gas sensor acquires multiple data points of gas presence and concentration by directing a laser through a target gas and receiving reflected laser light reflected from the surface of an object 50 (attenuated by absorption of wavelength bands by the target gas). These data points may be used to construct a texture, which can be applied to a point cloud. The apparatus and method do not rely on an artificial background structure (e.g., a reflective sheet) placed behind the object. Through the visualization techniques discussed herein, gas measurements can be digitally projected onto a 3D model of the object itself, and comprehensive detection and quantification of the gas can be achieved by sweeping the inspection device 10 across various areas of the object 50. The method is unique in its emphasis on concentration measurements obtained by reflection from the object, with attention focused on this data rather than on the plume spreading in the atmosphere.

[0194] Data from visual sensors and open-air optical path gas sensors are correlated with the same object observed by the sensor at any given time by correlating position coordinates, timestamps, or both. In this regard, during an inspection event, multiple areas of one object, or multiple different objects, may be inspected, and accurate correlation of data must be performed to construct a 3D model 54 by juxtaposing visual data and gas data (and optionally other types of data discussed herein) on the same 3D model. Furthermore, sensor position offsets are corrected so that the sensor's observation axis is (digitally) aligned with a single reference line, and the position offset is not reflected in the 3D model (i.e., presented as a "shift" of different textures). It can be recognized from this teaching that misalignment of data may result in inaccurate location of escape gas sources and, in some cases, errors in the accurate construction of the 3D model.

[0195] Data from the visual sensor and the open-air optical path gas sensor are collated to assign 3D coordinates to both types of data. In this regard, the method described herein does not rely merely on superimposing, or otherwise "shifting" or "matching," the images obtained from the visual sensor and the images obtained from the optical path gas sensor (and optionally, other sensors described herein), because this can result in positional shifts that lead to inaccuracies in the 3D model. The method described herein can be made more robust by assigning 3D coordinates to data points in the dataset (e.g., the dataset obtained from the visual sensor and the dataset obtained from the open-air optical path gas sensor) so that data points with corresponding 3D coordinates can be mapped to a 3D model. In some embodiments, the mapping may be performed in a downstream process, for example, on a laptop or desktop computer with data subsequently uploaded thereto; however, this teaching assumes that the same may be performed on the inspection device.

[0196] In this regard, collocation can be understood as meaning that a visual image can be represented as a pixel array on an XY coordinate system, and that the position or region on an object observed by the laser can be correlated with the coordinates on the pixel array, using a predetermined positioning of an open-air optical path gas sensor relative to a visual sensor. Furthermore, the depth or Z coordinate of a visual image, as identified by photogrammetry or other methods described herein (e.g., stereo cameras and / or LiDAR), can similarly be assigned to gas data. Thus, non-visual data points described herein (e.g., quantitative data points for gas concentration, heat, acoustic frequency, and similar) can be individually assigned 3D coordinates so that they can be accurately mapped to a 3D model.

[0197] This instruction can be understood that the inspection dataset may contain multiple duplicate or dual data points for a single 3D coordinate. Therefore, duplicate data can be recognized and discarded.

[0198] Although this example presents gas data, this instruction assumes that the above explanation can be similarly applied to data from a thermographic camera, microphone, or both.

[0199] The texture containing the gas data is presented as a heatmap, where brighter areas indicate higher gas concentrations than darker areas. It is also understood that the heatmap can be visually represented using various colors (e.g., red, green, and blue) and their hues corresponding to the gas concentrations. As depicted, the heatmap has roughly five main regions A-E, with the highest gas concentration in region A and gradually decreasing gas concentrations in regions B-E. The source of the gas leak can be identified from the heatmap in terms of where the highest gas concentration exists, or at least in close proximity to it (based on contextual data and the common sense of an operator who can understand the likely escape route of the gas). As depicted, a pipe extending vertically along a cylindrical tank and terminating just above the tank has the highest gas concentration at the termination point just above the tank, and the concentration gradually decreases as it moves away from the termination point of the pipe.

[0200] In this regard, it can be recognized that wind data affects the analysis. In the absence of wind, the gas concentrations would be expected to be distributed almost symmetrically around the point of gas leakage, as shown in Figures 5A and 5B. However, as depicted, the geometric shape of the gas plume is shifted to the right of the point of origin with respect to the XYZ coordinates of the 3D model depicted in digital space, due to the wind generally moving from left to right. In another example, Figures 6A and 6B show an almost symmetrical distribution of gas concentration regions A-C centered on the point of origin within region A, and therefore, the wind does not manipulate the escaping gas plume to the same extent as in Figures 5A and 5B. One or more vectors transmitting wind speed and wind direction may be provided as textures that can be visualized on the 3D model.

[0201] In these examples, the object itself can be perceived as functioning as a background reflector, with the gas concentration measurements projected onto a 3D model of the object. In this regard, the present device and method are distinguished from conventional gas detection methods that have been more interested in real-time gas visualization than in providing the modeling and more robust visualization and analysis techniques described herein.

[0202] Typically, facilities are equipped with stairs and elevated walkways (and even other elevated vantage points) that provide visual access to different angles of the object from which the gas may escape. This is especially true for facilities with vertically extending equipment, such as tall tanks extending more than 5 meters, 10 meters, 15 meters, 20 meters, 25 meters, or even 30 meters above the ground, and vertically extending pipelines. Users may traverse these mechanisms to view the object from different angles. This teaching also envisions other aerial observation means, such as aerial drones, to obtain different views of the object. Aerial observation means may be employed alone or in conjunction with a handheld version of the inspection device. Aerial observation means may comprise some or all of the components / mechanisms mounted on the inspection device. Elevated vantage points can be advantageous for obtaining a complete 3D model, but are not necessarily required.

[0203] As shown in Figure 6A, an elevated walkway is constructed adjacent to the tank. While the 3D model is viewed from a position away from the object, the 3D model can be rendered using sensor observations obtained from the ground and the elevated walkway, as well as from aerial methods.

[0204] Figures 5B and 6B show the boundaries around the point of leakage. The boundaries may be drawn by the user through modeling software or may be drawn automatically based on a predetermined set of rules. Within the boundaries, data points may be analyzed to identify peak concentration and release rate. This analysis may be advantageous for triaging repair activities across the facility, thereby allowing some repairs to be prioritized over others. For example, peak concentration may be of particular importance for flammable gases that have the characteristic of igniting at a certain atmospheric concentration, and release rate may be of importance when product loss or environmental standards are of concern. In some embodiments, minor leaks may be made known over time and monitored to identify whether those concerns escalate. With the unique and unconventional devices of this teaching, release rate may be identified by considering wind speed and direction to model the concentration measurements of the escaping plume, as well as the diffusion of the gas.

[0205] Figures 7A to 7I show the inspection device 58. The inspection device 58 comprises a handle 60 and a sensor unit 62. The sensor unit 62 comprises a plurality of sensors 64. The plurality of sensors 64 include a visual sensor 66, an open-air optical path gas sensor 68 comprising a radiator 70 and a receiver 72, and an anemometer 74. The plurality of sensors 64 are oriented forward in the inspection device 58 so that the user directs the inspection device 58 towards an object or area of ​​interest to perform measurements. In this regard, the inspection device 58 comprises a visible light laser 76 that irradiates the object or area of ​​interest with visible light, and thus assists the user in aiming the inspection device 58.

[0206] The visual sensor 66 may be adjustable so that its observation axis can substantially coincide with the observation axis of the open-air optical path gas sensor 20.

[0207] The anemometer 74 may be retractable within the inspection device 58. Retraction and extension may be performed manually or mechanically. While the inspection device 58 is operating, the anemometer 64 may be extended from the inspection device 58 to be exposed to the wind.

[0208] The inspection device 58 is equipped with a door 80 at its front end to cover one or more data transmission ports 84 and to prevent water and / or debris from entering. The one or more data transmission ports 84 may include one or more USB ports (e.g., two or more, three or more, or even four or more USB ports), one or more Ethernet ports, or both.

[0209] The inspection device 58 includes a power button 78 for turning the inspection device 58 on or off. The inspection device includes one or more (e.g., two) modular access points 82 for attaching accessories. Accessories may include additional sensors, lighting (e.g., floodlights to assist visibility), or the like.

[0210] The inspection device 58 is equipped with a graphical user interface 86 at its rear. The user can view real-time measurements, view real-time video feeds, control the inspection device 58, or combine them as desired, via the graphical user interface 86.

[0211] The handle 60 is provided with a power connection 88 at its bottom, but this teaching is not intended to limit the location of the power connection 88 to any particular place. The inspection device 58 may be equipped with an onboard battery that can be charged via the power connection 88.

[0212] Figure 8 shows exemplary escaping plumes 90 adapted to buoyancy, diffusion, and ejection models. The figure is not intended to be limiting, as it can be recognized from this teaching that the geometric shape of a escaping plume can be complex and take on numerous shapes. Escaped plume 90 [A] corresponds to the buoyancy model, escaping plume 90 [B] corresponds to the diffusion model, and escaping plume 90 [C] corresponds to the ejection model. The wind vectors (w) are equal in terms of direction and magnitude. The leakage vectors (l) are equal in terms of direction but differ in magnitude. Escaped plume [A] does not have a leakage vector (l) because it is assumed to be negligible, although normally all leakage can have direction and velocity at their point of origin 90.

[0213] Figure 8 shows that, in general, the models can be distinguished by the relative effects of the leakage vector (l) and wind vector (w) on the escape plume 90. A dashed line extending through the point of origin 90 is provided for visual aids. [A] shows that the geometric shape of the escape plume 90 is determined by the wind vector (w). [B] shows that the geometric shape of the escape plume 90 is mainly determined by the wind vector (w), with relatively little influence from the leakage vector (l). [C] shows that the geometric shape of the escape plume 90 is mainly determined by the leakage vector (l), with relatively little influence from the wind vector (w).

[0214] Figure 9 shows a digital emission tag as described herein. The digital emission tag can be viewed on the graphical user interface of the inspection device. The digital emission tag can be started by a start button on the graphical user interface, thereby initiating data collection. Methane concentration, emission rate, wind speed, maximum methane concentration, and video feed are recorded. A visualization of the region of interest, along with a visualization of the escaping plume placed on top of it, is provided on the graphical user interface. Data recording is tracked by a timer. At the end of data collection, the digital emission tag can be stopped by a stop button on the graphical user interface, and the digital emission tag can be saved by a tag button on the graphical user interface.

[0215] The descriptions and illustrations presented herein are intended to make the present invention, its principles, and its practical applications familiar to those skilled in the art. The above description is illustrative and not limiting. Those skilled in the art may adapt and apply the present invention to its many forms to best suit the requirements of a particular application.

[0216] Accordingly, specific embodiments of the Invention as described herein are not intended to exhaust or limit this teaching. The scope of this teaching should therefore not be determined by reference to this description, but rather by reference to the appended claims, together with the entire scope of the equivalents to which those claims are entitled. The omission of any aspect of the subject matter disclosed herein in the following claims does not constitute an abandonment of such subject matter, nor should it be deemed that the inventor did not consider such subject matter to be part of the subject matter of the disclosed invention.

[0217] Multiple elements or steps may be provided by a single, integrated element or step. Alternatively, a single element or step may be divided into multiple separate elements or steps.

[0218] The disclosure of “one (a)” or “one (one)” to describe an element or step is not intended to exclude any additional elements or steps.

[0219] Terms such as "first," "second," and "third" may be used herein to describe various elements, components, regions, layers, and / or parts, but these elements, components, regions, layers, and / or parts should not be limited by these terms. These terms may be used to distinguish one element, component, region, layer, or part from another region, layer, or part. For example, terms such as "first," "second," and other numerical terms, when used herein, do not suggest order or sequence unless explicitly indicated by the context. Therefore, the first element, component, region, layer, or part discussed below may be referred to as the second element, component, region, layer, or part without deviation from this teaching.

[0220] For example, spatially relative terms such as “inside,” “outside,” “beneath,” “below,” “lower,” “above,” “upper,” and similar terms may be used herein to facilitate explanation in describing the relationship of one element or feature to another, as shown in the figure. Spatially relative terms may be intended to encompass various orientations of a device in use or operation, in addition to the orientation depicted in the figure. For example, if the device in the figure is rotated, an element described as “below” or “below” another element or feature will now be positioned “above” the other element or feature. Thus, the exemplary term “below” may encompass both upward and downward orientations. The device may be oriented in other orientations (rotated 90 degrees or otherwise), and the spatially relative descriptive terms used herein may be interpreted accordingly.

[0221] All disclosures of articles and references, including patent applications and published gazettes, are incorporated by reference for all purposes. Other combinations, as can be read from the following claims, are also possible and are also incorporated by reference in this specification.

[0222] Unless otherwise stated, all numerical values ​​described herein include all values ​​from the lower value to the upper value in increments of one unit, provided that there is at least a two-unit interval between any lower value and any upper value. For example, if a component quantity, characteristic, or numerical value is described as being from 1 to 90, 20 to 80, or 30 to 70, then intermediate range values ​​(e.g., 15 to 85, 22 to 68, 43 to 51, 30 to 32, etc.) are intended to be within the scope of the teachings herein. Similarly, individual intermediate values ​​are also within the scope of the teachings herein. For values ​​less than 1, one unit is considered to be 0.0001, 0.001, 0.01, or 0.1, as necessary. These are merely examples of what is specifically intended, and it should be considered that all possible numerical combinations between the listed lowest and highest values ​​are explicitly described in a similar manner in this application. Unless otherwise stated, all ranges include both endpoints.

[0223] The use of "about" or "approximately" in relation to a range applies to both ends of the range. Therefore, "about 20 to 30" is intended to cover "about 20 to about 30" that include at least one specific endpoint.

[0224] The terms "approximately" or "substantially" used to describe measurements may mean approximately + / -10° or less, approximately + / -5° or less, or even approximately + / -1° or less. The terms "approximately" or "substantially" used to describe measurements may mean approximately + / -0.01° or more, approximately + / -0.1° or more, or even approximately + / -0.5° or more. [Explanation of Symbols]

[0225] [Reference number] 10...Inspection device, 12...Handle, 14...Sensor unit, 16...Multiple sensors, 18...Vision sensor, 20...Open-air optical path gas sensor, 22...Radiator, 24...Receiver, 26...Microphone, 28...Observation axis, 30...Visible light laser, 32...Power button, 34...Data transmission port, 36...Power connection, 38...Modular access point, 40...Graphical user interface, 42...Methane concentration, 44...Wind speed, 46...Brightness, 48...User, 50...Target object 52... Anemometer, 54... 3D model, 56... Shadow, 58... Inspection device, 60... Handle, 62... Sensor unit, 64... Multiple sensors, 66... ​​Visual sensor, 68... Open-air optical path gas sensor, 70... Radiator, 72... Receiver, 74... Anemometer, 76... Visible light laser, 78... Power button, 80... Door, 82... Modular access point, 84... Data transmission port, 86... Graphical user interface, 88... Power connection, 90... Escape plume, 92... Point of origin

Claims

1. An inspection device, Vision sensors, and Optical gas imager, At least one anemometer, Thermographic camera, microphone, Multiple sensors, including one or more of the following: A real-time clock configured to timestamp individual data points or groups of data points generated by the plurality of sensors, Inertial measurement unit, Equipped with, Each of the aforementioned multiple sensors is characterized by a central observation axis that is aligned in parallel, The plurality of sensors and the real-time clock are configured to operate simultaneously to generate the data points from at least two orientations of the inspection device with respect to the object observed by the inspection device. Inspection device.

2. The inspection apparatus according to claim 1, further comprising a position module configured to assign position coordinates to individual data points or groups of data points generated by the plurality of sensors, wherein the position module is configured to operate simultaneously with the plurality of sensors and the real-time clock to generate the data points from at least two orientations of the inspection apparatus with respect to the object observed by the inspection apparatus.

3. The inspection apparatus according to claim 2, wherein the position module is a GPS module.

4. The inspection apparatus according to claim 1, wherein the visual sensor is an RGB camera or a stereo camera.

5. The inspection apparatus according to claim 4, wherein the plurality of sensors include the optical gas imager and the at least one anemometer.

6. The inspection apparatus according to claim 5, wherein the at least one anemometer includes a hot-wire anemometer, the hot-wire anemometer extending from the front of the inspection apparatus and protruding beyond the other sensors of the plurality of sensors.

7. The inspection device according to claim 5, wherein the at least one anemometer includes a first anemometer and a second anemometer, and the inspection device has a first groove and a second groove in which the first and second anemometers are respectively located, the first groove is oriented perpendicular to the central observation axis, and the second groove is aligned parallel to the central observation axis.

8. The inspection apparatus according to claim 5, wherein the optical gas imager is a tunable semiconductor laser configured to perform tunable semiconductor laser absorption spectroscopy.

9. The inspection device according to claim 1, wherein the inspection device is handheld, and the handheld inspection device comprises the optical gas imager, the thermographic camera, and the microphone.

10. The inspection apparatus according to claim 9, wherein the inspection apparatus does not include a LIDAR sensor.

11. The inspection apparatus according to claim 10, further comprising a visible laser that generates a beam parallel to the central observation axis of the plurality of sensors, wherein the visible laser is configured to assist the user in tracing a path across the entire area of ​​the object to be observed.

12. The inspection apparatus according to claim 10, further comprising a graphical user interface configured to enable data from one or more of the plurality of sensors to be displayed on the graphical user interface in real time or substantially in real time.

13. below, (a) One or more wired data transmission modules and / or wireless data transmission modules, (b) Battery, (c) A printed circuit board comprising a processor and / or a non-temporary memory storage medium, The inspection apparatus according to claim 11, further comprising one or more of the following.

14. A method for operating an inspection device, Acquiring data from two or more of the multiple sensors of the inspection device, wherein the multiple sensors include a visual sensor and one or more other sensors, and includes a) sweeping the inspection device across a region in a substantially parallel path from a fixed position, and b) surrounding the region with the inspection device from the fixed position, The process of timestamping the data and / or assigning position coordinates to the data, wherein the position coordinates include the position and orientation of the inspection device, Correcting the positional offset of the observation axes of the two or more sensors, The data from the aforementioned visual sensor is collated with data from one or more other sensors among the plurality of sensors. In order to generate point clouds and 3D textures of the visual data, photogrammetry is performed on the visual data, The process involves generating a 3D model using one or more textures from the aforementioned multiple sensors, Methods that include...

15. The one or more other sensors include an optical gas imager, and the method is Discard duplicate and / or irrelevant data, In order to determine whether the illuminance is within a predetermined operating range for the optical gas imager, the illuminance is verified using the visual sensor, The method according to claim 14, further comprising one or more of the above.

16. The method according to claim 14, further comprising performing a time-lapse analysis by comparing data from the immediate inspection event with data from the pre-inspection event, which includes data for the same object observed in the pre-inspection event and the immediate inspection event.

17. The one or more other sensors include one or more of an optical gas imager, at least one anemometer, a thermographic camera, and a microphone, and the inspection device is A real-time clock configured to timestamp individual data points or groups of data points generated by the aforementioned multiple sensors, Inertial measurement unit, Furthermore, Each of the aforementioned multiple sensors is characterized by a central observation axis that is aligned in parallel, The plurality of sensors and the real-time clock are configured to operate simultaneously to generate the data points from at least two orientations of the inspection device with respect to the object observed by the inspection device. The method according to claim 14.

18. The method according to claim 17, wherein the inspection apparatus further comprises a position module configured to assign position coordinates to individual data points or groups of data points generated by the plurality of sensors, the position module configured to operate simultaneously with the plurality of sensors and the real-time clock to generate the data points from at least two orientations of the inspection apparatus with respect to the object observed by the inspection apparatus.

19. The method according to any one of claims 17 or 18, further comprising estimating the release rate of a gas leak that is released from a point of origin and discharged into the atmosphere, thereby forming a escapable plume.

20. The method according to claim 19, wherein the one or more other sensors include the optical gas imager and the at least one anemometer, and the estimation includes measuring the gas concentration with the optical path imager, measuring the wind speed and / or wind direction with the at least one anemometer, and generating an estimate of the discharge rate based on the gas concentration and the wind speed and / or wind direction.

21. The method according to claim 20, wherein the estimation further comprises generating a visualization of the geometric shape of the escape plume based on a plurality of gas concentration measurements, providing the visualization as input to a convolutional neural network, and adjusting the estimate based on the category output of the convolutional neural network.

22. The method according to claim 21, wherein the convolutional neural network is trained using a plurality of visualizations of the geometric shape of the escape plume, and the ground truth of each of the plurality of visualizations includes discrete categories of emission models defined by diffusion from the point of origin and positional shift relative to the point of origin.

Images