System and method for pipe monitoring

Abstract

Provided is a system and method for pipe monitoring. The system includes a sensor arrangement comprising a visual sensor, an inertial sensor, and a laser projector, and at least one processor configured to: project a laser on an inner surface of a pipe with the laser projector; emit light with a light-emitting device on the inner surface of the pipe; capture a plurality of frames of the inner surface of the pipe with the visual sensor, wherein at least a subset of frames of the plurality of frames is captured while the laser is projected on the inner surface, and wherein the at least a subset of frames or a different subset of frames is captured while the light illuminates the inner surface; and generate a 3D reconstruction of the inner surface of the pipe based on the plurality of frames.

Description

SYSTEM AND METHOD FOR PIPE MONITORINGCROSS REFERENCE TO REPLATED APPLICATIONS

[0001] This application claims the benefit of United States Provisional Patent Application No. 63 / 541,592 filed on September 29, 2023 and United States Provisional Patent Application No. 63 / 541,627 filed on September 29, 2023, the disclosures of which are hereby incorporated by reference in their entireties.GOVERNMENT RIGHTS NOTICE

[0002] This invention was made with government support under Grant No. DE-AR0001331 awarded by the U.S. Department of Energy (DOE). The government has certain rights in the invention.BACKGROUND1. Field

[0003] This disclosure relates generally to pipe monitoring and, in non-limiting embodiments, to systems and methods for pipe monitoring using simultaneous localization and mapping.2. Technical Considerations

[0004] Pipelines, crucial infrastructures supporting human civilization, may experience degradation from various factors like corrosion, geological subsidence, and improper plumbing or digging, leading to economic losses and hazardous incidents. For inspecting the inner surface of the pipes, existing techniques use closed-circuit television (CCTV) cameras for basic visual inspection, often relying on human operators for time-consuming data collection and video analysis. Due to the monomodal nature of 2D images, such methods may not be able to identify and localize anomalies in the pipeline.SUMMARY

[0005] According to non-limiting embodiments or aspects, provided is a system comprising: a sensor arrangement comprising a visual sensor, an inertial sensor, and a laser projector; at least one processor configured to: project a laser line on an inner surface of a pipe with the laser projector; emit light with a light-emitting device on the inner surface of the pipe; capture a plurality of frames of the inner surface of the pipe with the visual sensor, wherein at least a subset of frames of the plurality of frames is captured while the laser line is projected on the inner surface, and the at least a subset of frames or a different subset of frames is capturedwhile the light illuminates the inner surface; and generate a 3D reconstruction of the inner surface of the pipe based on the plurality of frames.

[0006] In non-limiting embodiments or aspects, the at least one processor is further configured to: repeatedly switch between projecting the laser line on the inner surface and emitting the light on the inner surface, wherein capturing the plurality of frames comprises capturing a plurality of profiling frames while the laser line is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface, the at least a subset of frames comprises the plurality of profiling frames, and the different subset of frames comprises the plurality of visual frames. In non-limiting embodiments or aspects, the sensor arrangement further comprises a LIDAR device, and the at least one processor is further configured to: capture a plurality of depth measurements of the inner surface of the pipe with the LIDAR device, the 3D reconstruction of the inner surface is generated based on the plurality of depth measurements. In non-limiting embodiments or aspects, the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm. In non-limiting embodiments or aspects, the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector. In nonlimiting embodiments or aspects, the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visual-depth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement. In non-limiting embodiments or aspects, the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization. In non-limiting embodiments or aspects, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

[0007] According to non-limiting embodiments or aspects, provided is a method comprising: projecting a laser line on an inner surface of a pipe with a laser projector; emitting light with a light-emitting device on the inner surface of the pipe; capturing a plurality of frames of the inner surface of the pipe with a visual sensor, wherein at least a subset of frames of the plurality of frames is captured while the laser line is projected on the inner surface, and the atleast a subset of frames or a different subset of frames is captured while the light illuminates the inner surface; and generating a 3D reconstruction of the inner surface of the pipe based on the plurality of frames.

[0008] In non-limiting embodiments or aspects, further comprising: repeatedly switching between projecting the laser line on the inner surface and emitting the light on the inner surface, wherein capturing the plurality of frames comprises capturing a plurality of profiling frames while the laser line is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface, the at least a subset of frames comprises the plurality of profiling frames, and the different subset of frames comprises the plurality of visual frames. In non-limiting embodiments or aspects, further comprising: capturing a plurality of depth measurements of the inner surface of the pipe with a LIDAR device, the 3D reconstruction of the inner surface is generated based on the plurality of depth measurements. In non-limiting embodiments or aspects, the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm. In non-limiting embodiments or aspects, the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector. In non-limiting embodiments or aspects, the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of a sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visual-depth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement. In non-limiting embodiments or aspects, the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization. In non-limiting embodiments or aspects, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

[0009] According to non-limiting embodiments or aspects, provided is a system comprising: a sensor arrangement comprising a visual sensor, an inertial sensor, a LIDAR device, and a laser projector; and at least one processor configured to: repeatedly switch between projecting a laser line on an inner surface of a pipe with the laser projector and emitting light with a light-emitting device onto the inner surface of the pipe; capture a plurality ofprofiling frames while the laser line is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface; capture a plurality of depth measurements of the inner surface of the pipe with the LIDAR device; and generate a 3D reconstruction of the inner surface of the pipe based on the plurality of profiling frames, the plurality of visual frames, and the plurality of depth measurements.

[0010] In non-limiting embodiments or aspects, the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm. In non-limiting embodiments or aspects, the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector. In non-limiting embodiments or aspects, the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visual-depth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement. In non-limiting embodiments or aspects, the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding- window-based nonlinear optimization. In non-limiting embodiments or aspects, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

[0011] According to non-limiting embodiments or aspects, provided is a method comprising: repeatedly switching between projecting a laser line on an inner surface of a pipe with a laser projector and emitting light with a light-emitting device onto the inner surface of the pipe; capturing a plurality of profiling frames while the laser line is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface; capturing a plurality of depth measurements of the inner surface of the pipe with a LIDAR device; and generating a 3D reconstruction of the inner surface of the pipe based on the plurality of profiling frames, the plurality of visual frames, and the plurality of depth measurements. In non-limiting embodiments or aspects, the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm.

[0012] In non-limiting embodiments or aspects, the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laserprojector. In non-limiting embodiments or aspects, the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of a sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visual-depth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement. In non-limiting embodiments or aspects, the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization. In non-limiting embodiments or aspects, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map. According to non-limiting embodiments or aspects, provided is a computer program product comprising a non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to perform any of the methods previously recited.

[0013] In non-limiting embodiments or aspects, the sensor arrangement further comprises at least one of the following: an ultrasonic sensor, a wheeled encoder, a sonde locator, an eddy current sensor, an electromagnetic sensor, a geographic positioning system (GPS) sensor, and / or any combination thereof. In non-limiting embodiments or aspects, the at least one processor is further configured to: simulate the sensor arrangement in a virtual environment; simulate the pipe in the virtual environment; and control the simulated sensor arrangement in the simulated pipe. In non-limiting embodiments or aspects, the at least one processor is further configured to: automatically generate a plurality of different simulated pipes in a virtual environment; simulate the sensor arrangement in a virtual environment; control the simulated sensor arrangement in the plurality of different simulated pipes; and record simulated sensor data while controlling the simulated sensor arrangement. In non-limiting embodiments or aspects, the method further includes: capturing additional sensor data with at least one of the following: an ultrasonic sensor, a wheeled encoder, a sonde locator, an eddy current sensor, an electromagnetic sensor, a geographic positioning system (GPS) sensor, and / or any combination thereof, the 3D reconstruction is generated based on the additional sensor data. In non-limiting embodiments or aspects, the method further includes: simulating the sensor arrangement in a virtual environment; simulating the pipe in the virtual environment; and controlling thesimulated sensor arrangement in the simulated pipe. In non-limiting embodiments or aspects, the method further includes: automatically generating a plurality of different simulated pipes in a virtual environment; simulating the sensor arrangement in a virtual environment; controlling the simulated sensor arrangement in the plurality of different simulated pipes; and recording simulated sensor data while controlling the simulated sensor arrangement.

[0014] Other preferred and non-limiting embodiments or aspects of the present invention will be set forth in the following numbered clauses:

[0015] Clause 1 : A system comprising: a sensor arrangement comprising a visual sensor, an inertial sensor, and a laser projector; at least one processor configured to: project a laser line on an inner surface of a pipe with the laser projector; emit light with a light-emitting device on the inner surface of the pipe; capture a plurality of frames of the inner surface of the pipe with the visual sensor, wherein at least a subset of frames of the plurality of frames is captured while the laser line is projected on the inner surface, and wherein the at least a subset of frames or a different subset of frames is captured while the light illuminates the inner surface; and generate a 3D reconstruction of the inner surface of the pipe based on the plurality of frames.

[0016] Clause 2: The system of clause 1, wherein the at least one processor is further configured to: repeatedly switch between projecting the laser line on the inner surface and emitting the light on the inner surface, wherein capturing the plurality of frames comprises capturing a plurality of profiling frames while the laser line is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface, wherein the at least a subset of frames comprises the plurality of profiling frames, and wherein the different subset of frames comprises the plurality of visual frames.

[0017] Clause 3: The system of any of clauses 1-2, wherein the sensor arrangement further comprises a LIDAR device, and wherein the at least one processor is further configured to: capture a plurality of depth measurements of the inner surface of the pipe with the LIDAR device, wherein the 3D reconstruction of the inner surface is generated based on the plurality of depth measurements.

[0018] Clause 4: The system of any of clauses 1-3, wherein the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm.

[0019] Clause 5: The system of any of clauses 1-4, wherein the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector.

[0020] Clause 6: The system of any of clauses 1-5, wherein the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation ofeach state of a plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visual-depth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement.

[0021] Clause 7: The system of any of clauses 1-6, wherein the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization.

[0022] Clause 8: The system of any of clauses 1-7, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

[0023] Clause 9: A method comprising: projecting a laser line on an inner surface of a pipe with a laser projector; emitting light with a light-emitting device on the inner surface of the pipe; capturing a plurality of frames of the inner surface of the pipe with a visual sensor, wherein at least a subset of frames of the plurality of frames is captured while the laser line is projected on the inner surface, and wherein the at least a subset of frames or a different subset of frames is captured while the light illuminates the inner surface; and generating a 3D reconstruction of the inner surface of the pipe based on the plurality of frames.

[0024] Clause 10: The method of clause 9, further comprising: repeatedly switching between projecting the laser line on the inner surface and emitting the light on the inner surface, wherein capturing the plurality of frames comprises capturing a plurality of profiling frames while the laser line is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface, wherein the at least a subset of frames comprises the plurality of profiling frames, and wherein the different subset of frames comprises the plurality of visual frames.

[0025] Clause 11 : The method of any of clauses 9-10, further comprising: capturing a plurality of depth measurements of the inner surface of the pipe with a LIDAR device, wherein the 3D reconstruction of the inner surface is generated based on the plurality of depth measurements.

[0026] Clause 12: The method of any of clauses 9-11, wherein the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm.

[0027] Clause 13: The method of any of clauses 9-12, wherein the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector.

[0028] Clause 14: The method of any of clauses 9-13, wherein the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of a sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visualdepth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement.

[0029] Clause 15: The method of any of clauses 9-14, wherein the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization.

[0030] Clause 16: The method of any of clauses 9-15, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

[0031] Clause 17: A system comprising: a sensor arrangement comprising a visual sensor, an inertial sensor, a LIDAR device, and a laser projector; and at least one processor configured to: repeatedly switch between projecting a laser line on an inner surface of a pipe with the laser projector and emitting light with a light-emitting device onto the inner surface of the pipe; capture a plurality of profiling frames while the laser line is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface; capture a plurality of depth measurements of the inner surface of the pipe with the LIDAR device; and generate a 3D reconstruction of the inner surface of the pipe based on the plurality of profiling frames, the plurality of visual frames, and the plurality of depth measurements.

[0032] Clause 18: The system of clause 17, wherein the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm.

[0033] Clause 19: The system of any of clauses 17-18, wherein the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector.

[0034] Clause 20: The system of any of clauses 17-19, wherein the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visualdepth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement.

[0035] Clause 21 : The system of any of clauses 17-20, wherein the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization.

[0036] Clause 22: The system of any of clauses 17-21, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

[0037] Clause 23: A method comprising: repeatedly switching between projecting a laser line on an inner surface of a pipe with a laser projector and emitting light with a light-emitting device onto the inner surface of the pipe; capturing a plurality of profiling frames while the laser line is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface; capturing a plurality of depth measurements of the inner surface of the pipe with a LIDAR device; and generating a 3D reconstruction of the inner surface of the pipe based on the plurality of profiling frames, the plurality of visual frames, and the plurality of depth measurements.

[0038] Clause 24: The method of clause 23, wherein the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm.

[0039] Clause 25: The method of any of clauses 23-24, wherein the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector.

[0040] Clause 26: The method of any of clauses 23-25, wherein the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of a sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weightedsum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visualdepth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement.

[0041] Clause 27: The method of any of clauses 23-26, wherein the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization.

[0042] Clause 28: The method of any of clauses 23-27, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

[0043] Clause 29: A computer program product comprising a non-transitory computer- readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to perform any of the methods previously recited

[0044] Clause 30: The system of any of clauses 1-8 or 17-22, wherein the sensor arrangement further comprises at least one of the following: an ultrasonic sensor, a wheeled encoder, a sonde locator, an eddy current sensor, an electromagnetic sensor, a geographic positioning system (GPS) sensor, and / or any combination thereof.

[0045] Clause 31 : The system of any of clauses 1-8 or 17-22, wherein the at least one processor is further configured to: simulate the sensor arrangement in a virtual environment; simulate the pipe in the virtual environment; and control the simulated sensor arrangement in the simulated pipe.

[0046] Clause 32: The system of any of clauses 1-8 or 17-22, wherein the at least one processor is further configured to: automatically generate a plurality of different simulated pipes in a virtual environment; simulate the sensor arrangement in a virtual environment; control the simulated sensor arrangement in the plurality of different simulated pipes; and record simulated sensor data while controlling the simulated sensor arrangement.

[0047] Clause 33: The method of any of clauses 9-16, further comprising: capturing additional sensor data with at least one of the following: an ultrasonic sensor, a wheeled encoder, a sonde locator, an eddy current sensor, an electromagnetic sensor, a geographic positioning system (GPS) sensor, and / or any combination thereof, wherein the 3D reconstruction is generated based on the additional sensor data.

[0048] Clause 34: The method of any of clauses 9-16, further comprising: simulating the sensor arrangement in a virtual environment; simulating the pipe in the virtual environment; and controlling the simulated sensor arrangement in the simulated pipe.

[0049] Clause 35: The method of any of clauses 9-16, further comprising: automatically generating a plurality of different simulated pipes in a virtual environment; simulating the sensor arrangement in a virtual environment; controlling the simulated sensor arrangement in the plurality of different simulated pipes; and recording simulated sensor data while controlling the simulated sensor arrangement.

[0050] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention.BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Additional advantages and details are explained in greater detail below with reference to the non-limiting, exemplary embodiments that are illustrated in the accompanying figures shown in the separate attachment, in which:

[0052] FIG. 1 is a schematic diagram of a system for pipe monitoring according to nonlimiting embodiments or aspects;

[0053] FIG. 2 is a chart of steps for capturing frames for a system for pipe monitoring according to non-limiting embodiments or aspects;

[0054] FIG. 3 is another schematic diagram of a system for pipe monitoring according to non-limiting embodiments or aspects;

[0055] FIG. 4 is a feature graph used with a system for pipe monitoring according to nonlimiting embodiments or aspects;

[0056] FIG. 5 shows parameters for determining a LIDAR cylinder factor according to non-limiting embodiments; and

[0057] FIG. 6. illustrates example components of a computing device used in connection with non-limiting embodiments.DETAILED DESCRIPTION

[0058] It is to be understood that the embodiments may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes described in the following specification are simply exemplary embodiments or aspects of the disclosure. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting. No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise.

[0059] As used herein, the terms “communication” and “communicate” refer to the receipt or transfer of one or more signals, messages, commands, or other type of data. For one unit (e.g., any device, system, or component thereof) to be in communication with another unit means that the one unit is able to directly or indirectly receive data from and / or transmit data to the other unit. This may refer to a direct or indirect connection that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the data transmitted may be modified, processed, relayed, and / or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives data and does not actively transmit data to the second unit. As another example, a first unit may be in communication with a second unit if an intermediary unit processes data from one unit and transmits processed data to the second unit. It will be appreciated that numerous other arrangements are possible.

[0060] As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may include, for example, a processor such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a microprocessor, a controller, and / or any other computational device capable of executing logic. A “computer readable medium” may refer to one or more memory devices or other non-transitory storage mechanisms capable of storing compiled or non-compiled program instructions for execution by one or more computing devices. Reference to “a processor” or “a computing device” as used herein, may refer to a previously-recited computing device and / or processor that is recited asperforming a previous step or function, a different server and / or processor, and / or a combination of computing devices and / or processors. For example, as used in the specification and the claims, a first computing device and / or a first processor that is recited as performing a first step or function may refer to the same or different computing device and / or a processor recited as performing a second step or function.

[0061] To acquire a more comprehensive digital record of the pipe interior, multi-sensor fusion Simultaneous Localization and Mapping (SLAM) techniques may be used to automatically collect sensory data and perform RGB-D (Red-Green-Blue-Depth) reconstructions in pipes, providing a combination of visual, 3D, and georeferencing information for more sophisticated pipe condition monitoring and assessment. Compared to single-sensor odometry and mapping methods, fusing data from multiple sensor modalities increases the reliability of state estimation and helps reduce ambiguity during the estimation, thereby producing more accurate results. Methods may make use of an inertial measurement unit (IMU) alongside visual or Lidar odometry when there is perceptual degradation or fast motion. These algorithms and the corresponding sensor suites are suited for generic indoor and outdoor environments.

[0062] In confined environments such as pipes, however, the sensing hardware and software employed for such methods experience low localization and mapping accuracy. From a hardware perspective, such methods may be unable to operate inside small pipes due to the limitation of minimal sensing range and bulky sensors. Algorithmically, a slower sensor motion in confined spaces can lead to insufficient IMU excitation and thus an incorrect metric scale or IMU bias initial estimation. The lack of visual and geometric features also poses a significant challenge to state estimation. Further, pipe environments may lack distinct 3D geometric features, such as edges, planes, and corners, compared to daily environments, rendering Lidar odometry methods that rely on these features less desirable. Likewise, the scarcity of 2D visual features in pipes may cause feature tracking failure and thus lead to inconsistent scale estimation during visual-based SLAM techniques.

[0063] Leveraging multi-sensor fusion and laser profiling, single-line structured light and visual-inertial-based approaches may be integrated and applied in short-range and confined space SLAM applications. Although such methods can create a detailed pipe scan while maintaining relatively consistent localization tracking in short distances, a non-negligible drift may occur over longer pipe segments. In environments where global positioning information is unavailable, a SLAM algorithm that only relies on spatially or temporally local measurements is likely to experience amplified odometry drift over time due to theaccumulation of uncorrected dead-reckoning errors. To mitigate this issue, the cylindrical shape of pipes has been used to correct the odometry drift. However, such an approach assumes that the pipe is completely straight, which is an oversimplified assumption.

[0064] Non-limiting embodiments provide systems and methods to map and localize pipeline anomalies and / or other features that adapt to the structural characteristics of pipe environments. Non-limiting embodiments provide for algorithms for enhancing SLAM and optimizing such algorithms for pipe environments. In some non-limiting embodiments, a unique sensor arrangement allows for an accurate 3D reconstruction of the inner surface of a pipe to be generated. Non-limiting embodiments provide for reconstruction of photorealistic 3D pipe scans, disruptive in-pipe locating, georeferencing, and multi-modal condition reporting capabilities.

[0065] Referring now to FIG. 1, a system 1000 for pipe monitoring is shown according to non-limiting embodiments. A sensor arrangement 102 is arranged in the interior of a pipe 101 (e.g., a pipe, a pipeline, and / or the like). In non-limiting embodiments, the sensor arrangement 102 may be arranged on and / or integrated with a robot 105, such as a robot 105 configured to travel through the pipe 101. The system 1000 also includes a computing device 112 in communication with the sensor arrangement 102. The computing device 112 may be arranged in the pipe 101 (e.g., as part of a robot 105 and / or sensor arrangement 102) and be in local communication with the sensor arrangement 102. Additionally or alternatively, the computing device 112 may be external to the pipe 101 and in communication with the sensor arrangement 102 through wired and / or wireless communication. In non-limiting embodiments, there may be a local computing device (not shown in FIG. 1) and a remote computing device 112. There may be various arrangements of computing devices.

[0066] In non-limiting embodiments, the sensor arrangement may be controlled using techniques described in U.S. Patent Application No. 17 / 924,570, filed November 10, 2022, the entirety of which is incorporated herein by reference.

[0067] The computing device 112 may be configured with one or more software applications, as an example, to process data received from the sensor arrangement 102 and to generate a 3D reconstruction of the inner surface of the pipe 101. The sensor arrangement 102 may include a visual sensor 104, an inertial sensor 106, a laser projector 108, and a LIDAR device 110. The sensor arrangement 102 may also include a light-emitting device 107, such as a light-emitting diode (LED). In some non-limiting embodiments, the sensor arrangement 102 may include a housing to support the visual sensor 104, inertial sensor 106, laser projector 108, LIDAR device 110, and / or light-emitting device 107. In some non-limiting embodiments, oneor more of the components shown in the sensor arrangement 102 in FIG. 1 may be separate from the other components, such as the light-emitting device 107 being located at another location on a robot 105. Moreover, in non-limiting embodiments the sensor arrangement 102 may include two or more separate sensor arrangements and / or housings.

[0068] In some non-limiting embodiments, the sensor arrangement 102 may not include the LIDAR device 110. In such embodiments, the depth information derived from the laser projector 108 and visual sensor 104 may be used as the only form of depth information. In some non-limiting embodiments, other types of sensors may be used in the sensor arrangement 102. For example, an ultrasonic sensor may be used to obtain ultrasound data that is combined with other modalities to generate a detailed 3D reconstruction. Additionally or alternatively, the sensor arrangement 102 may include one or more wheeled encoders, sonde locators, eddy current sensors, electromagnetic sensors, multi-spectrum vision sensors, geographic positioning systems (GPS), and / or additional visual sensors, IMUs, laser projectors, and / or the like. It will be appreciated that a combination of different modalities may be used.

[0069] In non-limiting embodiments, the visual sensor 104 may include an RGB CMOS camera with a fisheye lens. However, it will be appreciated that various visual sensors capable of capturing visual data may be used, including one or more RGB cameras or monochrome cameras. The visual sensor 104 may include different color modes for the 3D reconstruction output including, for example, RGB, monochrome, error-aware coded, and / or no color. For example, an RGB camera may provide grayscale, error-aware, and / or colorless 3D reconstructions.

[0070] In non-limiting embodiments, the inertial sensor 106 may include a MEMS-based 6-axis IMU, although it will be appreciated that other inertial sensors and / or a different number of axes may be used. In non-limiting embodiments, the LIDAR device 110 may include a Realsense L515 device, as an example. It will be appreciated that various LIDAR devices may be used. The laser projector 108 may include a laser projector configured to project a laser plane orthogonal to the optical axis of the visual sensor 104 by emitting a thin laser beam toward a conic mirror, such that the laser plane forms a ring 103 on the inner surface of the pipe 101. In non-limiting embodiments, the laser projector 108 may project structured light patterns such one or multiple lines, one or multiple planes, dots and / or dot grids, one or multiple circular shapes, one or more patterns (e.g., complex patterns or the like), and / or the like.

[0071] With continued reference to FIG. 1, in non-limiting embodiments, an alternating- shutter technique may be applied to strobe the laser projector 108 (e.g., projecting a red laser ring 103 as an example) and a light-emitting device 107 at separate time intervals insynchronization with the image shutter trigger of the visual sensor 104. The alternating-shutter technique may involve repeatedly switching between projecting a laser line (e.g., ring 103) on an inner surface of a pipe with the laser projector 108 and emitting light with the light-emitting device 107 onto the inner surface of the pipe 101, thereby alternating between the two, and capturing visual frames under both conditions with the visual sensor 104. The switching may be controlled by the computing device 112 and / or another computing device (e.g., such as a microcontroller that is part of the sensor arrangement 102 and / or robot). In this manner, visual frames Ivcontaining visual details and profiling frames Ipcontaining a bright laser ring 103 may be captured with minimal time gaps between frames. The streaming rate of the visuallaser frame pairs may be 30Hz as an example of a non-limiting embodiment. While the visual frames and profiling frames are captured, the computing device 112 and / or another computing device controls the LIDAR device 110 to obtain depth measurements of the inner surface of the pipe 101 substantially simultaneously with (e.g., concurrently with) the capturing of visual frames by the visual sensor 104. In this manner, one or more point clouds from the LIDAR device 110 may be associated with one or more frames. It will be appreciated that, in some non-limiting embodiments, LIDAR data may be captured non-simultaneously with the capturing of visual frames by the visual sensor 104. In non-limiting embodiments, LIDAR data, profile frames, and / or visual frames may be associated with a timestamp synchronized to a common clock (e.g., to a clock on a processor in communication with the sensor arrangement). The data (e.g., frames) may be stored in associated with such timestamp information. The timestamp information may be used to associate depth measurements with visual features.

[0072] In non-limiting embodiments, a calibration process may be performed before and / or during operation. A visual sensor 104 calibration may be performed, a visual sensor-laser calibration may be performed, a visual sensor-IMU calibration may be performed, and a visual- sensor-LIDAR calibration may be performed.

[0073] In non-limiting embodiments, the laser projector 108 may be configured to project a laser that can be detected even while the light-emitting device illuminates the inner surface of the pipe. In such non-limiting embodiments, an alternating-shutter technique may not be implemented because each frame may include both depth information from the laser projector 108 and feature information from the illumination. Each frame can then be processed to determine depth information (e.g., from the relative location of the projected laser) and feature information (e.g., from the illuminated, colorized image). In such non-limiting embodiments, the light-emitting device may maintain a consistent illumination setting, without alternatingshutter exposure times with the visual sensor. In such embodiments, a post-processing algorithm may be applied to extract both 3D profiles and calculate the true RGB color underneath the visible laser pixels, using motion and 3D mapping information. For example, the laser may oversaturate the color of the scene and the true color of the laser points (e.g., pixels) may be identified. In some examples, historical information may be used to determine the true RGB color underneath the visible laser pixels. In other non-limiting embodiments, rather than using a visible laser, the laser projector 108 may project a laser in an infrared spectrum that can be identified by the visual sensor 104 but does not interfere with the features. In such examples, an infrared (IR) projector may project a structured light pattern and use a single RGB-IR camera to capture both IR and RGB information at the same time. The captured IR information may then be used to compute the 3D profilometry.

[0074] Referring now to FIG. 2, a chart 200 of light pulses, frame captures, laser projections, and image frames is shown according to non-limiting embodiments. The chart shows actions performed over time steps to, ti, t2, ts, and t4. A visual sensor 204 is controlled to modify the exposure times, switching between a first exposure time and a second exposure time shorter than the first exposure time during each successive time step. The chart also shows the pattern of laser projection by a laser 208, pulsing the laser during the second exposure time (e.g., shorter exposure time). This allows for RGB information (from the frames captures while a light-emitting device 202 is on and illuminating) and depth information (from the laser projection) to be measured in a near-concurrent fashion. As shown, visual frames ( / v) and profile frames (Ip) are captured at different, alternating time steps such that the visual frames are captured while the light-emitting device 202 is illuminating the inner surface of the pipe, providing a colorized and illuminated image 212 of features inside the pipe. The profile frames are captured while the laser 208 is projecting a ring on the inner surface of the pipe and while the light-emitting device 202 is off (e.g., not illuminating the inside of the pipe), resulting in a darker image 214 that shows the laser ring.

[0075] In non-limiting embodiments, the exposure time for both types of frames may be the same (e.g., the first and second exposure times may be the same length of time). In such cases, the time period for the illumination, the strength (e.g., intensity) of the illumination, and / or the time period for the projection of the laser may be modified (e.g., modulated and / or altered) to fit within such exposure times. For example, the exposure times may be fixed but under different illumination, such that frames containing only the RGB view are captured under high illumination (by activating a light-emitting device) with the laser disabled, while framescontaining only the laser profiling images are captured under low illumination (by disabling the LEDs) with the laser enabled.

[0076] Referring now to FIG. 3, a system 3000 for monitoring a pipe is shown according to non-limiting embodiments. The system 3000 shown in FIG. 3 is configured to monitor the pipe by localizing the sensor arrangement and / or robot in a pipe and generating a 3D reconstruction of the inside surface of the pipe. A profiling frame lp302 may be input into a laser detection and triangulation module 310. The module 310 may include software, hardware, and / or a combination of software and hardware. For example, the module 310 may be a software function integrated into an application, a stand-alone software application, and / or the like. The laser detection and triangulation module 310 may process the input data independently as the data is streamed (e.g., as multiple profile frames lp302 are streamed while being captured). In non-limiting embodiments, the laser detection and triangulation module 310 detects and triangulates the laser points (e.g., forming a ring around the inner surface of the pipe) through HSV (hue, saturation, value) thresholding in the profiling frames to locate the corresponding 3D laser point for each 2D laser pixel detected in the frame. HSV thresholding may convert an RGB image to the HSV color space and then apply predetermined thresholds to isolate pixels within a range of HSV values. Through this method, the laser can be robustly extracted from the profile frames even when ambient light is captured in the same profiling frames, for example, near pipe entrances and exits. It will be appreciated that the laser points may be detected and triangulated in various other ways.

[0077] A visual frame lv304 may be input into a visual feature tracking module 312. The module 312 may include software, hardware, and / or a combination of software and hardware. For example, the module 312 may be a software functions integrated into an application, a stand-alone software application, and / or the like. The visual feature tracking module 312 may process the input data independently as the data is streamed (e.g., as multiple visual frames lv304 are streamed while being captured).

[0078] LIDAR point cloud data 306 may be input into a LIDAR preprocessing module 314. The module 314 may include software, hardware, and / or a combination of software and hardware. For example, the module 314 may be a software functions integrated into an application, a stand-alone software application, and / or the like. The LIDAR preprocessing module 314 may process the input data independently as the data is streamed (e.g., as LIDAR depth information is streamed while being captured). In non-limiting embodiments, the LIDAR preprocessing module 314 may crop LIDAR point cloud data 306 that is streamed to the module 314. The data may be streamed at 10Hz, as an example, but other speeds are alsopossible. For example, the data may be cropped to a region of interest between 0.4m and 6m from the sensor, spatially downsampled by a factor of 20, and transformed to the body frame (e.g., a spatial coordinate space relative to the sensor arrangement and / or a robot with the sensor arrangement), where the body frame is aligned to the frame of the inertial sensor. It will be appreciated that other dimensions may be used and / or no cropping may be performed in nonlimiting embodiments. If the remaining points after cropping satisfy a threshold, the LIDAR preprocessing module 314 may fit a cylinder shape to the pipe point cloud using, for example, Random Sample Consensus (RANSAC)-based cylinder fitting regardless of whether the captured point cloud is actually cylindrical or not. The output of the LIDAR preprocessing module 314 is a LIDAR data frame that contains the preprocessed LIDAR point cloud / ., the set of points-on-cylinder YL and the corresponding cylinder parameters if the cylinder is found, including the normalized axis vector w, an axis point p, and the radius r.

[0079] Inertial data 308 (e.g., from an IMU) may be input into an inertial preintegration module 316. The module 316 may include software, hardware, and / or a combination of software and hardware. For example, the module 316 may be a software functions integrated into an application, a stand-alone software application, and / or the like. The inertial preintegration module 316 may process the input data independently as the data is streamed (e.g., as IMU information is streamed while being captured).

[0080] Still referring to FIG. 3, the laser point cloud output by the laser detection and triangulation module 310, the output from the visual feature tracking module 312, and a local LIDAR map output by a local LIDAR map registration module 320 may be input into a visualdepth projective data association module 318. The modules 318, 320 may be a back-end component of a simultaneous localization and mapping (SLAM) process. The module 318 may, for example, associate 2D visual features captured by the visual sensor with depth measurements from the triangulated 3D laser points. The modules 318, 320 may include software, hardware, and / or a combination of software and hardware. For example, the modules 318, 320 may be a software functions integrated into an application, a stand-alone software application, and / or the like. The visual-depth projective data association module 318 may associate visual features with the corresponding depth data from laser profiling and LIDAR. The visual-depth projective data association module 318 may denote the set of visual features as F. A feature is defined to be a feature-on-laser if any of its observations is close to the laser pixels in the adjacent profiling frame Ip302. The visual-depth projective data association module 318 may denote the set of features-on-laser as . Similarly, a feature f £ Fi may be a feature-on-LIDAR if any of its observations is close to a point in the local LIDAR mapprojected to visual frame Iv, and the set of features-on-LIDAR is denoted as FL. For a feature fi g Fi U FL, the observation frame in which the feature pixel is the closest to the laser pixel or the projected LIDAR pixel is defined as its primary observation frame c . If fi E Ft U FL, its c is the first observation frame. If a feature fi E Ft U FL, there exists a depth association between the 2D feature and 3D data, with its associated depth at c denoted as dt.

[0081] In non-limiting embodiments, to bootstrap the SLAM process, a vision-only structure from motion (SfM) may performed to obtain the up-to-scale visual sensor poses and feature positions. This process may be followed by a visual-inertial alignment to align metric IMU pre-integration (e.g., output from module 316) with the SfM results. In this manner, a rough gyroscope bias estimation bgand a scale estimation s may be determined. The gyroscope bias may be prone to error due to insufficient rotational excitation, and the scale estimation can be orders of magnitude larger than the true scale in an in-pipe scenario. A fine-tuned scale estimation, s, may be computed by multiplying the rough scale by a factor determined by the associated depth d, of each visual feature f E Ft U FL. according to the following equation:

[0082] In the above equation, | • | denotes the cardinality of a set, di is the estimated depth of each feature from triangulation, a balances the contribution of laser and LIDAR depths to the scale correction and it is set to | FL 1 1 (| Ft | + | FL |) if | FL | > 0 or set to 1 otherwise.

[0083] Using the new scale after the scale correction, the position, velocity, and feature positions determined during visual-inertial alignment may be corrected to conclude the initialization process.

[0084] With continued reference to FIG. 3, a sliding-window factor graph optimization module 324 may receive, as input, the output of module 318 (e.g., associations between visual features and depth measurements), the output of module 314 (e.g., one or more LIDAR data frames of point cloud data), and the output of module 316 (e.g., modified inertial data). The module 324 may be activated to estimate an odometry 328 of the robot and visual feature depths, where the constraints in the cost function are structured with a factor graph. The module 324 may include software, hardware, and / or a combination of software and hardware. For example, the module 324 may be a software function integrated into an application, a standalone software application, and / or the like. The module 324 may be executed after estimator initialization is completed. In non-limiting embodiments, a sliding window-based tightly- coupled monocular Visual -Inertial Odometry (VIO) may be applied. The full state vector,consisting of robot or sensor arrangement states x and landmark (e.g., feature) states z in the sliding window, may be defined by the equation:where n, m are the total number of keyframes and visual features in the sliding window, respectively. , is the inverse feature depth of f in its primary observation frame c . baand bgare the accelerometer and gyro biases,is the pose of the &-th body frame with respect to the world frame. A maximum a posteriori estimation of the states X may be performed by minimizing the weighted sum of four factors in a factor graph, alongside the pose prior obtained from the last marginalization, as shown in FIG. 4 according to non-limiting embodiments. The four factors may include a LIDAR factor (e), a visual reproduction factor (d), an inertial odometry factor (c), and a visual-depth association factor (b) of the sensor arrangement and / or a robot comprising the sensor arrangement.

[0085] It will be appreciated that multiple different factors may be combined in nonlimiting embodiments. For example, one or more factors for each sensing modality and / or sensor type may be used in a factor graph. Each sensing modality and / or sensor type may be associated with zero, one, or multiple factors that may be used in the algorithm. Other factors may include, for example, an encoder wheel factor, an ultrasound factor, and / or the like.

[0086] FIG. 4 shows a factor graph according to non-limiting embodiments. The factor graph may represent various states with corresponding constraints within a sliding window. The visual reprojection factor measures the re-projection error of visual features between keyframes. For every feature£, its estimated 3D coordinates are projected to keyframe Cj in the window and computes the 2D pixel difference with the observation in its primary observation frame c based on the equation:

[0087] In the above equation, TTC(-) projects a 3D point onto the 2D image, and nc1back- projects a pixel onto the normalized image plane.

[0088] The visual-depth association factor, output by module 318, leverages the depth association between visual features and laser and / or LIDAR information to further constrainfeature depth estimation. For a feature f Fi FL, the residual between the associated depth dtfrom projective data association and the estimated feature depth may be calculated with the following equation:

[0089] The IMU residual definition may be followed to estimate vbk, ba, bg, and Tbk. The IMU factor may be used to assist state estimation when low-feature regions are encountered (e.g., regions with a number of features not satisfying a threshold).

[0090] The LIDAR factor, output by module 314, constrains the odometry estimation by aligning LIDAR frames through LIDAR point cloud matching. The alignment is performed between each LIDAR frame Lkin the current sliding window and a reference LIDAR frame Lref, captured prior to the window and updated periodically with a fixed update period Tref, with the pose Tbrefof the body frame known.

[0091] Two candidate LIDAR factors may be defined: 1) the LIDAR cylinder factor for long and straight pipe segments and 2) the LIDAR iterative-closest-point (ICP) factor for geometrically-diverse environments. Depending on the environmental characteristics, the algorithm may automatically select one LIDAR factor from these two candidates. The cylindrical structure regularity RYis defined with the following equation: if |£| > 0 otherwise

[0092] In the above equation, the larger Ryis, the more cylindrical the environment.

[0093] The LIDAR ICP factor may be selected if RY> (0, HY) for both Lkand Lref, which means that the environment is almost perfectly cylindrical. For two sets of points-on-cylinder, the same physical cylindrical environment is captured, all points are equidistant from one single cylinder axis. This cylindrical constraint is mathematically formulated in the following equation:where YkG Lkand Yref G Lref . These points-on-cylinder are assumed to lie on the same physical cylinder given Tref , which is empirically set based on the maximum curvature along the pipe and the speed of the robot. It may be set to 10s, as an example of some non-limiting embodiments. Qi is any point in Yk. ubk, pbkand ubref, pbref are the cylinder axis vectors and axis points in frame bkand bref , respectively. This is shown in FIG. 5, which shows parameters for a LIDAR cylinder factor according to non-limiting embodiments.

[0094] The LIDAR ICP factor is selected6 (0, HY] for both Lkand Lref. First, point-to-point ICP is performed on Lkand Lref to find matching point pairs in the LIDAR point clouds. Then, the residual shown in the following equation may be calculated by transforming the corresponding points into the world frame and computing the distance between each pair of correspondences:

[0095] The ICP analysis may be completed without ambiguity because the threshold check on the cylindrical structure regularity ensures that there exists sufficient geometric features to perform the alignment.

[0096] With continued reference to FIG. 3, a laser point cloud registration and coloring module 322 may be configured to generate a colorized point cloud map 326. The module 322 may include software, hardware, and / or a combination of software and hardware. For example, the module 322 may be a software functions integrated into an application, a stand-alone software application, and / or the like.

[0097] The odometry 328 from the optimization, which is output by module 324 and input into the local LIDAR map registration module 320, may be used to register two maps in the world frame: (1) a dense and colored map of the laser scans (e.g., laser map), and (2) a sparse map of LIDAR point clouds (e.g., LIDAR map). The laser map (1) may be used as the final output of the SLAM algorithm, where the color of each laser scan is estimated using adjacent visual frames. The LIDAR map (2) may be an intermediate product used for visual-depth association (e.g., may be input to module 318 for global depth information). To ensure bounded memory usage, in some examples only the LIDAR frames may be captured if the last Tref are kept in the LIDAR map, and the points may be downsampled using a voxel grid filter or the like. The LIDAR map allows for the visual-depth association to be determined between acurrent visual frame and historical LIDAR data, which may be particularly beneficial for long narrow pipes.

[0098] In non-limiting embodiments, the 3D reconstruction of the pipe may be a 3D model that can be interacted with in a digital environment. In non-limiting embodiments, the 3D reconstruction of the pipe may include an error-aware 3D model, such that the 3D model is visualized with different colors indicating the level of severity of geometric deformation at every point or every region of points in the model. This mode selection key (e.g., colorized key and / or keys associated with other color / non-color modes) may be toggled on or off in a 3D view.

[0099] In non-limiting embodiments, the 3D reconstruction of the pipe may be stored in relation to geodetic and / or geographic coordinates. For example, based on a beginning set of coordinates (e.g., latitude, longitude, and depth / altitude), the location may be tracked during the SLAM process relative to the beginning set of coordinates. In this manner, the pipe can be correlated to any map in the same coordinate system, such as a mapping system that allows for objects to be added based on geodetic and / or geographic coordinates. The 3D nature of the SLAM allows for location to be determined in x, y, and z axes. For example, each waypoint on a 3D map of points may be converted into geodetic coordinates using one reference GPS coordinate at the entrance of the point or starting point of the robot, either underground or above ground. One or more reference GPS coordinates along the path of the pipe may also be used to convert the waypoints in an accurate manner.

[0100] In non-limiting embodiments, one or more graphical user interfaces (GUIs) may be provided to view, add, modify, and / or delete pipe maps and 3D reconstructions, overlay the model on existing maps (e.g., satellite maps, street maps, topographic maps, and / or the like), annotate and display the features and conditions of the pipes, manage one or more databases database for storing and managing pipe data and analysis results, and / or the like. A GUI may provide a custom interface configured to control the operation of the sensor arrangement and the streaming of sensor data from the sensor arrangement, which may be executing on a computing device remote from the computing device viewing the GUI. The GUI may be cross platform and may be deployed on user devices such as smartphones, tablets, personal computers, and / or web-based or cloud-based portal websites.

[0101] In non-limiting embodiments, advanced video footage of the pipe may be created and streamed from the 3D model. For every 2D image frame in the raw video footage, the timestamp when it was captured, accurate coordinates (latitude, longitude, altitude), pipe diameter, and deformation statistics may be stored.

[0102] In non-limiting embodiments, the techniques described herein may be performed in real and simulated environments. For example, a simulated environment may be used to test modalities and geometric constraints. Such simulations may help accelerate development and testing processes by simulating the sensor arrangement and the pipe environment. In some nonlimiting embodiments, a software application may automatically generate simulated pipes with diverse and customizable lengths, shapes, topologies, appearances, and / or realistic textures. The simulation system may allow for a user to control the sensor motion in simulated pipes and convert the recorded, simulated sensor data into formats compatible with the SLAM and 3D reconstruction described herein.

[0103] Before field deployment of the sensor arrangement and / or robot, system verification may be conducted in both lab and open-field setups resembling the realistic in-pipe environment. For example, the system may be verified in the lab using a pipe (e.g., 12-inch- diameter or any other size) painted with realistic patterns that simulate hypothetical field conditions. Further testing may be performed in a relatively straight longer pipe (e.g., 72-foot or other larger size) placed above the ground in an open field. Based on the visual and texture data captured in the physical pipes in mock-up test sites, a simulated pipe environment may be developed (e.g., using a simulator platform such as, for example, Epic Games’ Unreal Engine 4 (UE4)), replicating the conditions of the physical samples. The simulation may include sensor arrangement (e.g., sensor suite) simulation and piping system generation, which may be achieved by combining basic pipe units and procedural random defect (e.g., texture, appearance, and / or the like) creation.

[0104] In non-limiting embodiments, basic pipe segments may be created using a 3D modeling platform (e.g., such as, for example, Blender) and subsequently creating random defects (e.g., texture, appearance, and / or the like) using geometry nodes. Next, procedural controls may handle model selection from multiple different models, positioning of one or more models in a virtual 3D environment, and connection of pipe segments in such an environment. A movable virtual (e.g., simulated) robot is centered in the pipe for data collection. In non-limiting embodiments, the simulated sensor suite and data capturing schemes and / or protocols may also match the real-world counterpart.

[0105] In non-limiting embodiments, such a simulation may be used to verify a robot and / or sensor arrangement and the associated SLAM algorithm (e.g., software process), thereby enabling the testing of pipes with various structures, shapes, and / or inner terrains, such as curved pipes, without the need to rely solely on physical trials. In non-limiting embodiments, to ensure reliability during field deployment, the system may be reinforced againsttransportation and environmental conditions. For example, taking into account a possibly wet and / or muddy deployment site, in non-limiting embodiments the robot modules may be made water-proofed and / or splash-proofed, electronics may be conformal coated, and robot enclosures may be reinforced to minimize wear. Connectors, such as industrial-grade connectors, may be implemented to secure power and signal connections. Additionally, a fully functional backup system may be prepared in some non-limiting embodiments, allowing for rapid replacement of any malfunctioning components and minimizing downtime.

[0106] In non-limiting embodiments, one or more GUIs may be provided to facilitate the remote control of the robot and / or sensor arrangement from a base station computer (a separate machine from the deployment platforms, such as a main computing device) through a wired connection or wireless connection, by receiving control signals from a controller (e.g., such as a gamepad or joystick) connected to the base station. In non-limiting embodiments, the system may also be fully autonomous (without human remote control) for tethered or tetherless operation.

[0107] Experiments conducted with non-limiting embodiments showed that such nonlimiting embodiments were able to generate a meaningful map of an inner surface of a pipe, with dense features and low noise, compared to using other approaches. The output of such non-limiting embodiments provides for denser and smoother maps than other methods, resulting in part from an accurate estimation through visual-depth association with laser and LIDAR. Non-limiting embodiments are able to achieve low odometer drift even over long distances in pipes.

[0108] Existing laser profiling robots provide either 2D visual images or monochrome 3D point clouds. One solution to colorize the point cloud is using Infrared (IR) lasers that do not interfere with the visible spectrum, requiring additional IR sensors. In non-limiting embodiments, a system for colorizing a point cloud of an inner surface of a pipe includes an algorithm, executed by one or more computing devices, that estimates the actual pipe color under the laser via image processing software, allowing the design of compact robots with minimal hardware and maximizing image frame usage. Non-limiting embodiments provide for in-pipe color estimation via video inpainting, using a lightweight machine-learning network fine-tuned with unsupervised learning for efficient in-pipe optical flow estimation. Nonlimiting embodiments also provide for polar coordinate transformation to reduce computation. Non-limiting embodiments also provide for demonstration of real-time effectiveness in natural gas pipes supporting downstream tasks.

[0109] In non-limiting embodiments, a real-time video inpainting algorithm may be based on Deep Flow-Guided Video Inpainting (DFGV) and include several modifications for realtime use. For example, bidirectional flow estimation may be replaced with unidirectional flow estimation and the Image Inpainting network may be removed. As the robot always moves along the pipe axial direction, all the corrupted pixels should be visible in previous frames, so single-frame Image Inpainting may be unnecessary. Next, the multi -iteration refinement is changed to single-pass for inference efficiency (e.g., the algorithm pipeline may be optimized to operate in a single-pass manner).

[0110] The algorithm includes steps for coordinate transformation, laser mask extraction, optical flow estimation, and flow warping. The input to the algorithm is an image pair of the current frame Io and the previous frame Ii. The goal is to estimate the color behind the red laser (or other colored laser), so that the output frame should be an image without the laser. Because the raw image from the sensor may be black on the sides (e.g., due to the fish-eye lens), the input image may be center cropped to 1024 x 1024. Such a high resolution may help preserve the small visual features. These features may be important for some downstream tasks, such as a SLAM-based feature tracker. However, the region of interest (ROI) is around the laser ring projected on the inside surface of the pipe, which only takes a small portion of the whole image. A Cartesian to Polar coordinate transformation (denoted as Cart — > Polar) may be performed to convert the circular ROI to a rectangle for the ease of cropping and reducing the processing region. The coordinate transformation may include a pixel-wise mapping from the {x, y} domain to the {9, p} domain, as shown in the following equations, where (Cx, Cy) is the image center, H and W are the height and width of the image, and R is the maximum radius, which is set to half of the image size (e.g., 512):

[0111] The laser line may be cropped along the horizontal axis, as visualized by [Ii, Io], When cropping, enough context information may be left to allow for optical flow estimation and to handle imperfect pipes where the laser pixels deviate more from a straight line.

[0112] The input to the algorithm may include image pairs from consecutive frames, where the goal is to estimate the color behind the red laser (or other colored laser) in each frame. The raw images from the camera (e.g., such as a fish-eye camera) may be center-cropped, and aCartesian-to-Polar coordinate transformation may performed. The transformation reduces the computational load by focusing processing on the laser line, which occupies a small portion of the image. After masking the laser strip, FastFlowNet and / or other like networks may estimate the optical flow, and the previous frame may be warped to fill in the missing regions. The processed frames may then transformed back into Cartesian coordinates.

[0113] The coordinate transformation brings additional overhead that can be reduced in non-limiting embodiments. Assuming for the purpose of example that the image frames captured by the sensor have the same size, then the {x, y] —> {9, p] mapping is constant. The result may be pre-calculated and cached in memory, and image warping may only be performed at inference time. Moreover, image warping may be accelerated with a GPU. Caching and GPU acceleration reduce the transformation time. In a similar manner, the Polar to Cartesian coordinate transformation (denoted as Polar — > Carf) may be accelerated.

[0114] After the ROI cropping, the image size may be reduced (e.g., reduced to 26% of the input in an example). Next, a color filter may be performed to extract the laser mask, which indicates the region to be inpainted. The masked images Ii, Io and the masks Mi, Mo may then be stacked together and input into a FastFlowNet and / or other like network to obtain the optical flow Fi, which represents the pixel movement from Ii to Io. The previous inpainted frame Oi may be warped with Fi to fill the masked-out region and get the inpainted output Oo. For the very first frame, Oo = Io may be set. The pixel propagation equation is shown in the following equation, where Mo is the laser mask that contains a collection of laser pixel indices and f = Fo (x, y) is the vector of the pixel movement at the given location.

[0115] The ROI may be replaced in the original Polar-coordinate image with Oi and it may be converted back to the Cartesian coordinate to get the output Oo without the laser. The result may be saved for processing the next input image lo+i.

[0116] The optical flow estimator may affect flow-based Video Inpainting algorithms. For example, if the estimated optical flow is inaccurate, the pixel propagation stage will fail and significantly influence the image reconstruction quality. State-of-the-art optical flow networks are reported to have very low Endpoint Errors (EPEs); however, most of these pre-trained networks perform poorly for in-pipe images because of the domain shift. Public large-scale optical flow datasets, such as KITTI, Sintel, and Flying Chairs, are all designed for natural images. Trained on these datasets, machine-learning networks learn to track the movement ofcars, humans, or other objects. On the contrary, features in pipelines may include rust, dust, or merely the texture of cast iron or other material. These small features are outside the training data distribution and cause the pre-trained networks to predict inaccurate results, so fine-tuning is necessary. Unlike a synthetic dataset, there are no in-pipe optical flow labels. To address this problem, unsupervised learning techniques may be used to fine-tune the optical flow network on our unlabeled in-pipe dataset collected by the robot.

[0117] Since the laser strips may be masked out in the input images, the output optical flow is incomplete in the laser regions. Rather than using a cascading network to predict the complete optical flow, in non-limiting embodiments the complete optical flow may be predicted directly from the optical flow estimation network for better efficiency. In nonlimiting embodiments, the optical flow may be estimated by a modified version of FastFlowNet, although it will be appreciated that other networks may be used.

[0118] To support real-time applications, a lightweight optical flow estimation network may be used. FastFlowNet balances well between accuracy and efficiency, and has three major components. The (1) Head Enhanced Pooling Pyramid (HEPP) and the (2) Center Dense Dilated Correlation (CDDC) module heavily compress the spatial size and the number of channels when encoding the feature pyramid. The (3) Shuffle Block Decoder (SBD) utilizes ShuffleNet blocks to reduce the computation when decoding the optical flow. With an objective to directly predict the complete optical flow, more information needs to be provided to the network. The mask Mi, which indicates what region is masked out, together with the image Io, may be stacked to form a 4-channel input tensor, e.g., mask channel and RGB channels. The input for the first convolutional layer may therefore be increased from 3 -channel to 4-channel. When loading the pre-trained network, the weights for the third channel are copied to the new channel. This helps the network converge faster than random initialization during training.

[0119] FastFlowNet uses a shared weight feature extractor for the two input images to reduce the number of parameters. However, due to input frames being sequential, calculating the feature pyramid for the same image twice is redundant. More specifically, at time to, the input to the flow estimator is {[Mo, Io], [Mi, Ii] } . At time t+i, the input becomes {[Mi, Ii], [M+i, I+i]{, where the frame [Mi, Ii] is processed twice. To remove the redundancy, FastFlowNet may be modified in non-limiting embodiments to output both the estimated optical flow Fi and the latent feature pyramid for [Mi, Ii], which is cached and used as part of the input at the next timestamp.

[0120] In non-limiting embodiments, an unsupervised loss function may be utilized for both improving the network performance on an unlabeled dataset as well as directly predictingthe complete optical flow. The photometric loss is defined in the following equation, where 0 is the set of learnable parameters of the estimator f, Ii is the first image warped with the network’s output. Ii and L are the masked image inputs:

[0121] L is the second image, which is used as the supervision, p is the function of pixelwise structural similarity (SSIM). The summation may be calculated for each pixel p. This loss function is used to optimize the optical flow such that the warped first image matches the second image. The inputs are corrupted images, but instead of using the same image pair as supervision, the photometric loss may be calculated with the complete image pair, such that the network learns to predict the complete flow directly.

[0122] The smooth regulation term is defined in the following equation, where F is the estimated optical flow:

[0123] This is to eliminate the ambiguity of texture-less regions, which is common in inpipe environments. The richness of textures is measured by the image gradient. For regions with small gradients in the image, small gradients in the optical flow domain may help produce a smooth flow output.

[0124] The overall loss function may be defined by the following equation, where = 50 is a hyper-parameter:

[0125] In non-limiting embodiments, data augmentation may be used to help increase the size of the training set and prevent over-fitting. Two types of data augmentation techniques may used for training. The first one, speed augmentation, trains the network to handle different robot moving speeds. To achieve this, instead of giving two consecutive frames {L, L+i] in the dataset as the input, some frames may be skipped and {L, Ii+n} may be used to simulate the robot moving faster. A value of n may be randomly chosen in the range of [1, 3] during the training phase. The second data augmentation technique may be a random laser mask generation. Certain defects may change the shape of the pipe, but non-limiting embodiments can continue to operate without a straight pope. For example, laser masks may be synthesizedwith imperfection using B-Splines. Four knots at the top, right, bottom, and left may be first created to form a perfect circle, and then random perturbations may be added to make the circle imperfect. It will be appreciated that various other methods may be used to generate the laser masks. During the evaluation phase, real laser masks may be used that are extracted from the images.

[0126] In non-limiting embodiments, the real-time video inpainting algorithm may be plugged into existing systems that provide a video stream. The real-time video inpainting algorithm may be integrated into existing systems and / or algorithms, such as the SLAM algorithm described herein. By eliminating the need to alternate between laser-on and laser-off frames, non-limiting embodiments streamline the input pre-processing pipeline. This enables the robot to capture both depth and color information simultaneously without altering other SLAM components.

[0127] Components in the SLAM algorithm described herein may intercommunicate using the Robot Operating System (ROS) message-passing interface or other like systems. In operation, the robot sensor driver may output three topics, where two image topics / profile and / visual are used to obtain depth and color information and perform feature tracking. The inertial topic / imu and the tracked features may be sent into the backend for online optimization. With such an approach, the laser projector no longer needs to be repeatedly turned off, and the / visual topic from the sensor can be removed. The added inpainting module takes the / profile topic as the input, inpaints the laser area, and outputs the equivalent / visual topic. Since the inpainting module is decoupled from the SLAM algorithm and only used for input pre-processing, the other parts of the SLAM system may remain unchanged.

[0128] Referring now to FIG. 6, shown is a diagram of example components of a computing device 900 for implementing and performing the systems and methods described herein according to non-limiting embodiments. For example, the computing device 900 may correspond to the computing device 112 shown in FIG. 1. In some non-limiting embodiments, device 900 may include additional components, fewer components, different components, or differently arranged components than those shown in FIG. 6. Device 900 may include a bus 902, a processor 904, memory 906, a storage component 908, an input component 910, an output component 912, and a communication interface 914. Bus 902 may include a component that permits communication among the components of device 900. In some non-limiting embodiments, processor 904 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 904 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit(APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 906 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 904.

[0129] With continued reference to FIG. 6, storage component 908 may store information and / or software related to the operation and use of device 900. For example, storage component 908 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid state disk, etc.) and / or another type of computer-readable medium. Input component 910 may include a component that permits device 900 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally, or alternatively, input component 910 may include a sensor for sensing information (e.g., a photo-sensor, a camera, a thermal sensor, an electromagnetic field sensor, a global positioning system (GPS) component, a laser projector, an accelerometer, a gyroscope, an actuator, etc.). Output component 912 may include a component that provides output information from device 900 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 914 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 900 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 914 may permit device 900 to receive information from another device and / or provide information to another device. For example, communication interface 914 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0130] Device 900 may perform one or more processes described herein. Device 900 may perform these processes based on processor 904 executing software instructions stored by a computer-readable medium, such as memory 906 and / or storage component 908. A computer- readable medium may include any non-transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical storage devices. Software instructions may be read into memory 906 and / or storage component 908 from another computer-readable medium or from another device via communication interface 914. When executed, software instructions stored in memory 906 and / or storage component 908 may cause processor 904 to perform one or more processesdescribed herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software. The term “programmed or configured,” as used herein, refers to an arrangement of software, hardware circuitry, or any combination thereof on one or more devices.

[0131] Although embodiments have been described in detail for the purpose of illustration, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed embodiments, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment can be combined with one or more features of any other embodiment.

Claims

WHAT IS CLAIMED IS1. A system comprising: a sensor arrangement comprising a visual sensor, an inertial sensor, and a laser projector; at least one processor configured to: project a laser on an inner surface of a pipe with the laser projector; emit light with a light-emitting device on the inner surface of the pipe; capture a plurality of frames of the inner surface of the pipe with the visual sensor, wherein at least a subset of frames of the plurality of frames is captured while the laser is projected on the inner surface, and wherein the at least a subset of frames or a different subset of frames is captured while the light illuminates the inner surface; and generate a 3D reconstruction of the inner surface of the pipe based on the plurality of frames.

2. The system of claim 1, wherein the at least one processor is further configured to: repeatedly switch between projecting the laser on the inner surface and emitting the light on the inner surface, wherein capturing the plurality of frames comprises capturing a plurality of profiling frames while the laser is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface, wherein the at least a subset of frames comprises the plurality of profiling frames, and wherein the different subset of frames comprises the plurality of visual frames.

3. The system of claim 1, wherein the sensor arrangement further comprises a LIDAR device, and wherein the at least one processor is further configured to: capture a plurality of depth measurements of the inner surface of the pipe with the LIDAR device, wherein the 3D reconstruction of the inner surface is generated based on the plurality of depth measurements.

4. The system of claim 1, wherein the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm.

5. The system of claim 4, wherein the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector.

6. The system of claim 4, wherein the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visual-depth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement.

7. The system of claim 6, wherein the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization.

8. The system of claim 3, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

9. A method comprising: projecting a laser on an inner surface of a pipe with a laser projector; emitting light with a light-emitting device on the inner surface of the pipe; capturing a plurality of frames of the inner surface of the pipe with a visual sensor, wherein at least a subset of frames of the plurality of frames is captured while the laser is projected on the inner surface, and wherein the at least a subset of frames or a different subset of frames is captured while the light illuminates the inner surface; and generating a 3D reconstruction of the inner surface of the pipe based on the plurality of frames.

10. The method of claim 9, further comprising: repeatedly switching between projecting the laser on the inner surface and emitting the light on the inner surface, wherein capturing the plurality of frames comprises capturing a plurality of profiling frames while the laser is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface, wherein the at least a subset of frames comprises the plurality of profiling frames, and wherein the different subset of frames comprises the plurality of visual frames.

11. The method of claim 9, further comprising: capturing a plurality of depth measurements of the inner surface of the pipe with a LIDAR device, wherein the 3D reconstruction of the inner surface is generated based on the plurality of depth measurements.

12. The method of claim 9, wherein the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm.

13. The method of claim 12, wherein the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector.

14. The method of claim 12, wherein the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of a sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visual-depth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement.

15. The method of claim 14, wherein the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization.

16. The method of claim 11, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

17. A system comprising: a sensor arrangement comprising a visual sensor, an inertial sensor, a LIDAR device, and a laser projector; and at least one processor configured to: repeatedly switch between projecting a laser on an inner surface of a pipe with the laser projector and emitting light with a light-emitting device onto the inner surface of the pipe; capture a plurality of profiling frames while the laser is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface; capture a plurality of depth measurements of the inner surface of the pipe with the LIDAR device; and generate a 3D reconstruction of the inner surface of the pipe based on the plurality of profiling frames, the plurality of visual frames, and the plurality of depth measurements.

18. The system of claim 17, wherein the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm.

19. The system of claim 18, wherein the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector.

20. The system of claim 18, wherein the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surfaceof the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visual-depth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement.

21. The system of claim 20, wherein the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization.

22. The system of claim 17, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

23. A method comprising: repeatedly switching between projecting a laser on an inner surface of a pipe with a laser projector and emitting light with a light-emitting device onto the inner surface of the pipe; capturing a plurality of profiling frames while the laser is projected on the inner surface and a plurality of visual frames while the light is emitted onto the inner surface; capturing a plurality of depth measurements of the inner surface of the pipe with a LIDAR device; and generating a 3D reconstruction of the inner surface of the pipe based on the plurality of profiling frames, the plurality of visual frames, and the plurality of depth measurements.

24. The method of claim 23, wherein the 3D reconstruction is generated based at least partially on a simultaneous localization and mapping algorithm.

25. The method of claim 24, wherein the simultaneous localization and mapping algorithm is configured to output a dense and colored map of laser scans from the laser projector.

26. The method of claim 24, wherein the simultaneous localization and mapping algorithm is configured to: determine, with a factor graph, (i) an estimation of each state of a plurality of states of a sensor arrangement and / or a robot comprising the sensor arrangement and (ii) an estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe, based on minimizing a weighted sum of a LIDAR factor, a visual reproduction factor, an inertial odometry factor, and a visual-depth association factor of the sensor arrangement and / or a robot comprising the sensor arrangement.

27. The method of claim 26, wherein the estimation of each state of the plurality of states of the sensor arrangement and / or a robot comprising the sensor arrangement and the estimation of each state of a plurality of states of at least one visual feature on the inner surface of the pipe is determined based on a sliding-window-based nonlinear optimization.

28. The method of claim 23, wherein generating the 3D reconstruction of the inner surface of the pipe comprises: generating a first map from laser scans from the laser projector; and generating a LIDAR point cloud map.

29. A computer program product comprising a non-transitory computer- readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to perform any of the methods of claims 9-16 or 23-28.

30. The system of any of claims 1-8 or 17-22, wherein the sensor arrangement further comprises at least one of the following: an ultrasonic sensor, a wheeled encoder, a sonde locator, an eddy current sensor, an electromagnetic sensor, a geographic positioning system (GPS) sensor, and / or any combination thereof.

31. The system of any of claims 1-8 or 17-22, wherein the at least one processor is further configured to: simulate the sensor arrangement in a virtual environment; simulate the pipe in the virtual environment; and control the simulated sensor arrangement in the simulated pipe.

32. The system of any of claims 1-8 or 17-22, wherein the at least one processor is further configured to: automatically generate a plurality of different simulated pipes in a virtual environment; simulate the sensor arrangement in a virtual environment; control the simulated sensor arrangement in the plurality of different simulated pipes; and record simulated sensor data while controlling the simulated sensor arrangement.

33. The method of any of claims 9-16, further comprising: capturing additional sensor data with at least one of the following: an ultrasonic sensor, a wheeled encoder, a sonde locator, an eddy current sensor, an electromagnetic sensor, a geographic positioning system (GPS) sensor, and / or any combination thereof, wherein the 3D reconstruction is generated based on the additional sensor data.

34. The method of any of claims 9-16, further comprising: simulating the sensor arrangement in a virtual environment; simulating the pipe in the virtual environment; and controlling the simulated sensor arrangement in the simulated pipe.

35. The method of any of claims 9-16, further comprising: automatically generating a plurality of different simulated pipes in a virtual environment; simulating the sensor arrangement in a virtual environment; controlling the simulated sensor arrangement in the plurality of different simulated pipes; and recording simulated sensor data while controlling the simulated sensor arrangement.

Images