System and method for automatic adjustment of robot reference frame

Abstract

A method for controlling movement of a robot in an environment, where the robot has a plurality of sensors associated therewith, includes: determining an initial scanning reference frame of the robot, where the scanning reference frame is calculated based on an initial origin; calculating a new origin based on readings from at least one of the sensors; and adjusting the initial scanning reference frame to determine a new scanning reference frame based on the new origin, wherein the adjusting step is based on a difference between the initial scanning reference frame and the new scanning reference frame.

Description

[0001] Priority statement and related applications

[0002] This application claims priority to provisional application No. 65 / 571,263, filed March 28, 2024, the entire contents of which are incorporated herein by reference, and is a continuation-in-part of U.S. Patent Application No. 18 / 492,662, filed October 23, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to systems and methods for performing processes using robotic tools in enclosed or hazardous spaces. Background Technology

[0004] To automate processes more efficiently, especially in environments where operators cannot or should not be present, new systems and methods need to be developed. One such environment is in underwater and sewer piping infrastructures. During pipeline inspection, accurate identification of all pipeline characteristics and defects is crucial. Conventional techniques rely on human intervention to detect features and defects, resulting in error-prone and inefficient processes. Highly accurate automated inspection methods are necessary to address this need.

[0005] Some existing mapping systems utilize visual cameras that rely on manual observation to detect features within a scanned environment. Other systems use infrared cameras to detect temperature changes within the scanned environment. However, no system exists that can correlate these disjoint datasets.

[0006] However, the inability to create accurate digital maps is only one problem. A second problem is the inability to perform operations accurately and efficiently within the environment, such as cutting water pipes or sewer pipes from within a conduit.

[0007] Concepts associated with pipeline inspection and operational processes such as cutting are applicable across multiple industries and domains. For example, pipeline inspection and remote operational processes are crucial for water pipelines, sewer lines, gas pipelines, and oil pipelines. Similarly, the same or similar concepts will be used to detect features and defects in other enclosed spaces, or simply to map inspections of other enclosed spaces, including locations that may contain hazardous chemicals or are too small for human inspection. Furthermore, accurate representation and operation in other environments will also benefit from the same or similar concepts, even those that are not enclosed spaces. Therefore, there is a need for highly accurate automated inspection systems and methods that can be adopted across multiple domains in multiple industries to address the need for accurate inspections, and systems and methods for performing operational processes in such domains once such inspections have been completed. Summary of the Invention

[0008] An overview of this disclosure is provided at the end of this specification to introduce, in a simplified form, the selection of concepts further described below in the detailed description. This content is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to addressing any or all of the shortcomings mentioned in any part of this disclosure. Attached Figure Description

[0009] Reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, wherein the same reference numerals denote the same elements.

[0010] Figure 1 This is an exemplary illustration of an operating environment for the systems and methods disclosed herein.

[0011] Figure 2 This is an exemplary schematic diagram of a robot having a processor and multiple sensors configured to practice the systems and methods disclosed herein.

[0012] Figure 3 This is an exemplary schematic diagram of a walking vehicle tethered to a robot with an onboard processor.

[0013] Figure 4 This is an exemplary hardware configuration constructed in accordance with this disclosure.

[0014] Figure 5 This is an exemplary flowchart of the control for scanning and / or cutting operations according to this disclosure.

[0015] Figure 6 This is an exemplary flowchart illustrating the process of constructing a 3D model of a scanned environment using a multi-sensor configuration.

[0016] Figure 7a This is an exemplary flowchart of an algorithm that can be deployed to practice the methods disclosed herein.

[0017] Figure 7b It is an exploded external perspective view of a piping system with lateral pipes extending from it.

[0018] Figure 7c This is an exemplary schematic diagram showing the distance between lateral pipes in a pipe and the orientation of such lateral pipes extending from the pipe.

[0019] Figure 8 This is an exemplary flowchart of the processing in an RGB recognizer.

[0020] Figure 9 This is an exemplary flowchart of the processing in an IR identifier.

[0021] Figure 10This is an exemplary flowchart for processing in a point cloud recognizer.

[0022] Figure 11 This is an exemplary flowchart for processing in an integrated predictor.

[0023] Figure 12a This is an exemplary flowchart illustrating the process of scanning the pipe environment before and after inserting the liner into the pipe.

[0024] Figure 12b This is an exemplary flowchart of an algorithm that can be deployed to practice the methods disclosed herein.

[0025] Figure 13 This is an exemplary block diagram of an articulated robot with multiple sensors mounted throughout the robot.

[0026] Figure 14 It highlights the reference frame. Figure 13 An exemplary block diagram of a robot.

[0027] Figure 15 It shows a tool for a robot that rotates at a certain angle. Figure 9 A flowchart of 'a'.

[0028] Figure 16 This is an exemplary flowchart of the cutting process.

[0029] Figure 17 and Figure 18 Includes an exemplary flowchart illustrating an operator-assisted augmented reality cutting process.

[0030] Figure 19 This is an example photo displayed by the augmented reality operator.

[0031] Figure 20 This is an example displayed by the operator.

[0032] Figure 21 This is an example flowchart illustrating the anomalies that may be encountered during the cutting operation and the corrective actions to be taken. Detailed Implementation

[0033] System Overview. This disclosure relates to systems and methods for mapping features of a three-dimensional environment for controlling the maneuvering of robots within an environment using robotics and artificial intelligence. Applications of this disclosure may include, but are not limited to, mapping in enclosed or remote spaces, toxic areas, water and gas infrastructure, and other applications.

[0034] This disclosure includes combinations of multiple sensors, including but not limited to cameras, infrared sensors (IR), light detection and ranging (LIDAR), motion sensors, and other sensors integrated into the robotic system, as well as the fusion of data from those sensors in a sensor fusion system to create a consistent representation of a three-dimensional (“3D”) environment. Heuristics and statistical data analytics are used to train artificial intelligence (“AI”) models to predict and label the location of features to create a digital twin of the environment. The systems and methods of this disclosure enable real-time detection, labeling, and localization of features through environmental scanning, and the scan can be used to enhance control over the robot's operations while it performs other functions.

[0035] The system includes hardware and software. While the description herein will focus on exemplary hardware and software functionality, it should be understood that some hardware functionality may be provided in software, and some software functionality may be implemented in hardware. It should also be understood that while some functionality will be described as discrete components of the overall system, the scope of this disclosure includes components that can be integratedly connected to other components.

[0036] The hardware components include multiple sensors. These may include vision and infrared cameras, spatial distance sensors, which can be, for example, LiDAR components, ultrasound, stereo vision, or other methods for creating 3D point clouds. The vision camera can be configured to provide two-dimensional and red / green / blue (RGB) color output and will be referred to herein as an RGB camera. Additionally, an inertial measurement unit (IMU) or other suitable sensing device capable of capturing environmental and motion data can be included as one or more of the multiple sensors. The IMU can be, for example, a multi-axis IMU capable of measuring vibrations in certain configurations known in the art. Combinations of infrared cameras, RGB cameras, motors, and encoders can also be utilized. Such sensors provide measurements such as distance, angle, 3D point clouds, images, and features to facilitate the robot's spatial perception and detection. The sensors can be strategically placed on the robot for deployment within the environment.

[0037] The software components include a variety of algorithms and computational techniques for processing sensor data and extracting meaningful information, as described in more detail in this paper. These algorithms employ advanced techniques such as object detection, feature extraction, data association, odometry estimation, probabilistic modeling, and optimization methods.

[0038] While this system has multiple applications across various sectors, this disclosure uses non-limiting embodiments in which a robot-controlled cutting tool is employed to restore the operational characteristics of the pipeline after a relining process. For example, an in-situ cured pipe (CIPP) process may already exist, in which a liner is installed and cured within the original main pipe. After lining, the liner must be cut to re-establish fluid communication with existing services, making the piping system operational again. A service is a pipe or conduit extending from the main pipe or trunk line that allows fluid to flow into or out of the main pipe. These may be referred to in this specification, in various ways, as “branch pipes,” “branch conduits,” and “laterals.” While this disclosure will be described in relation to the cutting tool, this is merely exemplary. Other tools, including drills, probes, grinders, etc., can be robot-controlled for application on a variety of materials in various applications and operating environments, including but not limited to wood, stone, metal, and plastic.

[0039] In one aspect, this disclosure includes a robot that scans and digitally maps the main pipe prior to the relining process, rescans the main pipe after the relining process, and then uses analysis based on multimodal sensor readings to control a cutting tool to access a branch pipe associated with the project. This disclosure also includes using a digital map during the cutting process to locate the robot within the pipeline and to pinpoint the branch pipe to be cut, and then executing an automated cutting path to reach the branch pipe point through the robot.

[0040] During the scanning operation or in post-scan processing, the detection of all observations will be labeled within a digital map of the pipeline. This will include a marker and unique enumeration for each observation, described within the scanning reference frame as discussed in more detail herein.

[0041] This disclosure includes various features, including but not limited to autonomous cutting of surfaces, autonomous navigation of robots to avoid obstacles, dynamic adjustment of cutting tools based on the material being cut, and the use of immersive technologies such as virtual reality (VR) and / or augmented reality (AR) to provide a robust real-time user interface to allow operators to control the cutting as needed.

[0042] The systems and methods disclosed herein improve upon existing techniques by generating a smooth feed rate for a cutting tool advancing against the material to be cut (e.g., a liner inside a pipe) and optimal profile and safety around the edge of the branch pipe. Based on additional training of the process using feedback from use, different materials that may react differently to the cutting process can be identified and adapted. Depending on the type of material (known beforehand), the thickness of the material (known beforehand), and the curing parameters (known beforehand), the relative hardness of the material (or the material's resistance to cutting), and how the material will react to the cutting process can be predicted.

[0043] In one embodiment, the material hardness can be predicted in advance based on the corresponding properties of the main pipe and liner materials. For example, the main pipe can be made of brick, clay, concrete, asbestos, metal, or any other type of material used for transporting water or wastewater. Each of these materials can be represented in a drop-down menu or fed as input into a rule-based algorithm or lookup table to determine the initial hardness of the main material.

[0044] Similarly, the liner material and curing method can be selected from the predetermined list set forth above. The liner material can include, for example, carbon fiber reinforced polymer (CFRP); UV-cured reinforced glass fiber, polyester polyurethane, or any other suitable liner material.

[0045] Environmental conditions, including temperature, pressure, and humidity, can also be determined and used to calculate predicted material hardness. The thickness of the pipe and liner materials can also be a variable to consider when predicting material hardness.

[0046] Where the type of the main material or liner material, or its associated properties, is not pre-programmed into the system setup, the characteristics of the main material or liner material can be input into an AI / ML learning algorithm to determine the initial hardness of the main material. Similarly, hardness measurements calculated from the cutting process can be used to update tables or machine learning algorithms to make advance predictions of material hardness more accurate.

[0047] Predicting material hardness allows for adjustment of the feed rate, also referred to as the cutting speed in this paper, to prevent the cutting tool from overheating and to maximize the likelihood of a smooth cut. If the material is predicted to be softer than expected, the cut can be made further from the edge, and a finishing brush can be applied to provide a smooth edge.

[0048] Furthermore, different operating environments can impose different constraints on the desired cutting results. In one case, a single large fragment / debris / sample—referred to herein as the material being removed—may be advantageous. In such a case, the cutting path will be defined as an optimized path around the maximum safe outer path, such that only a single outer profile of the entire contour exists. In another case, the removed material can be ground into small pieces. In such a case, the cutting path can be an outward spiral or zigzag path. This allows the cutting head to overlap with the desired amount of material to reduce the removed material to fine dust. The opening will gradually become larger because there will be no single large cutting path.

[0049] Therefore, given different parameters that are predetermined or calculated for the liner, the cutting path can be adjusted in real time. Such parameters can be fed into an algorithm within the algorithm to translate the parameters into the mechanical motion of the cutting tool.

[0050] These algorithms employ advanced techniques such as object detection, feature extraction, data association, odometry estimation, probabilistic modeling, and optimization methods. Algorithm schemes include interpolation methods, A... Algorithms / Dyskstra algorithms, optimization control, and other machine learning algorithms.

[0051] It should be understood that this disclosure may use cutting operations within a piping environment as an example in this detailed description, but the systems and methods disclosed herein are applicable across a wider range of applications.

[0052] On one hand, the cutting tool can be a drill with one or more cutting heads, each of different sizes and shapes and selected based on its intended use. On the other hand, other tools can include rotary discs, saws, milling cutters, or any other mechanical cutting tool that can be adapted to the cutting environment and the material to be cut. Several sensors, such as motor RPM sensors and voltage and current sensors, can be present in the cutting tool. Other sensors described herein can be associated with the robot. Communication between the two can be via an API or a direct interface. Furthermore, the functionality between the cutting tool and the robot as defined herein is exemplary, and some sensors from the cutting tool can be associated with the robot, and vice versa. Additionally, the robot and the cutting tool can be separate and connected to each other via mechanical and electrical connections, or they can be integrated and manufactured as a single unit.

[0053] Operating environment. Figure 1This is an exemplary view of one of several different operating environments disclosed herein. In this non-limiting example, operating environment 10 is a closed pipe 5 with an interior 6. A robot 2, attached to a tether 3, is disposed within the interior 6. As discussed in more detail below, robot 2 includes multiple sensors whose datasets are fused together by controller 0 in a sensor fusion system. Unless otherwise explicitly stated herein, the controller, processor, and sensor fusion system are interchangeable. Through a series of iterative processes, robot system 10 combines one or more sensor measurements and estimates the presence and location / orientation of features within the scanned environment, and records data processed by an onboard processor to map the internal conditions of pipe 5 in real-time or near real-time, and then transmits those conditions to a traveling vehicle 7 connected to the other end of the tether 3. For example, robot 2 and traveling vehicle 7 can communicate across tether 3 using an Ethernet protocol. Traveling vehicle 7 can be connected to a server (not shown) or a cloud computing platform for further computation and / or storage.

[0054] The tether 3 can span the length of the environment, with an additional length, to cross from one side to the other. In one aspect, the environment could be a fifty-foot pipe, and the tether could be 100 feet long. Although the implementation will be described with respect to tether 3, it will be understood that the onboard processor can communicate with an external processor via near-field communication such as Wi-Fi, Bluetooth®, or other types of wireless communication including local area networks and / or wide area networks. The tether 3 may also include an encoder configured to measure the distance traveled by the robot 2 across the environment 10.

[0055] In one embodiment, robot 2 may have multiple sensors, including but not limited to RGB color sensors, infrared sensors, LiDAR sensors, motion sensors, and other sensors. For example, there may be one or more vision cameras for collecting static or moving video data, one or more infrared cameras for collecting infrared data, one or more inertial measurement unit (“IMU”) sensors for collecting attitude and position data, one or more motor encoders for sensing and collecting position data, one or more LiDAR sensors or cameras for generating 3D point cloud data, and other motion and / or dedicated sensors.

[0056] refer to Figure 2 An exemplary robotic system 2 exists, containing multiple sensors. Although Figure 2 An exemplary system is shown as having multiple discrete sensors that are collectively shown as being integrated into the robot 2; however, it should be understood that the sensors according to this disclosure may include onboard sensors, attached sensors, or remote sensors. The sensors may be fixed to an arm extending from the body of the robot system 2.

[0057] exist Figure 2 Among the exemplary sensors shown are a LIDAR sensor 24, an IMU sensor 26, a wheel sensor 25, a motor encoder sensor 23, a camera (both infrared and vision) 22, and a dedicated sensor 21. Each of these will be described in more detail below.

[0058] Sensor interface 27 is also shown, which can be communicatively coupled to each sensor in sensor 20, for example, directly or indirectly. Sensor interface 27 can receive raw sensor data from each sensor in sensor 20 and store the raw sensor data in database 13. Sensor interface 27 can be connected to sensor fusion system 29, which may include, for example, one or more software programs operating on processor 12. Sensor fusion system 29 can analyze sensor data from one or more sensors 20 and / or control the operation of one or more sensors 20. Sensor interface 27 can also convert the raw sensor data received from each sensor in sensor 20 into a format for further processing by sensor fusion system 29. Sensor fusion system 29 may include an AI / ML engine. In one example, raw infrared data from an infrared scanner of sensor 20 can be compared with data from an initial scan of the main pipe prior to lining. Sensor fusion system 29 fuses (broadly, “compares”) this infrared sensor data and the initial scan data to predict the location and shape of branch openings in the main pipe covered by the liner. Sensor interface 27 may also include a bidirectional communication path to sensor 20 to provide control and commands from sensor fusion system 29 to sensor 20. For example, the sensor fusion system running on processor 12 may process data received from vision camera 22 via sensor interface 27 and then issue commands to vision camera 22 to change its focus or orientation to create additional camera input for processing. This disclosure envisions other command and control functions from the processor to the sensor.

[0059] Sensor 20 can be calibrated periodically. Factory calibration can occur during manufacturing or be recalibrated to factory settings upon deployment. Alternatively, field calibration can be performed. For example, processor 12 can use known historical data from database 13 to recalibrate one or more sensors 20 in the field. Thus, for example, if there is a discrepancy between the readings of motor encoder sensor 23 and IMU sensor 26 due to slippage, any one or both of those sensors can be recalibrated to reconcile the readings. It should be noted that recalibration can occur periodically via a preset schedule and / or timer, or non-periodically via command.

[0060] Figure 2The diagram also shows a communication interface 28. Communication interface 28 can be a wired or wireless communication interface that provides bidirectional communication to an external computer network or server (not shown). For wired communication, communication interface 28 may include interfaces for... Figure 1 The logic for communication via tether 3. On one hand, communication interface 28 can use the Ethernet protocol to communicate with... Figure 1 The vehicle 7 communicates. For wireless communication, the communication interface 28 may include one or more wireless communication functions and protocols, including cellular, which may be 5G cellular, wide area network protocols, local area network protocols, near field communication functions and protocols, Wi-Fi and / or Bluetooth functions.

[0061] Robot 2 may include a combination of processor 12 and memory 11 that can operate in series, wherein memory 11 stores instructions that, when executed by processor 12, perform the functions described herein. Processor 12 may include an AI / ML engine. Such functions may include command and control of sensors 20. Such functions may also include real-time or near real-time processing of sensor data collected by various sensors 20. In one aspect, one or more central processing units (CPUs) may be present. Alternatively, a combination of one or more CPUs and one or more graphics processing units (GPUs) may be present. For example, while it is desirable to perform most of the computation on the robot to reduce latency, some computations may not be as fast and can be offloaded to the walking vehicle 7 or the cloud. This would improve the efficiency of the onboard processor. Processor 12 may be, for example, an Nvidia® Jetson Xavier NX processor. However, this disclosure and the associated claims should not be limited to any particular configuration of the processor.

[0062] Onboard processing of sensor data can provide low latency for sensor data processing. For real-time or near-real-time processing, processor 12 may include a direct interface to sensor interface 27. For other functions, processor 12 may include a direct interface to database 13 and perform operations on stored data. Such stored data may include, for example, historical sensor data used for calibration or continued training of artificial intelligence / machine learning algorithms.

[0063] In an embodiment, the sensor fusion system 29 can perform an iterative process to combine sensor data from multiple types of sensors 20 to form an estimate of the robot 2's position and orientation within the scanned environment, and can store this estimate in a database 13. This includes, for example... Figure 1Mapping the environment inside pipe 5 6 and detecting objects in the environment can be performed by processor 12 using sensor data collected from multiple sensors 20. The map may include objects, branches, features, landmarks and / or other relevant information, which may vary based on the specific application of the technology. As robot 2 moves through the environment, various sensor readings can be aggregated to construct a virtualized 3D representation of the scanned location to perform mapping, feature identification and other functionalities.

[0064] In this embodiment, robot 2 can be connected via tether 3. Figure 1 7. (Reference) Figure 3 The traveling vehicle 7 can be located in Figure 1 The external portion or opening of pipe 5. The traveling vehicle 7 may be stationary relative to robot 2 and pipe 5. The traveling vehicle 7 may include a communication interface 9, which enables communication with robot 2 via communication interface 28. In one aspect, the protocol between communication interface 9 and communication interface 28 may be Ethernet or any other suitable wired and / or wireless communication protocol. Another protocol may include Controller Area Network (CAN). It should be understood that, depending on the application, communication may be wireless, wired, or both. Communication interface 9 may also include communication functionality to and from a cloud computing platform (not shown) and to / from a display and / or other input devices such as a keyboard, touchscreen, or voice processor (not shown).

[0065] Figure 3 The document also illustrates a combination of processor 4 and memory 8 that can operate in series, wherein memory 8 stores instructions that, when executed by processor 4, implement the functions described herein. Processor 4 may include an AI / ML engine. Such functions may include processing sensor data provided by robot 2. Processor 4 may be, for example, an Nvidia® processor such as Jetson Xavier NX. Both processor 12 and processor 4 may operate in series and / or in parallel to process sensor data. As a design choice, processor 12 may be assigned processing of certain functions, while processor 4 may be assigned processing of other functions. Alternatively or additionally, each processor may be assigned similar processing, or all processing may be performed in either processor 12 or processor 4. This disclosure and the associated claims should not be limited to any particular configuration of the processors or the functional division between processors. Similarly, each processor may consist of one or more GPUs and / or one or more CPUs.

[0066] refer to Figure 4An exemplary system configuration of this disclosure is shown. Four components are shown: a robotic device 34 configured to traverse an environment, which may be, for example, the interior of a pipe or other enclosed area; a vehicle housing 32 communicating with the robotic device 34; a control computer configuration 31, which may be located, for example, on-site, or in some configurations, remotely; and a cloud storage and processing facility 30 having a portal 35. A robot 42 is shown relative to the robotic device 34, which may include, for example, […]. Figure 1 and Figure 2 The diagram illustrates some or all of the functionality of robot 2. Robot 42 can be connected to robot attachment 45 and communicate with robot attachment 45 via Ethernet connection and / or Controller Area Network (CAN) bus or any other communication interface. In one aspect, Power Line Communication (PLC) protocol can be used for communication between robot 42 and the traveling vehicle 43. Robot attachment 45 can be physically attached to robot 42 or integrated into robot 42. In some configurations, robot attachment 45 may not be physically attached to robot 42, but may be remotely and communicatively controlled by robot 42. Robot attachment 45 can be, for example, a cutting tool as described in more detail herein. It should be understood that, in addition to supporting the operator display station, computer configuration 31 may also provide additional or alternative AI / ML engine functionality.

[0067] Robot 42 can communicate with vehicle 43, which may include some or all of the functionalities of vehicle 7 described above. Vehicle 43 can communicate with robot 42 using a physical bus such as a CAN interface. Vehicle housing 32 may also include a switch 44 connected to the robot via Ethernet. Switch 44 and vehicle 43 can communicate via Ethernet connection. In embodiments, vehicle housing 32 may be located outside and / or adjacent to the environment of interest, such as outside a pipe or other enclosed area operated by robot 42.

[0068] The control computer configuration 31 can be, for example, a personal computer having a processing computer 47, a display 48, and a custom control board 49. It communicates with components including the vehicle housing 32. The computer 47 and display 48 configurations are standard and known in the art. The custom control board 49, although it may use other communication and connection protocols, can be attached to the computer 47 via a USB connection.

[0069] The mobile vehicle 43 can communicate with the control board 49 via a CAN interface. The mobile vehicle 43 can also communicate with the display 48. The switch 44 can communicate with the computer 47 via Ethernet and may include a communication interface to a wireless network 46 such as a 5G and / or LTE interface. Figure 4As shown, the Ethernet connection can be continuous and / or switched from robot station 45 back to computer 47, and includes robot 42, switch 44 and trolley 43.

[0070] It should be noted that the above-listed items, and as such Figure 4 The communication and connection protocols shown are merely exemplary, and other protocols and connections may be used.

[0071] Scanning operation. (See reference) Figure 5 A high-level exemplary flowchart 50 is shown for a process of scanning and / or cutting pipes within an environment. The process begins at 51, and a decision to scan or cut is made at 52. If the process is scanning, scanning occurs at 53, where raw and processed sensor data are stored in database 55. The scanning process continues at 56, where scanned mapping data is stored in database 55. At 57, a determination is made as to whether the job is complete. If the process is complete, for example, if only the scanning operation was performed, the process flow concludes at 58.

[0072] If the job is not completed at point 57, the process loops back to point 52, where it is determined whether the scan should continue along the scan path at point 53 or whether a cutting operation should be performed at point 54. If a cutting operation is to be performed, the map data from the previous scan stored in database 55 is transferred to the cutting process at point 54, and the raw and processed cutting data are transferred to and stored in database 55. The process then continues at point 57 to determine whether the job is complete and ends at point 58, or whether the loop continues at point 52.

[0073] refer to Figure 6 The diagram illustrates an exemplary process 600 for creating a 3D image during a scanning operation. At 601, data from multiple sensors is captured. For example, the data from multiple sensors may include IMU sensor data, LiDAR sensor data, IR and visual RGB sensor data, motor encoder data, pressure data, odor data, gas data, air pressure data, humidity data, and ultrasonic data. At 602, the sensor data is received by a processor. At 603, the position of the robot performing the scan is tracked. At 604, an AI algorithm uses some or all of the sensor data to perform feature identification. At 605, the features are correlated with the robot's position. At 606, the 2D features are mapped to a 3D representation of the features. At 607, the various sensor inputs may be weighted. At 608, an output is provided to predict the features and their locations. At 609, this process may be repeated to create multiple layers that ultimately comprise a 3D model at 610.

[0074] Using multiple sensors operating in the aforementioned environment, one or more robots can perform scanning operations within the environment to create a digital image of it while moving through it. If the environment is enclosed, such as the interior of a pipe, the scanning operation can be performed by a robot traversing inside the pipe. During the scanning operation, or in post-processing of the scan, all detected observations will be marked on a map of the pipe. This will include a marker and unique enumeration for each observation, described within the scanning reference frame as described below.

[0075] Operation Method. Figure 7 illustrates an exemplary flowchart of using a deep learning-based feature recognizer to process sensor data from multiple sensors 20. By acquiring and using training data across multiple sensors, various sensor data can be embedded to generate a feature vector representing multiple data related to the detected features. In embodiments, RBG sensor data, point cloud data, IR and vision camera data, IMU data, and application domain-specific sensors can be embedded in a single space to generate a feature vector representing pipeline (or other environmental) features. Using the feature embeddings collected from sensor 20 and historical training data, a deep neural network model can be created to automatically map and detect pipeline features in real time.

[0076] Figure 7 illustrates multiple exemplary sensor outputs 31, including an RGB video feed 32, an IR video feed 33, point cloud data 34, and other sensor data outputs 35, which may include, for example, any of the aforementioned sensors or domain-specific sensor outputs. Each of the sensor outputs 31 may be processed by one or more corresponding recognizers 40, including an RGB recognizer 41 associated with the RGB video feed 32, an IR recognizer 42 associated with the IR video feed 33, a point cloud recognizer 43 associated with the point cloud data 34, and other domain-specific recognizers 44 associated with the other sensor data 35. The recognizers 40 are described in more detail below.

[0077] Continuing with the description of the flowchart in Figure 7, the outputs of the RGB recognizer 41 and the IR recognizer 42 are projected into 3D spatial representations 46 and 47, respectively. The outputs of the 3D spatial projections, as well as the outputs of the point cloud recognizer 43 and other domain-specific recognizers 44, include feature detection and associated 3D coordinates of those features. The features and their corresponding coordinates are then sent to the ensemble predictor 50, which combines and weights the features and coordinates to produce a vector prediction result 51. The ensemble predictor is described in more detail below.

[0078] It should be understood that the flowchart in Figure 7 is merely exemplary and simplified for clarity. Figure 7 is not intended to limit this disclosure to such data processing flows. For example, sensor data is not limited to... Figure 3The data shown can be input from multiple sensors into one or more identifiers. Similarly, the identification of the identifiers in FIG7 is non-limiting, and identifiers deployed according to this disclosure may include some of all the identifiers shown in FIG7, as well as additional identifiers not shown in FIG7. The disclosure and scope of the claims should not be limited to what is shown with respect to FIG7.

[0079] RGB Recognizer. The RGB Recognizer 41 is functionally designed to receive 2D RGB sensor data from a video camera feed and generate bounding boxes and 2D coordinates of features and defects within the operating environment. In an embodiment, a deep learning model that uses a pipeline to inspect video frames can be used to detect features and defects based on the RGB video feed. The deep learning model can be trained using historical data from the same or similar environments. According to this disclosure, the performance of the deep learning module and the reception rate of the RGB video feed enable the algorithm to generate object detection results in real time at thirty (30) frames per second. Figure 8 The image shows an example of the output of the RGB recognizer 41, i.e., the 3D spatial projection 46.

[0080] RGB Recognizer 41 Functionality Figure 8 The diagram is shown in more detail below. The RGB recognizer 41 can detect features, for example, using the YOLO algorithm or a convolutional neural network (CNN). In one aspect, YOLO v. 8 can be used. For example, the RGB feed 32 can be input to the detection function 42. The detection function 42 can detect various features of the RGB data received from the RGB feed 32. For example, it can analyze the image pixels or their corresponding representations received from the RGB feed 32 to determine if a feature is located in a given pixel. Analysis may include comparing a pixel with neighboring pixels, identifying the pixel's color count (e.g., R value, G value, B value), determining the pixel's brightness value, etc. The result of the detection function 42 can then be passed to the classification function 43. The classification function 43 can, for example, determine one of five categories in which the detected feature can be found. In some cases, the classification function 43 can map the determined category to one of three subcategories.

[0081] Image segmentation 44 can be based, for example, on color or contrast. In a deep learning model, neural networks can be used to extract features using an encoding algorithm, then those features corresponding to the input feed 41 can be decoded, and long-range neural network connections can be made to generate scale and improve accuracy. Convolutional layers can be used to encode each digit feature, and each digit feature can, for example, capture edges to generate a feature vector. While CNNs can be used, other types of deep learning neural networks can also be utilized. In some cases, image segmentation 44 can occur in parallel with the operations of detection function 42 and classification function 43.

[0082] In one aspect, the identifier 41 can determine the appropriate Pipeline Assessment and Certification Procedure (“PACP”) code developed by Nassco. Such PACP codes are known in the industry and, in addition to other characteristics within the piping system, can indicate a broken, cracked, deformed, or collapsed section of the pipe.

[0083] Infrared Recognizer. The infrared recognizer is functionally designed to receive 2D IR sensor data and generate bounding boxes and 2D coordinates of features and defects within the operating environment. Features and defects within the environment can be detected by a deep learning model using IR video feeds from one or more cameras 22. The deep learning model can be trained using historical IR video images from the same or similar environments. An example of the output of the IR recognizer 42 is shown in Figure 7 as a 3D spatial projection 47.

[0084] Infrared reader 42 can be based on and Figure 9The RGB recognizer 41 shown operates on the same principle and according to the same exemplary process. The IR recognizer 42 can, for example, utilize the YOLO algorithm to detect features. YOLO v. 8 can be used. For example, the IR feed 33 can be input to the detection function 56, which can detect various features contained in the IR data. The classification function 57 can, for example, determine one of five categories for classifying the detected features. The classification function can also determine a subcategory of the detected features (e.g., one of three subcategories). After identifying the subcategory, the process can continue at the segmentation function 58. For example, in this case, image segmentation 58 can be based on a temperature gradient, which can then be color-coded. Alternatively or additionally, machine learning can be used to set the temperature range from which the most useful data can be collected. For example, the temperature data can be segmented into ranges representing the coldest and hottest expected temperature readings. This segmentation is useful for identifying the boundaries between hot and cold regions, and it is useful for detecting the location of lateral openings into the main pipe. For example, the IR recognizer can be controlled to ignore infrared data from the IR feed 33 that falls outside the high-end and / or low-end temperature ranges. The high-end and low-end temperature ranges can be calculated by the controller or set by a human operator. Factors to consider when setting the temperature range may include known thermal information about the main duct environment, curing temperature information, etc. Furthermore, it should be understood that a temperature “range” can be defined at both ends, or it may have only a single boundary, such as a temperature greater than or less than a certain set value. In a deep learning model, a convolutional neural network can be used to extract features using an encoding algorithm, then decode those features corresponding to the input feed 41, and perform long-range neural network connections to generate scale and improve accuracy. Convolutional layers can be used to encode each numerical feature, which can, for example, capture edges to generate feature vectors. While CNNs can be used, other types of deep learning neural networks can also be utilized. In some cases, image segmentation 58 can occur in parallel with the operation of detection function 56 and classification function 57.

[0085] Infrared sensor 42 can detect features that are not present inside the pipe but may be obscured behind a barrier. Furthermore, the infrared sensor can detect features that the RGB sensor 41 cannot. For example, water collected behind the pipe can be detected by the infrared sensor 42 instead of the RGB sensor 41.

[0086] Alternatively or additionally, machine learning can be used to define the temperature range from which the most useful data can be collected. For example, temperature data can be divided into ranges representing the expected temperature readings for the coldest and hottest areas. This segmentation is useful for identifying the boundaries between hot and cold regions, and for detecting the locations of lateral openings.

[0087] 3D spatial projection. Before the outputs from the infrared recognizer 42 and the RGB recognizer 41 are input to the integrated predictor 50, the 2D recognizer outputs are mapped and projected into 3D space. The projected 3D location can be a function of the camera position, camera field of view, pipe diameter, and frame coordinates.

[0088] Point cloud recognizer. As those skilled in the art will understand, a point cloud is a discrete collection of data points in 3D space. A point cloud can be generated by a 3D scanner or by software from multiple points on the outer surface of a measured object. In embodiments, the point cloud can be generated by any spatial distance sensor. In embodiments, a LiDAR sensor or camera can be used. However, other spatial distance sensors can include ultrasound, stereo vision, or other methods for generating point clouds. The point cloud recognizer 43 can include an artificial intelligence algorithm, such as a deep neural network, to transform the point cloud data into 3D bounding boxes and 3D coordinates of features and defects in the surrounding environment.

[0089] In this embodiment, point clouds can represent segments of the pipeline. Predictions of pipeline features and defects can be generated using heuristic-based or deep learning-based methods. In deep learning-based methods, historical point cloud data collected by sensors 20 in the real pipeline can be used as training data.

[0090] On the one hand, the point cloud recognizer 43 can use the PointNet architecture or its variants, such as PointNet++. (Reference) Figure 10 The diagram shows the LiDAR feed 51 passed to the point cloud recognizer 43. At 52, high-level classification is performed. At 53, part segmentation is performed. At 54, bounding box estimation is applied. At 55, semantic segmentation is performed. The result is a 3D model providing feature detection and recognition.

[0091] The point cloud recognizer 43 will operate to detect and classify specific aspects of the environment. For example, the point cloud recognizer 43 can detect lateral pipes in a pipeline environment where one pipe connects to another, which could include, for example, an exhaust pipe from a house to a mainline exhaust pipe. Figure 7b An example configuration is shown. When two pipes intersect, the circumferential measurements from the LiDAR data may change. Assuming the main discharge pipe has a cylindrical shape, typically with protrusions arranged radially outward from the cylinder, the point cloud recognizer will utilize a combination of statistical analysis and deep learning models to detect this change, defining the segmentation boundary at the profile of the lateral pipes by listing a series of 3D coordinates.

[0092] An example of 3D coordinate projection relative to the detection of the lateral pipe is shown in Figure 7. This schematic defines the distance and relative location of the lateral pipe. As the robot 42 drives through the pipe, the 3D point LiDAR sensor scan will generate, for example... Figure 7b The data shown. Then, the point cloud recognizer 43 will calculate the 3D coordinates describing the contour of the lateral tube.

[0093] An integrated predictor. Combinations of various recognizer outputs can be performed in the integrated predictor 50, where a linear rule-based model combines the predictions from each of the aforementioned recognizers. The integrated predictor 50 can combine sensor data and recognizer outputs to cross-reference readings to ensure or increase accuracy. Additional cross-referencing between 2D pixels and 3D coordinates can also be performed. For example, 2D data from an RGB sensor can be converted to 3D for comparison with LiDAR 3D data. Simultaneously, 3D LiDAR data is converted to 2D for comparison with 2D RGB sensor data. These comparisons can be combined to reach a final conclusion about the specific feature to be detected.

[0094] Weights can be assigned to each of the different recognizer outputs to derive results with predicted confidence levels. Rules can be adopted, tested, and updated using real-world test data accumulated from various sensors 20. The results of processing high-quality multimodal data collected from sensors within the pipeline can be optimized.

[0095] Ensemble predictors can use heuristic weighting algorithms. For example, and refer to... Figure 11 The diagram illustrates an integrated predictor 50 with outputs projected onto 3D spatial coordinates from various recognizers. Inputs may include one or more of 3D RGB image detection 61, 3D IR image detection 62, point cloud feature detection 63, and / or 3D domain-specific feature detection 64. Each input may be weighted by weights 65, 66, 67, and 68, respectively. The algorithm then operates on the inputs adjusted by the corresponding weights to derive predictive features 69 with their associated probabilities.

[0096] As an example, different sensors 20 can record data from the current location at different times based on each sensor's individual viewpoint and field of view. Each sensor knows its pose within its own scanning reference frame, i.e., its position and orientation. The sensor fusion system 27 knows the reference frame of each of the multiple sensors with respect to the other sensors and its relative reference frame with respect to the scanning reference frame. In operation, one sensor may collect data relative to a specific location, while other sensors may not collect any data or only collect minimal data relative to that specific location. When this occurs, the weights of each sensor in the system can be dynamically adjusted as the specific location is mapped.

[0097] According to another example, weights can be used and dynamically adjusted based on confidence levels. The exact level of confidence will vary depending on the specific detection algorithm used. However, most deep learning methods and statistically based methods will have associated confidence scores related to the predicted output. In an embodiment, the weights used by the integrated predictor 50 can be dynamically adjusted based on the confidence levels derived for each detector in the detectors. In some cases, the weights can be directly equal to the confidence level or based on relative confidence levels among the various detectors. Additionally, in alternative embodiments, the weights can also be based on historical data, including previous readings, prior information, or manually entered information, or other parameters that may be unique to the combination of the operating environment and the sensors used.

[0098] Pipeline-specific recognizers. According to this disclosure, additional domain-specific recognizers may be included. For example, in a pipeline scenario, a lateral recognizer can be used to detect and map lateral features of the pipeline system. In the case of a lateral recognizer, a deep learning-based visual detection model can be used to operate on detected point cloud outliers outside a defined region, such as an ellipse. For example, a lateral recognizer focuses on the fact that a robot is located in a pipeline. The most likely geometry to be found is circular geometry. Even lateral openings are circular or nearly circular. Therefore, the recognizer can specifically look for differences from the expected circular shape to identify features.

[0099] Additionally or alternatively, a depression detector can be deployed. The depression detector can use LiDAR sensor data such as water intensity and image segmentation models.

[0100] Additionally or alternatively, a joint offset recognizer can be deployed. In the case of a joint offset recognizer, the sensor can detect diameter changes based on point cloud diameter measurements. In this case, a deep learning-based algorithm can be used based on a visual detection model. Alternatively or additionally, the IMU sensor can detect bumps in the pipeline that represent joints within the pipeline.

[0101] Additionally or alternatively, a root identifier may be deployed. In the case of a root identifier, a deep learning-based algorithm can be used on visual sensor data and / or point cloud data from camera 22 to map the locations of roots that may have compromised the integrity of the pipeline.

[0102] Referring to Figure 12, an exemplary process 700 for post-liner insertion scanning is shown, which can be used to confirm that the liner is correctly installed in the pipe, enabling the positioning of the branch pipe and the performance of the cutting operation. At 702, according to the above description relative to... Figure 6 The described exemplary process performs a pre-lining scan. At 703, the liner is installed according to a process known to those skilled in the art. At 704, a common reference frame is established between the pre-lining scan and the current scan operation. At 705, a post-lining scan is performed. A series of checks can then be performed. For example, at 706, it can be determined whether there are differences between the pre-lining scan and the post-lining scan. The post-lining scan can detect that the liner is not installed correctly or is damaged. For example, the liner may have collapsed. Groundwater may have accumulated between the pipe and the liner. Other anomalies associated with the liner installation may have been detected, including dents at various locations. One difference that may be detected is a reduction in pipe diameter based on the thickness of the installed liner. In one embodiment, data on standard pipe size, liner thickness, and curing type can be stored in a lookup table for operator selection. Knowing the liner thickness and the corresponding expected reduction in diameter can also be used as a check for the quality of the liner installation. If no anomalies are detected at 706, the liner installation is successful at 710. If discrepancies exist, it is determined at 707 whether these discrepancies were expected. If not, an anomaly may be reported at 711. If expected discrepancies exist at 707, it is determined at 708 whether any action is required. For example, expected discrepancies may exist because the branch pipe is not open but is blocked by the liner. Expected discrepancies could be water in the pipe or temperature differences detected by an IR scan due to the installation of the liner. If no unexpected discrepancies exist, the liner installation is successful and reported at 710. If expected discrepancies exist that can be corrected, corrective actions are performed at 709 to complete the successful installation process at 710.

[0103] In post-liner scanning, the mapping calculations may need to be adjusted to account for the liner thickness. For example, if the liner is 4.5 mm thick, the robot will be lifted by that amount, and the inner diameter of the tube will be reduced by half that amount. Therefore, a new reference frame can be calculated for post-liner scanning to account for the different dimensions. The IMU can then touch the inner wall to recalibrate the reference frame to a new starting point (0,0,0) and reorient the robot within the environment.

[0104] Furthermore, the distance with the reduced diameter will also change and needs to be recalibrated. The new distance can be represented as follows: D Final =D Initial + Lining correction factor.

[0105] The correction factor will reduce any errors in navigating the robot to the liner location. Inertial sensors can be used to recalculate the distance traveled. Cable encoders can also be used as a coarse measurement to provide a second data point relative to the distance traveled. A table mapping rotations to distance can be constructed.

[0106] Referring to 12b, an exemplary flowchart of post-lining scan 720 is shown. At 721, a new reference frame is calculated based at least on the reduced diameter of the pipe with the liner installed. At 722, a liner correction factor is calculated for comparing the pre-lining scan and the post-lining scan. At 723, LiDAR is used to export a point cloud image. At 724, a visual inspection using a 2D, RGB camera is performed. For example, the visual inspection may show bulges in the liner at the location of the lateral pipe. At 725, an IR thermal image is created, which may, for example, be mapped to 3D space. At 726, a multi-sensor digital model of the interior of the liner is built. At 727, the pre-lining scan and the post-lining scan are correlated using at least the liner correction factor. At 728, a confidence score is calculated, which is a measure of the confidence in the accuracy of the post-lining scan, i.e., whether the branch pipe is correctly aligned. In some instances where a sufficiently high level of confidence is not achieved, the system will automatically default to allowing a human operator to locate and re-establish the connection with the lateral opening. It should be noted that the order of the steps described above is merely exemplary, and some steps may be performed in an alternative order. For example, in some embodiments, an IR image may be created prior to visual inspection.

[0107] Using a deep learning module, confidence scores can be calculated and increased over time by retraining and updating the model. For example, based on a comparison of pre-lining and post-lining images, the deep learning module can look for correlations between the two and may weight the pre-lining scan higher than the post-lining scan. IR images can be weighted more than point cloud images because temperature gradients identified in IR images are more indicative of branch pipes.

[0108] Robotic systems. Figure 13 This is an example robot system 800 based on this disclosure. Figure 8 The robot system 800 can be Figure 1 or Figure 2 The example of the robot shown may include references. Figure 3 The ability to handle discussions.

[0109] The robot system may include different sensors at different locations on the body 71 and joints of the robot system. Each sensor can perform a different measurement, and the different measurement values ​​from the various sensors can be processed to provide useful information. Example information may include the distance moved by the robot, the absolute position in a given environment, the relative position between positions in the environment, the orientation of the observation within the environment, the diameter or other size of the environment (e.g., a pipeline environment), or combinations thereof. The collection, processing, and generation of measurement values ​​will be described in more detail below.

[0110] Robot system 800 may include various hardware and associated sensors for collecting signals and measurements for further processing and analysis. For example, robot system 800 may include a body 71, an arm 72, a camera 22, a LiDAR camera 24, a tool 73, and a set of wheels 74-a and 74-b. Joints may be defined between corresponding adjacent components of robot system 800. For example, joint 75-a may be defined between body 71 and arm 72; joint 75-b may be defined between arm 72 and camera 22; and joint 75-c may be defined between camera 22 and tool 73. Each corresponding joint may include a motor (not shown) that may be configured to actuate components of robot system 800 relative to adjacent components defining the joint. For example, a motor may be located at joint 75-a, which may actuate arm 72 relative to body 71. A motor may be located at joint 75-b, which may actuate an arm or portion including camera 22 relative to arm 72. A motor may be positioned at joint 75-c, which can actuate a portion of the arm including tool 73 and / or a portion of the arm including LIDAR camera 24 relative to a portion of the arm including camera 22. In some cases, actuation may be rotation or pivoting, with the corresponding adjacent component acting as the pivot point for actuation.

[0111] The robot system 800 may include one or more motor encoders 23. The encoders can measure the rotation of the motor spindles of the motors used for joints 75 and record changes in rotation via electrical signals. For example, encoder 23-a may record measurements of the movement of the motors mounted in joint 75-a, encoder 23-b may record measurements of the movement of the motors mounted in joint 75-b, and encoder 23-c may record measurements of the movement of the motors mounted in joint 75-c. These recordings can result in a count of electrical pulses associated with relative rotation. Each motor encoder pair may include a different gear ratio to determine the absolute rotation angle based on the encoder counts. These configurations may be based on mechanical properties, calibration and testing results, and any runtime feedback provided in the system.

[0112] In some cases, encoders may be located at or near each wheel to record the rotation of the corresponding wheel. For example, robot system 800 may include a pair of wheels 66-a and 66-b. Each wheel 74-a and 74-b may include associated encoders 23-d and 23-e. The encoders corresponding to the wheels can record linear movement, such as forward or backward. As an example, if it is determined that one wheel is moving and the other wheel is not moving, the encoder measurements can determine whether the robot is turning. In some cases, joints corresponding to the robot head, robot arms, robot cameras, and / or robot tools may each include a corresponding encoder. The encoders can measure the rotation of the corresponding joint.

[0113] The robot system may also include multiple IMUs 26 located at various sites along the system. For example, body 71 may include IMU 26-a, camera 22 may include IMU 26-b, and tool 73 may include IMU 26-c. IMUs may be housed within or on the robot, and may provide acceleration and / or orientation readings. These readings may be available via a CAN bus. In some cases, measurements acquired from the IMUs may be used to generate and update the reference frame of the robot system 800, as will be described in more detail below. For example, measurements collected by the IMU associated with body 71, IMU 26-a, may form the origin of a scan reference frame. Measurements collected by the IMU associated with camera 22, IMU 26-b, may be the origin of a sensor reference frame. The system may use measurements collected by the IMU associated with tool 73, IMU 26-c, to identify or determine the relative orientation of the tool with respect to the camera and body.

[0114] Feedback and control of the robotic system will operate on the robot during real-time operation. As the robot moves within the environment, sensor data, particularly LiDAR sensor data, will be processed. This allows the robot's motion to be calculated, and any corrective actions can be performed in real time.

[0115] To move between two positions, a robotic system can utilize inverse kinematics to transform 3D points (XYZ) into joint movements. Joints are defined as prismatic (linear) or rotary (rotational). Prismatic joints result in linear motion along a certain axis of movement. Motion on multiple axes is handled as multiple connected prismatic joints. Rotational joints result in motion about a rotational axis. There are no practical restrictions on the connection of joints. Motion calculations are used to control the movement and sequence of joint operations. Joints are interconnected to allow the movement of one joint to affect the relative position and orientation of downstream joints.

[0116] Inverse kinematics calculations will be based on trajectory (path) planning operations. The path planning operation is optimized based on system calibration to produce a smooth and continuous motion from the current position to the target position. The planned path will take into account the differences in relative velocities of the joints, as well as understanding the limitations of motion for any joint. For example, some joints may be able to move freely throughout their entire range of motion. This would include rotary joints that can rotate a full 360 degrees without mechanical stopping. This could also include prismatic joints without motion limitations, such as wheeled devices capable of rolling without boundaries in either direction. Other joints may have limitations or motions based on the physical construction of the robot system. Rotation of rotary joints can be limited because rotation results in one joint arm engaging with another.

[0117] There may be other restrictions on joint movement, which are not based on the robot's physical construction but on other parameters. To prevent undesirable positions, there may be rotary joints that prohibit rotation beyond their limits. This could be to eliminate the risk of gimbal lock-up, to prevent tipping, or to always provide the operator with a helpful visualization.

[0118] Another reason for limiting joint movement is to prevent collisions within the environment. One of the main collisions to avoid is accidentally cutting pipes where they are not intended to be cut. The map from the initial scan will have the locations of all branches to be cut, as well as a complete digital twin representation of the pipes. This will allow the robotic system to know the 3D coordinates in advance to limit the range of motion of each joint's operating window. This means that, in the case of an 8-inch diameter, the cutting head cannot extend beyond that diameter unless an active attempt is made to cut into the branch.

[0119] Knowing all constraints, the robotic system iterates through motion to allow the robot to move smoothly through its environment. This motion also gathers real-time feedback from sensors within the robotic system to detect and react to unexpected conditions. The robot's operation can be improved by weighting sensor readings. For example, when a joint attachment of the robot moves (e.g., for cutting a liner), its motion is tracked by encoders at both the joint and the IMU. The readings are compared, but the readings from the encoder are given greater weight than the IMU readings, especially when both readings are consistent within tolerances. However, if the readings diverge beyond the tolerance, the IMU reading is given greater weight. This is because divergence in readings is most likely to indicate conditions that caused the encoder to malfunction (e.g., the attachment has engaged an obstacle). Similarly, the robot's tether can be accurately read to determine the distance the robot has traveled along a pipe, and thus its position within the pipe. Again, the IMU can be used to establish position. When the IMU reading and the tether distance are consistent within tolerances, the tether distance reading is given greater weight. However, if the readings diverge from each other outside the tolerance, the IMU reading is given greater weight. Furthermore, if a large divergence occurs, it is more likely that a situation has been encountered that compromises the accuracy of tether distance measurement, rather than an IMU-related issue.

[0120] Machine learning models can be used to apply inverse kinematics calculations to control robot movement and to adjust path, posture, speed, or other parameters using real-time feedback from sensors. For example, given the end-effector location of a robot attachment (e.g., a cutting tool), neural network algorithms or anomaly detection algorithms can be used to calculate and coordinate the movement of various joints and wheels associated with the robot to deliver the attachment to a desired location, which could be, for example, a cutting tool moved to a position for lateral cutting. For example, when the robot has reached a location marked as containing a lateral opening in a previous scan, a scan for the lateral opening can begin. Segmentation of sensor readings occurs and bounding boxes are digitally drawn. Confidence levels are calculated. If the confidence level exceeds a threshold, the location of the lateral opening is confirmed and the cutting operation can begin. If the confidence level is not high enough, further scans can be performed to approach confirmation of the lateral opening location through an iterative process.

[0121] Previous scans may have been used to create a digital twin of the environment. Such previous scans may have been performed both before and after the liner was inserted into the pipe. As illustrated above, such previous scans may have been created using a machine learning model. Therefore, the model may have been trained with training data in which joint movements are correlated with various locations and poses of the robot within the environment.

[0122] LiDAR sensor data is not static because the robot may move during the scanning operation. A LiDAR reference frame can be calculated based on the robot's kinematics, which will be tracked as the LiDAR's pose. Updates to the pose can be calculated continuously or updated over specific time periods to support real-time processing of the scanning and navigation processes. Specific constraints can vary based on the current operation of the environment and can be dynamic to minimize any possible errors associated with delays during movement.

[0123] Reference Frames. To accurately scan and locate the branch pipe area as part of the scanning operation, a reference frame within the internal pipe needs to be determined. That is, there can be a global reference frame, a scanning reference frame, a robot reference frame, and a sensor reference frame. Each of the aforementioned reference frames considers the robot's pose within the environment, which may include, for example, x, y, z coordinates and orientation within the environment, while the global reference frame can encompass the entire project.

[0124] A scan reference frame can be absolute data within a given environment, such as a pipeline, but relative to the location of that environment in a global reference frame. In some cases, output data and reports can be provided based on a scan reference frame. Scan reference frame data (e.g., distance is the distance from the pipeline's origin to the access point) can include the origin 0. + / - distances can be measured in directions that can be, for example, + / - X, Y, or Z. In the example where the given environment is a pipeline, +Z can be along the pipeline, level with the ground plane (e.g., without inclination). +X can be vertical (12 o'clock = 0°). +Y can be horizontal to the right (3 o'clock = 90°). +X and +Y can be mapped to be orthogonal to +Z. In some cases, rotational data can be referenced in a clockwise direction. Thus, for example, 12 o'clock can be 0°, which can be the +X axis; 3 o'clock can be 90°, which can be the +Y axis; 6 o'clock can be 180°; and 9 o'clock can be 270°.

[0125] The robot's main reference frame can be calculated as a relative reference frame for the robot body. This reference frame can be managed by the robot's IMU 26-a. This data can mitigate the possibility of the robot tipping over and becoming stuck, such as if the robot is performing automated cutting. For the robot's main reference frame, +Z can be directly forward. +X can be directly upward. +Y can be directly to the right. +X and +Y can be mapped to be orthogonal to +Z. In some cases, rotational data can be referenced in a clockwise direction. For example, 12 o'clock can be 0°, which can be the +X axis; 3 o'clock can be 90°, which can be the +Y axis; 6 o'clock can be 180°; and 9 o'clock can be 270°.

[0126] A sensor reference frame can be a relative reference frame to the sensor platform associated with the robot. The sensor platform can pan, tilt, and rotate relative to the robot's body reference frame. Sensor data can be transformed into a scanning reference frame to perform calculations. For the sensor reference frame, +Z can be directly forward. 0 distance can be the start of data aggregation and / or after calibration. + distance can be in the +Z direction. +X can be directly upward. +Y can be directly to the right. +X and +Y can be mapped to be orthogonal to +Z. In some cases, rotational data can be referenced in a clockwise direction. For example, 12 o'clock can be 0°, which can be the +X axis; 3 o'clock can be 90°, which can be the +Y axis; 6° clock can be 180°; and 9° clock can be 270°.

[0127] During the scanning operation, the pose XYZ coordinates can be derived within the fusion layer based on the robot's kinematics. The fusion layer is formed by comparing and weighting individual sensor readings together, as discussed elsewhere in this document. The pose XYZ coordinates can be reported as the center of the LiDAR readings within the scanning reference frame and are used as the origin of the sensor reference frame associated with the LiDAR (i.e., the LiDAR reference frame) for such calculations and subsequent calculations until a new, updated pose XYZ is received. Each updated pose XYZ can be timestamped to reconcile and correlate the various readings and verify that calculations are being performed using the latest pose XYZ data.

[0128] Attitude orientation angles can also be derived within the fusion layer based on the robot's kinematics, IMU readings, and other sensor data. Attitude orientation can be reported as rotation angles around the XYZ axes based on a scanned reference frame of the LiDAR reference system.

[0129] As an example, for each LiDAR XYZ sample, the LiDAR XYZ data can be determined by transforming the pose XYZ from the LiDAR reference frame into a scan reference frame. The LiDAR XYZ data can then be rotated around the pose XYZ to determine the scan XYZ, which is the 3D output point within the scan reference frame.

[0130] Figure 14 and Figure 15 This is an exemplary diagram illustrating the robot system, the body reference frame, and the sensor reference frame.

[0131] Service location. As service pipes are detected and their locations are recorded, the system saves and records all service pipes within the map, which will then be used for reference during the cutting step. The mapping step of the first scan will be used to create a sufficient digital representation of the environment, which can be used for localization in later stages. The map will include all 3D representation points of the environment, including nominal diameters, locations of all key observations, and references within the pipes. As the robot system moves through the environment, this map will be continuously updated and adjusted based on the latest readings.

[0132] Specifically, the location of the branch pipe will be described by the 3D coordinates of the outer contour line describing the branch pipe opening, so as to fully describe a series of points outlining the contour. On one hand, there may be an option to determine the branch pipe requiring a circular cut using center coordinates and the radius of the desired circular cut. This can include enhancing the description of the circle by stretching or compressing different sizes of the circle. Other compressed shape descriptions are also possible, including ellipses with different heights and widths, rectangles with four corners, or any number of sides of an irregular polyhedron including a list of points sufficient to describe the irregular polyhedron. Each point in the list will correspond to a vertex of the polyhedron in the order of the outer contour line describing the polyhedron's contour. For a more precise representation, more points can be used in the list. The system can dynamically adjust the number of points necessary to uniquely describe the branch pipe based on the complexity of the branch pipe, memory usage, and the time spent operating on the branch pipe.

[0133] The map can be reloaded by the same robotic system or stored and uploaded to another robotic system. This allows work to occur at different times. Therefore, scanning can be performed at different times of the day and possibly by different human operators. Each scan will have a unique descriptor that allows the entire robot fleet and fleet management system to coordinate the data and map between each robot. For example, the descriptor may include the date and time of day of the scan, the serial number of the robot performing the scan and / or the sensors used by the robot, external environmental data that may affect the scan, operator identification, and other data that will uniquely describe the scan. Another benefit of this data synchronization is the ability to handle robot equipment failures without delay or loss of functionality and to allow operators to switch to a backup robot.

[0134] Navigate and cut within the mapped environment. (Reference) Figure 16An exemplary process 100 is illustrated. Before performing any service, such as cutting, the robotic system performing the service loads a scanned digital image at 101. During the loading process, the system can then determine an optimized navigation path and classify the service based on the optimized path. At 102, the robot can navigate to the first service location. For example, the robot can first navigate to the furthest location to perform the first process and then return in reverse direction towards the origin as it performs additional processes at each service location. In addition to generating efficiency, processing in this order can also prevent any processing, such as cutting operations, from causing navigation problems through the mapped environment. By reaching the furthest location first, the full scan and digital map can be verified and localized by the robot as it runs the detection for the cutting service. It should be understood that navigating to the furthest service first is merely an example, and the order of processing at various service locations can vary. As a further example, classification can be based on the type of process to be performed at various service locations, which can reduce the number of times a particular drill bit or tool needs to be replaced.

[0135] At point 103, the cutting path can be presented, which may include, for example, the outline of the target cut, drill bit selection, drill bit speed, and other parameters associated with the cut. At point 104, the cutting operation is initiated. As used herein, the “cutting path” of the cutting tool includes radial movement toward and away from the robot, lateral movement relative to the robot, pivoting, rotation, or a combination of these movements. During the cutting operation, real-time sensor feedback will be generated and received at point 105. At point 106, the operator can provide feedback on the cutting process. At point 107, any adjustments to the cutting process that are deemed necessary or preferred can be made. At point 108, the cutting operation is considered complete. At point 109, the results of the cutting process are recorded, and at point 110, the sensor feedback and the results of the cutting process can be fed back into the AI ​​model for further training.

[0136] At the end of the cutting process or sub-process, the robot and cutting tools can be repositioned for subsequent cutting using pre-programmed navigation and cutting path supplementation via IR overlay on the RGB camera display.

[0137] An inspection phase can exist separately from the scanning phase, or the inspection and scanning phases can be the same. The movement speed during navigation can vary and differ from the movement speed during inspection or cutting operations. During the inspection phase, the robot system can have a configurable forward movement speed. This speed can be adjusted based on customer requirements, operator preferences, environmental conditions, or based on system and sensor feedback. On one hand, the robot can move as fast as possible to complete the scanning operation as quickly as possible. On the other hand, the speed can be limited to ensure that the RGB camera video is visible and undistorted, avoiding situations where all observations are not clearly displayed. During non-inspection operations such as the cutting phase, the robot can be allowed to move more quickly between different branches to reduce the total time spent in the environment.

[0138] Navigation through the environment will be monitored in real time based on various onboard sensors associated with the robot. Initially, movement can be set to a relatively fast initial speed, and then, if adverse environmental conditions are detected, the robot can slow its speed or change its route to navigate through such adverse conditions safely or more efficiently. Adverse environmental conditions can be based on sensor and system feedback from the environment, as well as operator feedback. Sensors such as IMUs, RGB cameras, and LiDAR can detect conditions that may contribute to reduced navigation speed. If the surface is smooth and relatively flat, the robot may be able to move faster than in conditions with cracks, bumps, and obstacles. Sensor feedback can be used to calibrate the navigation speed. Additionally or alternatively, adaptive machine learning algorithms can use sensor feedback to adjust for different environments.

[0139] If the robotic system has been preloaded into the map from previous scans, the movement speed can also be varied. The images from previous scans contain 3D mapping information, as well as other prior sensor data that can be invoked to predict and calculate the desired movement speed. If the robot observed cracks or bumps during a previous operation, it might reduce its speed before encountering the same conditions during a subsequent scan. This helps prevent potential damage or unintended behavior of the system by using references from previous scans to aid future movement requests. Additionally, previous scans can contain all previously observed locations and branches where the robot can be programmed to reduce speed or stop.

[0140] The disclosed system will move the robot to the location of the first service point. The optimized planned path will be executed by the robot using inverse kinematics calculations, comparing the robot's current 3D coordinates (robot reference frame) with the 3D coordinates of the first service point (scan reference frame). The difference in position between the current robot point and the next service point will determine the direction of movement.

[0141] There may be discrepancies between the LIDAR point cloud image and the IR image projected into 3D space relative to the cutting target. In embodiments, the initial cut may be limited to the overlap between the point cloud image and the IR image to reduce the risk of incorrect alignment between the branch pipe opening and the cut. Once the overlapping portion has been cut, a secondary cut or filing may be performed to further align the cut with the branch pipe opening. For example, a four-inch diameter branch pipe opening may have an initial cut of three inches in diameter, and a secondary cut or filing operation may then be performed to increase the cut diameter to match the branch pipe diameter at four inches. The secondary cut can be determined based on the actual branch pipe opening boundary (or a portion thereof) revealed by the initial cut. For example, but not limited to, optical (e.g., RGB) scanners work well to define the entire opening when the edges of the opening are optically perceptible. Therefore, a secondary optical scan can provide more accurate information about the size, shape, and location of the branch pipe opening. It should be understood that other sensors may be used individually or in combination to perform the secondary scan. The boundary can be used to more accurately calculate the location of the opening, allowing the liner to be cut without contacting the pipe. Similarly, the actual boundary can be determined by detecting the contact between the cutter and the pipe during the initial cutting process.

[0142] Cutting operation. Once the robot reaches the location of the branch pipe, it will present a cutting path. There are many factors that affect the cutting path, including but not limited to the cutting speed, cutting accuracy, safety margin of the main pipe, shape and size of the branch pipe opening leading to the main pipe, or the shape and size of the chips (i.e., shavings or scrapes).

[0143] The cutting path will be calculated by the robotic system based on a map of the branch pipe. The points describing the branch pipe will describe the outline of the polyhedron and will be used as the boundaries of the cutting path.

[0144] While the ultimate goal of the cutting action is to complete the entire cutting path as quickly and efficiently as possible, the initial cutting location can prioritize the accuracy, precision, and safety of the operation. In one aspect, the cutting path can begin at the safest point on the branch pipe to ensure a safe start to the process. The safest cutting point can be based on the maximum distance from any possible edge of the main pipe. This will be the location where sensors, mapping, positioning, and all processing operations predict and calculate the least likely point of impact and damage to the main material. For example, in one instance, the branch pipe may extend at an angle to the main pipe. This can pose a challenge to cutting to re-establish fluid communication with the branch pipe, as the wall of the branch pipe will be positioned close to the opening leading to the main pipe. The orientation of the branch pipe can be determined through an initial scan or otherwise known. Using this information, the robot can automatically position itself relative to the branch pipe opening, allowing the cutting tool to extend from the robot at an angle approximately parallel to (or coincident with) the central axis of the angled branch pipe for cutting the liner. This substantially reduces the chance of impacting all branch pipes.

[0145] The initial cut begins by initiating a cutting tool to rotate the drill bit and move the cutting tool toward the liner. This can be accomplished via an extension that can take the form of one or more arms. The movement of the arms is driven by one or more motion motors on the robot and tracked by sensors such as encoders. When the initial cut begins, feedback is provided on how the material reacts to the cutting operation. Many variables influence the cutting operation and the interaction with the material. Some of these conditions can be known in advance and can be calibrated or adjusted in calculations. These will include, but are not limited to, the type of cutting head used, the thickness of the liner to be cut, the method of curing the liner, etc. Other conditions can be varied, but such other conditions can also be derived or calculated from sensors or other devices. These include, but are not limited to, drill bit aging / wear, cutter RPM, current drawn from the cutter, current drawn from other motion motors, cutting angle, and the temperature of the liner at the cut, etc.

[0146] During the cutting operation, the robotic system will make adjustments in real time using known, sensed, and derived inputs, or non-real-time adjustments before re-engaging the cutting operation if the cutting operation is paused or otherwise delayed. The size of the cut will limit the cutting path, but the movement within these limits can be adjusted to ensure safe and reliable cutting. Some pre-calculated adjustments may exist based on known information. In one aspect, if the liner to be cut is known or determined to be thick, the adjustment may be to slow down the movement during cutting. Different drill bits may have different calibrations and configurable feed rates. If the cutting drill bit is removed from the grinding wheel, the feed rate of the cutting movement may be slowed down.

[0147] Additionally, the robotic system can automatically adjust based on sensed and derived inputs. If a decrease in the motor's revolutions per minute (RPM) is sensed, the cutting action may be compromised, potentially damaging the drill bit. As a corrective mechanism, the robot can slow the feed rate or reverse the direction of motion to prevent damage. As another example, if the current drawn by the cutter is higher than expected, or if it causes the cutter to travel a longer distance into the liner than intended, the RPM and / or feed rate can be slowed down in these cases. In some instances, the cutting operation can be stopped if sensor readings exceed a predetermined threshold.

[0148] When the robotic system performs a cut, parameters and sensors can be adjusted for the cutting motion using instantaneous real-time sensor feedback. All sensor data, input information, operator override, and any other available data are saved and recorded for use in each branch cut. This information can be used for subsequent cuts within the same pipe or as training for future cuts in other environments. For example, if a feed rate reduction is required for a cut in an environment, a predictive algorithm can determine that the feed rate should be reduced for other cuts in the same environment. This allows for improved real-time cutting timing and reliability adjustments in the robotic system's operation.

[0149] Offline and post-processing review and analysis can be used to further optimize and refine path planning and error recovery planning. This will utilize metrics from scanning and cutting operations to identify potential areas for improvement. This could include reducing total scanning time, limiting time spent moving between branches, reducing time spent cutting, or many other reported metrics. This allows learning and feedback from a single robot to be applied quickly and easily across the entire fleet without each robot needing to encounter the same conditions.

[0150] Operator assistance. While this disclosure has been described to date as relating to systems and methods for controlling and employing autonomous robots deployed in confined environments, it also includes an operator interface that can be based on a digital twin of the environment or a digital twin augmented with real-time sensor data such as two-dimensional cameras or three-dimensional thermal images, thereby creating images based on virtual reality or augmented reality. The operator interface can be local to the environment or remote. Additionally, an operator can intervene in the movement of the robotic system. The operator can observe conditions that the robot has not yet detected, or other reasons where human intervention is advantageous.

[0151] refer to Figure 17 and Figure 18The diagram illustrates an exemplary process 1100, which includes a high-level description of an operator's use of an augmented reality system. At 1101, a robot with cutting tools navigates to a first branch pipe location. At 1102, an associated frame of reference is established. At 1103, RGB video and IR images are transmitted from the robot's sensors to a display to create an augmented reality image 1104. In one embodiment, the augmented reality image 1104 is overlaid on a video monitor that can be viewed by an operator. The augmented reality image 1104 may be, for example, the shape and location of a sensor leading to the main pipe. At 1105, the display uses the augmented reality image to display the target and the cutting path. In some embodiments, the operator selects a cutting path from a predetermined list of cutting paths. A series of steps may then follow, for example, at 1106, to determine whether the cutting process is ready to commence. If not, any adjustments are made at 1107, and the process continues at 1106 with the same inquiry. If the process is ready, it is determined whether the cutting process should be fully automatic at 1108 and 1109, semi-automatic at 1110 and 1112, or manual at 1112. Specifically, when using the manual process 1112, controls are provided that allow the operator to adjust the transparency of the augmented reality image 1104 overlaid on the monitor so that it appears to be on the liner in the main pipe. Increasing transparency may be useful for the operator when performing manual cutting. It is also conceivable that the transparency of the augmented reality image 1104 could be controlled by a controller.

[0152] Continuing at point A, at 1113 the live feed of the cut is provided to the display monitor. Real-time sensor feedback is received at 1114. At 1115, it is determined whether any anomalies related to the cut have been detected. An example of an anomaly is a mismatch between the measured cut feed rate and the predicted cut feed rate. If no anomalies are detected, at 1116 the process is queryed to determine if it is complete. If not, the process continues at 1117. If the process completes at 1116, a post-processing check is performed at 1123.

[0153] However, if an anomaly is detected at 1115, a decision point regarding whether to abort the process exists at 1118. If so, the process terminates at 1119. If aborting the process is not necessary, a determination is made at 1120 whether to pause the process, and the process is paused at 1121. Regardless of whether the process is paused, any necessary or desired adjustments are performed at 1112, and the process proceeds as planned. Figure 11 Continue as shown in b.

[0154] As described in the advanced section above, the cutting path and the boundaries of the branch pipe to be cut can be enhanced to appear as an overlay on a 2D image from a live video feed, such as... Figure 19As shown. This will allow the operator to observe the predicted and detected locations of the branch pipe. Without augmentation overlay, the operator will only have visual information from the camera. The augmented display of the branch pipe can include confidence information to represent the safety margin of the cutting path and the calculated probability that the sensor correctly locates the branch pipe within the map.

[0155] Figure 20 Another example view shown is displayed by the operator as the robot moves through the pipe. In this example, the robot's speed and the distance traveled by the robot are shown on the left. The black-and-white photograph on the left shows a digital map of the inside of the pipe, while the color photograph on the right is a digital map highlighting the IR output, where the various colors represent the temperature gradient within the field of view.

[0156] During the cutting process, the sensor can provide the following actionable feedback: The encoder on each joint can provide some measurements of linear motion in the forward or backward direction.

[0157] An IMU can provide feedback on the rotation of the orientation of a rigid body within a scanning reference frame. For example, the reference frame could be a robot reference frame (movement at the robot's base), a sensor scanning reference frame (the direction the camera is aiming), and a tool reference frame (i.e., the cutting drill bit).

[0158] Acceleration can be calculated to indicate whether the robot is tilting, shaking, moving, or stuck.

[0159] Real-time RGB camera feeds provide video and augmented reality views of the environment.

[0160] An IR camera provides thermal mapping to detect and confirm what might be in openings outside the liner or pipe material. IR images can be overlaid on digital images and RGB camera feeds as further enhanced realistic images. Operators and / or artificial intelligence systems can have the ability to adjust the temperature range used by the IR camera in its processing. Such fine-tuning can enhance contrast for better detection of branch locations. For example, underground temperatures in Phoenix and AZ will be significantly different from those in Minneapolis and MN, resulting in very different thermal images. Vapor-cured pipes can have different thermal properties than water-cured pipes, and water-cured pipes can differ from UV-cured pipes. In addition to allowing operators to fine-tune the thermal range, such adjustments can also be automated through training systems.

[0161] Figure 21 An exemplary simplified block diagram is shown, illustrating the detection and classification of the anomaly at 1300 and the corrective action taken at 1301. Although Figure 21The diagram shows that each anomaly has a corresponding corrective action, but in reality, each anomaly can be mapped to more than one corrective action, and multiple anomalies can be mapped to a single corrective action.

[0162] When unexpected conditions occur, there are many possible actions that can be performed automatically or with operator assistance. These actions will include, but are not limited to, the following: The robot may attempt to reverse its direction of movement. The robot may stop all joints and move toward a safe coordinate. This safe coordinate can be calibrated or adjusted, but in a preferred embodiment, it will be a point located at the center of the pipe. The robot may present a warning to the operator and require manual intervention before proceeding with automatic movement.

[0163] The detected conditions may present more complex situations. The robot might get stuck on one joint, but the reaction could be triggered by the movement of another joint. This could happen if the cutting drill rises and collides with the wall and gets stuck as the robot moves forward. The robot can utilize a calibrated set of actions, or it can employ deep learning or other machine learning methods to adapt its motion based on the current input. Corrective motion planning can be dynamically adjusted based on the history of current sensor inputs and previous actions to attempt the most likely successful solution steps. The system will also subsequently record all corrective motion steps across the fleet, allowing learning from one system to be directly applied to another, preventing the repetition of the same faults.

[0164] One feedback method may include monitoring the motor current flowing from the joint motors and detecting any unexpected changes in the current. A sudden spike in the current might indicate that the robot has accidentally collided with or gotten stuck on something. When this is detected, the system can stop moving and take corrective action.

[0165] Another feedback method would be to monitor the IMU sensors for the orientation of all joints in the robot. If the IMU detects that the robot is veerging or tilting unintentionally, the currently planned path may not be optimal due to changes in the environment. If the IMU detects unintended movement, the robot system can take one of its possible corrective motion plans.

[0166] Another feedback mechanism will detect slippage. Because of the presence of water, cracks, or other obstacles in the system, wheel movement may not always result in forward motion. When slippage is detected, the robot system will take corrective action. If the wheel encoder is changing, but the IMU or other sensors are reporting no change in position, the wheel is slipping. The robot can then take one of the possible corrective actions.

[0167] The robot system's feedback and control will utilize real-time operation on the device. This means that as the robot moves within its environment, sensor data will be processed, motion will be calculated, and corrective actions will be executed in real time. This will ensure the robot adjusts and reacts, preventing damage and ensuring efficient operation.

[0168] Artificial intelligence algorithms. The collected sensor data can be collected and analyzed in real time by edge devices or sensor fusion networks 27. Various AI algorithms can be used to process the sensor data. As examples only, data from a single sensor, autoregressive integral moving average (“ARIMA”), or statistical analysis models that use time series data to better understand the dataset or predict future trends can be used to analyze single sensor data. In another embodiment, long short-term memory (LSTM) networks and recurrent neural networks (RNNs) can be used to analyze single or multiple sensor inputs. For pattern detection, Gaussian distribution models can be used.

[0169] For anomaly detection, AI algorithms can include multi-scale convolutional recurrent encoder-decoder (MSCRED), local outlier factor (LOF), or model autophagy disorder (MAD), each of which can be used to perform anomaly detection and diagnosis in multivariate time series data.

[0170] In each case, after training the AI ​​model, the input to the AI ​​algorithm can be a sequence of sensor data for the immediate preceding time period, which can be measured in seconds, minutes, or longer, for example, and the output of the AI ​​algorithm can be a predicted data sequence for future time periods.

[0171] Various aspects of this disclosure. The following are various exemplary and non-limiting aspects of this disclosure.

[0172] Aspect set A. Robots with sensor packages including IMU and LiDAR.

[0173] Aspect 1. A robot, the size and shape of which are designed to be housed in a pipe, the robot comprising: a chassis configured for movement of the robot on the pipe; a tool supported by the chassis for movement relative to the chassis; multiple sensors including an inertial measurement unit (IMU), an encoder, and an optical detection and ranging sensor (LIDAR) associated with the robot; and a sensor fusion system operable to combine readings from the IMU, encoder, and LIDAR to determine the position of the robot within the pipe.

[0174] Aspect 2. The robot according to Aspect 1, wherein the sensor fusion system is operable to use machine learning to determine the robot's position.

[0175] Aspect 3. The robot according to Aspect 2, in which multiple sensors also include a two-dimensional camera, and the sensor fusion system is operable to map the interior of the pipe.

[0176] Aspect 4. Based on the robot of Aspect 3, wherein the sensor fusion system correlates video data from a 2D camera with the robot's position to map the interior of the pipe.

[0177] Aspect 5. The robot according to Aspect 2, wherein the sensor fusion system is operable to resolve discrepancies in data received from the IMU and encoder to determine the robot's position within the pipeline.

[0178] Aspect 6. The robot according to Aspect 1, wherein the encoder is configured to measure the distance traveled by the tether attached to the robot.

[0179] Aspect 7. A robot, the size and shape of which are designed to be housed in a pipe, the robot comprising: a chassis configured for movement of the robot within the pipe; wheels connected to the chassis for movement of the robot relative to the pipe; multiple sensors including an inertial measurement unit (IMU), an encoder, and a light detection and ranging (LIDAR) sensor associated with the robot; and a sensor fusion system operable to combine readings from the IMU, encoder, and LIDAR to create a digital map of the interior of the pipe.

[0180] Aspect 8. The robot according to aspect 7 also includes a two-dimensional camera, and wherein the sensor fusion system is operable to combine three-dimensional data from the LIDAR sensor and two-dimensional data from the camera to map the interior of the pipe.

[0181] Aspect 9. The robot according to aspect 8 also includes an infrared camera configured to detect the relative temperature associated with the inside and outside of the pipe.

[0182] Aspect 10. A robot configured to traverse an environment, the robot including a chassis having a plurality of wheels configured to rotate relative to the chassis, the wheels further including: an encoder configured to determine the distance traveled by the wheel relative to the environment; a plurality of sensors associated with the robot configured to utilize the robot traversing the environment and also configured to sense data in real time while the robot is traversing the environment; and a sensor fusion system operable to combine readings from the plurality of sensors to create a digital map of the environment.

[0183] Aspect 11. The robot according to Aspect 10, wherein the first sensor among a plurality of sensors is a two-dimensional camera, and the second sensor among a plurality of sensors is a light detection and ranging (LIDAR) sensor configured to generate a three-dimensional image, and wherein a sensor fusion system is operable to compute a first reference frame of the two-dimensional camera and a second reference frame of the LIDAR sensor, and to combine the outputs from the two-dimensional camera and the LIDAR sensor adjusted based on the first and second reference frames to create a digital map.

[0184] Aspect 12. According to the robot of aspect 11, the third sensor among multiple sensors is an inertial measurement unit (IMU), and the sensor fusion system is operable to refine the dimensions associated with the digital map.

[0185] Aspect 13. The robot according to aspect 12, wherein the fourth sensor among multiple sensors is an infrared camera, and wherein the digital map is created using added thermal features of the environment based on the infrared camera at different points in the environment.

[0186] Aspect 14. The robot according to aspect 13, wherein thermal features indicate elements of the environment that are not visible to two-dimensional cameras or LiDAR sensors.

[0187] Aspect 15. The robot according to aspect 10, wherein the sensor fusion system is also operable to control the operation of multiple sensors.

[0188] Aspect 16. The robot according to Aspect 10, wherein one of the multiple sensors is an inertial measurement unit (IMU), and the IMU is mounted on an articulated arm attached to the robot.

[0189] Aspect 17. The robot according to aspect 16, wherein the sensor fusion system is also operable to control the movement of the articulated arm.

[0190] Aspect 18. According to aspect 17, the robot, wherein the sensor fusion system is also operable to calculate a reference frame associated with the articulated arm based on sensor data from the IMU.

[0191] Aspect 19. The robot according to aspect 10 also includes tools, and wherein a sensor fusion system is operable to control the use of the tools.

[0192] Aspect 20. The robot according to aspect 19, wherein the robot is configured to traverse the environment and locate the tool at a predetermined point in the environment based on a digital map.

[0193] Aspect 21. The robot according to aspect 20, wherein the first sensor among a plurality of sensors is a two-dimensional camera, and the second sensor among a plurality of sensors is a light detection and ranging (LIDAR) sensor configured to generate a three-dimensional image, and wherein a sensor fusion system is operable to calculate a first reference frame of the two-dimensional camera and a second reference frame of the LIDAR sensor, and combines the output from the two-dimensional camera and the LIDAR sensor adjusted based on the first and second reference frames to create a digital map showing predetermined points.

[0194] Aspect set B. Real-time feedback loop for multi-sensor applications.

[0195] Aspect 1. A method comprising: having a robot assess characteristics of an environment based on data from one of a plurality of sensors, wherein the robot is situated in the environment; having the robot compare the characteristics of the environment with expected characteristics of the environment; having the robot create a feedback loop based on the comparison step; and adjusting the operating conditions of the robot based on the feedback loop.

[0196] Aspect 2. According to the method of Aspect 1, wherein the sensor is an inertial measurement unit (IMU) and the feedback loop includes the measurement of vibration.

[0197] Aspect 3. According to the method of aspect 2, the adjustment step includes adjusting the speed of the attached motor, and wherein the characteristic of the feature is the hardness of the feature.

[0198] Aspect 4. According to the method of aspect 1, wherein one or more of the plurality of sensors are current sensors, and wherein the feedback loop includes spikes in the current measurement.

[0199] Aspect 5. According to the method of aspect 4, the adjustment step includes adjusting the speed of the attached motor, and wherein the characteristic of the feature is the hardness of the feature.

[0200] Aspect 6. The method of aspect 1, wherein one or more of a plurality of sensors determine the rotational speed of the attached motor, and wherein the method further includes comparing the rotational speed of the motor with a desired rotational speed of the motor.

[0201] Aspect 7. The method according to aspect 6 further includes determining the characteristics of the feature, wherein the characteristic is the hardness of the feature, and wherein the adjustment step is changing the speed of the motor.

[0202] Aspect 8. According to the method of aspect 1, wherein the plurality of sensors include one or more joint encoders, the one or more joint encoders being configured to encode the pose of the hinge portion of the robot, and wherein the feedback loop includes changes in the pose of the hinge portion of the robot.

[0203] Aspect 9. According to the method of Aspect 1, wherein the sensor is an inertial measurement unit (IMU) and the feedback loop includes the measurement of vibration.

[0204] Aspect 10. According to the method of aspect 9, the adjustment step includes adjusting the robot's posture to smooth the vibration.

[0205] Aspect 11. The method according to aspect 9 further includes determining the vibration based on material properties within the environment, and the adjustment step includes adjusting the speed of the motors attached to the robot.

[0206] Aspect 12. According to the method of aspect 1, the feature is a material property related to the environment.

[0207] Aspect 13. According to the method of aspect 12, wherein the material property is the hardness of the material that forms part of the environment.

[0208] Aspect 14. The method according to aspect 1 also includes the robot traversing the environment, wherein the feedback loop is based on sensed changes in the environment.

[0209] Aspect 15. According to the method of Aspect 1, the expected characteristics of the environment are probabilities based on the expected characteristics using a machine learning algorithm.

[0210] Aspect 16. According to the method of aspect 1, the expected characteristics of the environment are determined in advance, and a comparison step is performed using a machine learning algorithm.

[0211] Aspect 17. According to the method of aspect 1, the feedback loop is configured to provide input to a machine learning algorithm.

[0212] Aspect 18. According to the method of aspect 18, the feedback loop is configured to further train the machine learning algorithm.

[0213] Aspect set C. Automatic adjustment of the robot's reference frame.

[0214] Aspect 1. A method for controlling the movement of a robot in an environment, wherein the robot has a plurality of sensors associated therewith, the method comprising: determining an initial scanning reference frame for the robot, wherein the scanning reference frame is calculated based on an initial origin; calculating a new origin based on readings from at least one of the sensors; and adjusting the initial scanning reference frame based on the new origin to determine a new scanning reference frame, wherein the adjustment step is based on the difference between the initial scanning reference frame and the new scanning reference frame.

[0215] Aspect 2. According to the method of Aspect 1, wherein the initial scanning reference frame for the robot includes an initial scanning reference frame for each of the plurality of sensors based on the pose of each of the plurality of sensors, and wherein the adjustment step includes adjusting the initial scanning reference frame for each of the plurality of sensors to a new scanning reference frame for each of the plurality of sensors.

[0216] Aspect 3. The method according to aspect 2 further includes calculating a correction factor, wherein the adjustment step includes applying the correction factor to calculate a new scanning reference frame for each of the plurality of sensors.

[0217] Aspect 4. According to the method of Aspect 1, the environment is modified, and the step of calculating a new origin is initiated by modifying the environment.

[0218] Aspect 5. According to the method of Aspect 1, wherein at least one of the sensors is an inertial measurement unit (IMU), and wherein the IMU calculates a new origin based on the IMU's position within the environment.

[0219] Aspect 6. According to the method in aspect 5, the IMU calculates the new origin while traversing the environment.

[0220] Aspect 7. According to the method of aspect 6, it also includes: calculating the correction factor based on the difference between the initial origin and the new origin.

[0221] Aspect 8. According to the method of aspect 7, the robot moves within the modified environment, and the robot uses a correction factor to calculate the distance it will travel in the environment.

[0222] Aspect 9. According to the method in Aspect 8, where the IMU is used to calculate the distance.

[0223] Aspect 10. According to the method of aspect 8, a motor encoder is used to calculate the distance.

[0224] Aspect 11. The method according to aspect 7 also includes a table that compares the original distance in the environment with the modified distance in the modified environment based on a correction factor.

[0225] Aspect 12. According to the method of Aspect 7, the robot uses machine learning to perform the aggregation and initiation steps.

[0226] Aspect 13. According to the method of aspect 1, the robot is configured to (1) collect data from multiple sensors, and (2) initiate adjustment steps based on the data collection.

[0227] Aspect 14. A method for controlling the movement of a robot in a modified environment, wherein the modified environment is based on an original environment, the method comprising: scanning the modified environment using a scanning device while the robot traverses the modified environment; monitoring the position of the scanning device in the modified environment relative to a reference frame of the original environment; and periodically adjusting the reference frame of the scanning device relative to the reference frame of the original environment based on the monitored position of the scanning device.

[0228] Aspect 15. According to the method of aspect 13, the periodic adjustment of the reference frame includes adjustments based on the size difference between the original environment and the modified environment.

[0229] Aspect 16. According to the method of aspect 15, the original environment is a pipe, and the modified environment is a liner placed inside the pipe.

[0230] Aspect 17. A method for fusing sensor data associated with a robot in an environment, wherein the robot has a plurality of sensors associated with it, the method comprising: determining an initial scanning reference frame for the robot, wherein the scanning reference frame is based on an initial pose of the robot; determining an initial sensor reference frame for each of the plurality of sensors, wherein the initial sensor reference frame is based on a corresponding initial pose of each of the plurality of sensors; establishing a new origin, the new origin modifying the initial pose of the robot and the corresponding initial pose of each of the plurality of sensors; calculating an updated reference frame based on the difference between the initial pose of the robot and the modified pose of the robot and the difference between the corresponding initial pose of each of the plurality of sensors and the modified corresponding pose of each of the plurality of sensors; and fusing the sensor data based on the calculation steps.

[0231] Aspect 18. According to the method of aspect 17, wherein the robot's initial frame of reference is based on an initial environment through which the robot is traversing, and the robot's updated frame of reference is based on a modified environment, wherein the modified environment has at least one dimensional metric that is different from the initial environment.

[0232] Aspect 19. According to the method of aspect 18, wherein the initial environment is a pipe and the modified environment is a liner placed inside the pipe, and wherein the different dimensional measure is the changed radius of the pipe.

[0233] Aspect set D. The robot uses digital maps for autonomous driving.

[0234] Aspect 1. A method comprising: creating a digital map of an environment; loading the digital map onto a mobile robot, wherein the robot is placed in the environment; generating a trajectory path plan from a current location to a desired location using the digital map, the trajectory path having multiple waypoints; enabling the robot to traverse the environment according to the trajectory path plan; collecting sensor data in real time while the robot is traversing the environment; detecting, based on a collection step, at each waypoint whether there is an anomaly between existing waypoints and subsequent waypoints; and performing corrective actions on the robot based on a detection step.

[0235] Aspect 2. According to the method of aspect 1, the anomaly is one of the cracks or protrusions on the surface of the environment.

[0236] Aspect 3. According to the method of Aspect 2, the corrective action is to slow down the robot's speed until the robot navigates from the current path point to a subsequent path point.

[0237] Aspect 4. According to the method of Aspect 3, the sensor data includes motion data from either an IMU or a motor encoder, and the speed is adjusted based on the motion data.

[0238] Aspect 5. According to the method of Aspect 1, where the anomaly is a blockage, and the corrective action is to reverse the robot's direction from the current path point to the previous path point.

[0239] Aspect 6. According to the method of Aspect 1, the anomaly is water in the environment.

[0240] Aspect 7. According to the method of aspect 6, it also includes assessing whether the robot is able to navigate through water and performing corrective actions based on the assessment steps.

[0241] Aspect 8. According to the method of aspect 1, the mobile robot autonomously traverses the environment.

[0242] Aspect 9. According to the method of Aspect 1, the environment is modified after a digital map is created, and sensor data is used to create a second digital map based on the modification.

[0243] Aspect 10. According to the method of aspect 9, a digital map is created based on a first scanning reference of the robot, and a second digital map is created based on a second reference frame of the robot.

[0244] Aspect 11. According to the method of aspect 10, wherein the environment is the interior of the pipe, and the modification is to insert a liner into the pipe.

[0245] Aspect 12. According to the method of Aspect 9, wherein an adaptive machine learning method is used to create a second digital map based on the digital map.

[0246] Aspect 13. A method for performing operations using a robot in a pipe having a liner installed therein, the method comprising: moving the robot through the liner according to a predetermined movement plan; sensing conditions in the liner using sensors associated with the robot; determining whether the sensed conditions require a change in the robot's movement in the liner; and if it is determined that the sensed conditions require a change, automatically changing the robot's movement in the main pipe.

[0247] Aspect 14. According to the method of aspect 13, automatically changing the movement of the vehicle includes changing at least one of the vehicle's speed or the vehicle's orientation.

[0248] Aspect 15. The method according to aspect 13 also includes referencing a digital map of the pipe made prior to the insertion of the liner, wherein the predetermined movement plan is based on the digital map.

[0249] Aspect 16. The method according to aspect 15 further includes: creating a second digital map of the pipe after the liner is inserted, while the robot is moving inside the liner pipe.

[0250] Aspect 17. The method according to aspect 16 further includes: determining the difference between the digital map and the second digital map, wherein the difference is used to adjust the movement of the robot within the lining pipe.

[0251] Aspect 18. According to the method of aspect 13, the sensing conditions are based on sensor data received from an inertial measurement unit (IMU), a two-dimensional camera, and a LiDAR, and the changes are based on the sensor data.

[0252] Aspect 19. According to the method of aspect 18, adaptive machine learning techniques are used to adjust the robot's movement after the liner is installed.

[0253] Aspect set E. Control of robotic tools in space-constrained environments.

[0254] Aspect 1. A method for controlling a tool carried by a robot in a main pipeline, wherein the tool is operable to extend into a branch conduit extending from the main pipeline, the method comprising: moving the robot in the main pipeline to a location in the main pipeline corresponding to the location of the branch conduit opening into the main pipeline by referring to a digital map of the main pipeline; extending the tool from the robot toward the branch conduit opening at an angle corresponding to the angle formed by the branch conduit and the main pipeline, so as to reduce the likelihood of the tool contacting the wall of the branch conduit.

[0255] Aspect 2. According to the method of aspect 1, the extension tool from the robot includes an extension tool at an angle approximately equal to the angle formed by the branch conduit and the main conduit.

[0256] Aspect 3. According to the method of aspect 2, the mobile robot further includes a robot for positioning the branch conduit opening relative to the angle between the branch conduit and the main conduit.

[0257] Aspect 4. The method according to aspect 3 further includes: performing a first scan of the main pipeline to create a digital map, and then lining the main pipeline with a liner.

[0258] Aspect 5. The method according to aspect 4 also includes cutting the liner in the main pipe with a tool to re-establish fluid communication between the branch conduit and the main pipe through the branch conduit opening.

[0259] Aspect 6. According to the method of aspect 1, the mobile robot further includes a robot for positioning the branch conduit opening relative to the angle between the branch conduit and the main conduit.

[0260] Aspect 7. The method according to aspect 1 further includes: performing a first scan of the main pipeline to create a digital map, and then lining the main pipeline with a liner.

[0261] Aspect 8. The method according to aspect 7 also includes cutting the liner in the main pipe with a tool to re-establish fluid communication between the branch pipe and the main pipe through the branch pipe opening.

[0262] Aspect 9. A robotic system for operating in a main pipe, the robot comprising: a body configured to move along the length of the main pipe within a lined main pipe; a tool mounted on the body for moving relative to the body to perform an operation within the lined main pipe; and a controller operatively connected to the body and the tool for controlling the movement of the tool, the controller being configured to reference a digital map of the main pipe containing information about the intersections of branch conduits and the main pipe to control the angle at which the tool extends from the main pipe, thereby avoiding contact with the wall of the branch conduit.

[0263] Aspect 10. According to aspect 9, the robot is a cutting tool configured to cut into the liner that lining the interior of the main pipe.

[0264] Aspect 11. The robot according to aspect 9 also includes an arm supported by a main body and equipped with tools, the arm being movable relative to the main body.

[0265] Aspect 12. The robot according to aspect 11, wherein the arm has joints connecting the arm to the body, and the robot also includes motors for driving the movement of the arm at the joints.

[0266] Aspect 13. The robot according to aspect 12, wherein the joints include encoders for tracking the movement of the arm.

[0267] Aspect 14. The robot according to aspect 12, wherein the motor is an electric motor, and the robot also includes a current sensor for detecting the current drawn by the motor.

[0268] Aspect 15. The robot according to aspect 14, wherein the controller is operatively connected to a current sensor for receiving data from the sensor.

[0269] Aspect 16. The robot according to aspect 15, wherein the controller is configured to stop the movement of the arm when the detected current exceeds a threshold.

[0270] Aspect 17. The robot according to aspect 9 also includes a display, wherein the controller displays information about the angle of the branch conduit relative to the main conduit.

[0271] Aspect set F. Operational assistance for re-establishing fluid communication in the lining pipe.

[0272] Aspect 1. A method for locating a branch conduit opening leading to a main conduit after lining a main conduit with a liner, the method comprising: moving a robot along the lined main conduit; transmitting a visual image of the liner in the main conduit from a camera associated with the robot to a monitor outside the main conduit for viewing by a human operator; displaying an internal view of the lined main conduit from the transmitted visual image on the monitor; and overlaying an image indicating the location of the branch conduit opening on the internal view displayed on the monitor such that the image appears on the monitor as being located on the liner, as shown in the internal view.

[0273] Aspect 2. According to the method of aspect 1, wherein the overlay image includes information on the confidence level of the overlay image representing the location of the branch duct opening.

[0274] Aspect 3. The method according to aspect 1 further includes: executing override commands in response to intervention by a human operator to influence the operation of the robot.

[0275] Aspect 4. The method according to aspect 1 further includes: using sensors located in the main pipe of the liner to monitor the position of the robot in the main pipe of the liner, and adjusting the overlay of images in response to the detected position.

[0276] Aspect 5. According to the method of Aspect 1, the operator selects from the menu a cutting path for cutting the liner using a robot-supported cutting tool.

[0277] Aspect 6. The method according to aspect 1 also includes providing infrared image data of the liner in the main pipeline.

[0278] Aspect 7. The method according to aspect 6 also includes adjusting the range of infrared image data used to locate features in the main pipeline.

[0279] Aspect 8. The method according to aspect 1 also includes adjusting the transparency of the overlaid images.

[0280] Aspect 9. A system for re-establishing fluid communication between a branch conduit and a main conduit after lining a main conduit with a liner, the system comprising: a robot configured to move within the lined main conduit; a cutting tool supported by the robot for cutting the liner to re-establish fluid communication between the branch conduit and the main conduit; a sensor supported by the robot for acquiring image data of the lined main conduit as the robot moves within the main conduit, the sensor being configured to transmit the image data; a controller operatively connected to the sensor for receiving the image data; and a display operatively connected to the controller for receiving and displaying an internal view of the main conduit based on the image data from the sensor, wherein the controller is configured to overlay an image of an opening of a branch conduit leading to the main conduit at a location determined by the controller as a branch conduit opening on the liner onto the internal view based on the image data from the sensor.

[0281] Aspect 10. The system according to Aspect 9, wherein the controller determines a confidence level associated with the location of the determined branch duct opening and displays the confidence level on a display.

[0282] Aspect 11. According to the system of aspect 9, wherein the controller is operatively connected to the robot for controlling the robot.

[0283] Aspect 12. According to the system of aspect 11, the controller is configured to receive input from a human operator so that the robot can be directly controlled by the human operator.

[0284] Aspect 13. According to the system of aspect 12, the controller is configured to analyze image data and determine the presence of abnormal conditions in the main pipe of the liner, and provide a warning of the presence of abnormal conditions via a display.

[0285] Aspect 14. The system according to Aspect 9 also includes sensors associated with the robot for scanning the main pipe of the liner as the robot moves inside. The sensors are connected to a controller to provide sensor data used by the controller to determine the location of the robot in the main pipe of the liner.

[0286] Aspect 15. According to the system of aspect 14, the controller is configured to adjust the image superimposed on the display according to the location of the robot.

[0287] Aspect 16. According to the system of aspect 9, the controller is configured to display a menu of the cutting path of the cutting tool on the display for the operator to select in order to control the movement of the cutting tool to cut the liner.

[0288] Aspect 17. According to the system of aspect 9, one of the sensors is an infrared sensor configured to provide infrared image data to the controller.

[0289] Aspect 18. According to the system of aspect 17, the controller adjusts the range of infrared image data used to locate features in the main pipeline in response to user input.

[0290] Aspect 19. The system according to aspect 9 also includes a transparency selector for selecting the transparency of an image of the opening of the branch duct.

[0291] Aspect 20. According to the system of aspect 19, a transparency selector is positioned for actuation by an operator.

[0292] Aspect set G. Controlled cutting to re-establish fluid connectivity in the liner pipe.

[0293] Aspect 1. A method for re-establishing fluid communication between a main pipe and a branch pipe, the branch pipe extending from the main pipe after being lined with a liner, the method comprising: moving a robot supporting a cutting tool along the lined main pipe to a location near the branch pipe, the cutting tool being selectively extendable from the robot to cut the liner; extending the cutting tool from the robot toward the liner at a location where the branch pipe has an opening to the main pipe on the opposite side of the liner; cutting the liner at the location using the cutting tool; and monitoring the cutting during the step of cutting the liner, and adjusting the cutting operation based on information acquired by monitoring the cutting tool.

[0294] Aspect 2. According to the method of aspect 1, wherein the step of monitoring the cutting tool includes monitoring at least one of the following: (a) the current draw of the motor that powers the cutting tool, (b) the current draw of the motor that drives the extension of the cutting tool, (c) the temperature of the liner at the cutting location, and (d) the position of the robot relative to the location of the branch conduit.

[0295] Aspect 3. The method according to aspect 1 also includes predicting the resistance of the liner to cutting based on data received by the controller of the cutting tool.

[0296] Aspect 4. According to the method of aspect 3, the data received by the controller of the cutting tool includes at least one of the following: (a) the material of the liner, (b) the thickness of the liner, and (c) the curing parameters of the liner.

[0297] Aspect 5. According to the method of aspect 4, the diameter of the opening to be cut by the cutting tool in the liner is selected based on at least one of the liner material, the liner thickness and the liner curing parameters.

[0298] Aspect 6. According to the method of aspect 3, the prediction of resistance includes executing a machine learning model.

[0299] Aspect 7. A system for re-establishing fluid communication between a main pipe and a branch pipe, the branch pipe extending from the main pipe after being lined with a liner, the system comprising: a robot, the robot being sized and shaped to be housed in and movable along the main pipe; a cutting tool supported by the robot and mounted on an extension configured to move the cutting tool away from and toward the robot, the extension including an electric motor; one or more sensors supported by the robot and operable to detect the environment in which the robot is located; and a controller operably connected to the robot, the cutting tool, and the sensors, the controller being configured to position the robot in the main pipe near the liner of the branch pipe and to extend the cutting tool from the robot toward the liner at a location where the branch pipe has an opening on the opposite side of the liner to the main pipe for cutting the liner with the cutting tool, the controller being further configured to activate one or more sensors to monitor the cutting while the cutting tool is activated to cut the liner, and to adjust the cutting operation based on information acquired by monitoring the cutting tool.

[0300] Aspect 8. The system according to aspect 7, wherein at least one sensor includes at least one of the following: a sensor for monitoring the current draw of a motor powering a cutting tool; a sensor for measuring the current draw of an electric motor of an extension; a sensor for measuring the temperature of the liner at the cutting location; and a sensor for detecting the position of the robot relative to the location of the branch conduit.

[0301] Aspect 9. According to the system of aspect 7, the controller is programmed to predict the resistance of the liner to cutting based on data received by the controller.

[0302] Aspect 10. The system according to aspect 9, wherein the controller is configured to receive input including at least one of the following: (a) the material of the liner, (b) the thickness of the liner, and (c) the curing parameters of the liner.

[0303] Aspect 11. The system according to Aspect 9 also includes a machine learning model, wherein the controller executes the machine learning model to predict resistance.

[0304] Aspect 12. A method for re-establishing fluid communication between a main pipe and a branch pipe, the branch pipe extending from the main pipe after it has been lined with a liner, the method comprising: positioning a robot supporting a cutting tool in the lined main pipe near the branch pipe, the cutting tool being selectively extendable from the robot to cut the liner; extending the cutting tool from the robot toward the liner at a location where the branch pipe has an opening to the main pipe on the opposite side of the liner; cutting the liner with the cutting tool; and controlling the cutting using an input comprising at least one of a material including the liner, a resin for curing the liner, and wear of the cutting tool.

[0305] Aspect 13. According to the method of aspect 12, wherein the step of controlling the cutting includes determining the size of the opening to be cut based at least in part on at least one of the input of the liner material, the resin used to cure the liner, and the wear of the cutting tool.

[0306] Aspect 14. According to the method of aspect 12, wherein the step of cutting the liner includes moving the cutting tool in a helical path.

[0307] Aspect 15. According to the method of aspect 13, wherein the step of cutting the liner includes: moving the cutting tool in a zigzag path.

[0308] Aspect 16. According to the method of aspect 12, input is provided on a drop-down menu.

[0309] Aspect set H. Positioning the branch conduit within the lining pipe.

[0310] Aspect 1. A method for locating a branch conduit opening leading to a main conduit after lining a main conduit with a liner, the method comprising: moving a robot along the lined main conduit; scanning the liner using an infrared scanner mounted on the robot; sensing a temperature drop outside the liner with the infrared scanner to acquire infrared data; comparing the infrared data with other location data regarding the branch conduit opening; and determining the location of the branch conduit opening outside the liner based on the infrared data and the other data.

[0311] Aspect 2. According to the method of Aspect 1, the step of comparing the infrared data with other location data includes comparing the infrared data with a digital map of the main pipeline.

[0312] Aspect 3. According to the method of Aspect 1, the step of detecting the branch catheter opening includes: separating the infrared data acquired by the infrared scanner in the step of scanning the liner with the infrared scanner to represent the range of the highest and lowest temperatures expected to be detected, and ignoring those infrared data outside the range for the step of detecting the branch catheter opening.

[0313] Aspect 4. The method according to aspect 3 also includes selectively adjusting the range of the infrared scanner.

[0314] Aspect 5. According to the method of aspect 1, the step of detecting the branch duct opening includes using software to locate circular features and determine the deviation from the circular features in the data acquired during the scan.

[0315] Aspect 6. The method according to aspect 1 further includes assigning confidence scores to detected branch duct openings.

[0316] Aspect 7. According to the method of aspect 6, the confidence score is changed during the iteration process for locating the branch duct opening.

[0317] Aspect 8. A system for locating branch conduit openings leading to a main conduit after lining a main conduit with a liner, the system comprising a robot, an infrared scanner supported on the robot, and a controller for receiving data from the scanner and controlling the operation of the robot and the infrared scanner, the controller being configured to move the robot along the lined main conduit and simultaneously activate the infrared sensor to scan the liner, the controller being programmed to detect branch conduit openings outside the liner by sensing a temperature drop outside the liner compared to adjacent surfaces using the infrared scanner, to compare the location of the branch conduit opening indicated by the infrared sensor scan with other data regarding the location of the branch conduit opening.

[0318] Aspect 9. The system according to Aspect 8, wherein the controller is programmed to compare the location of the branch conduit opening determined by an infrared sensor with the location of the branch conduit opening based on a digital map of the main conduit previously obtained from prior knowledge of the main conduit prior to the liner.

[0319] Aspect 10. According to the system of aspect 8, the controller is programmed to separate the infrared data acquired by the infrared scanner, obtained by scanning the liner using an infrared scanner, into ranges representing the highest and lowest temperatures expected to be detected, and for detecting branch duct openings, the controller ignores infrared data outside those ranges.

[0320] Aspect 11. According to the system of aspect 10, the ranges of the highest and lowest expected temperatures are adjustable.

[0321] Aspect 12. According to the system of aspect 8, the controller is configured to execute a program to identify circular features in the data acquired from the scan and to determine the deviation from the identified circular features.

[0322] Aspect 13. According to the system of aspect 8, the controller is programmed to assign confidence scores to detected branch duct openings.

[0323] Aspect 14. According to the system of aspect 13, the controller is programmed to change the confidence score during the iteration process for locating the branch duct orifice.

[0324] Aspect 15. A method for re-establishing a branch conduit connection in a main conduit that has been lined with a liner, the method comprising: moving a robot through the lined main conduit; receiving the robot's location in the lined main conduit based on a previous scan of the environment within the main conduit and location data generated by sensors associated with the robot; receiving two-dimensional image data from a camera mounted on the robot, the two-dimensional image data including a view of the environment from the robot's perspective; receiving infrared image data from an infrared camera mounted on the robot; fusing the infrared image data and the camera image data; initiating a cutting process to re-establish fluid communication between the branch conduit and the main conduit through the branch conduit opening when the robot's location is near the location of a branch conduit opening covered by the liner; receiving real-time sensor data during the cutting process; and adjusting the cutting process based on the real-time sensor data.

[0325] Aspect 16. The method according to aspect 15 further includes the step of adjusting the temperature gradient of the infrared camera based on the environmental conditions of the main pipeline.

[0326] Aspect 17. According to the method of aspect 16, wherein the step of adjusting the temperature gradient includes using data from at least one or a combination of the steps of receiving two-dimensional image data and fusing the two-dimensional image data with infrared image data to retrain the infrared data processing algorithm.

[0327] Aspect 18. According to the method of aspect 17, retraining is performed by machine learning.

[0328] Aspect 19. According to the method of aspect 15, the step of initiating the cutting process includes setting parameters for cutting the liner that lines the main pipe to access the branch conduit opening leading to the main pipe.

[0329] Aspect 20. According to the method of aspect 19, the setting parameters include at least one of the following: setting the cutting based on the size of the branch duct opening, setting the cutting speed, and setting the type of cutting to be performed.

[0330] Aspect 21. According to the method of aspect 15, the step of initiating the cutting process includes: performing a cutting operation using a cutter mounted on a robot, and sensing conditions of at least one of the cutter and the robot; and changing the cutting operation based on the sensed conditions.

[0331] Aspect 22. According to the method of aspect 21, wherein the sensed conditions include the movement of the robot.

[0332] Aspect 23. According to the method of aspect 22, wherein the sensed conditions include the cutting rate of the cutter passing through the liner during the cutting operation.

[0333] Aspect 24. The method according to aspect 15 further includes: receiving three-dimensional image data from a LIDAR sensor mounted on the robot; and overlaying the LIDAR image data with at least one of image data from a camera and infrared image data.

[0334] Aspect 25. The method according to aspect 24 further includes: evaluating LIDAR images of protrusions of the liner toward the central axis of the main conduit; and identifying such protrusions as possible locations of branch conduits.

[0335] Aspect 26. The method according to aspect 24 also includes analyzing camera image data, infrared image data, and LiDAR image data in an integrated predictor.

[0336] Aspect 27. According to the method of aspect 26, wherein analyzing camera, infrared, and LiDAR image data includes weighting at least one of the camera image data, infrared image data, and LiDAR image data differently.

[0337] Aspect 28. A method for locating a branch conduit opening leading to a main conduit after lining a main conduit with a liner, the method comprising: moving a robot along the lined main conduit; setting an upper sensitivity range of an infrared scanner to exclude temperatures below a predetermined minimum temperature; setting a lower sensitivity range to exclude temperatures above a predetermined maximum temperature; scanning the liner using an infrared scanner mounted on the robot; and sensing a temperature drop outside the liner using the infrared scanner.

[0338] Aspect 29. According to the method of aspect 28, wherein the steps of setting the upper sensitivity range and setting the lower sensitivity range are each based on the environmental conditions of the main pipeline.

[0339] Aspect 30. A method for determining the resistance of a material, the method comprising: positioning a robot adjacent to a target branch, wherein the robot includes one or more joints and a plurality of sensors, wherein at least one of the one or more joints supports a cutting drill bit powered by a motor; initiating an interaction between the drill bit and the material at the target branch; receiving feedback from one of the plurality of sensors, wherein the feedback is based on the interaction and indicates the resistance of the material; and adjusting the drill bit based on the feedback.

[0340] Aspect 31. According to the method of aspect 30, the sensor includes one or more joint encoders, and the feedback includes changes in encoder position over a time period.

[0341] Aspect 32. The method according to aspect 31 further includes: comparing the position change with a predicted position within a time period; and determining the hardness of the material based on the comparison step.

[0342] Aspect 33. According to the method of aspect 30, the feedback includes the movement speed of one or more joints and the speed of the motor, and the adjustment step is based on a comparison between the movement speed of one or more joints and the speed of the motor.

[0343] Aspect 34. According to the method of aspect 30, the feedback includes the movement speed of one or more joints and vibration data received from an inertial measurement unit (IMU), and the adjustment step is based on a comparison of the movement speed of one or more joints with the vibration data.

[0344] Aspect Set I. Automatic Path Cutting Calculation

[0345] Aspect 1. A method for determining a cutting path to re-establish fluid communication between a main pipe and a branch pipe, the branch pipe extending from the main pipe after being lined with a liner, the method comprising: calculating an automatic cutting path for a cutting tool based on shape and position data relating to an opening of the branch pipe into the main pipe; moving the cutting tool along the calculated cutting path to cut the liner, thereby re-establishing fluid communication between the branch pipe and the main pipe.

[0346] Aspect 2. The method according to aspect 1 also includes creating a digital model of the branch duct opening based on shape and location data, wherein calculating the cutting path includes calculating the cutting path to avoid contact between the cutting tool and the modeled branch duct opening.

[0347] Aspect 3. According to the method of Aspect 2, the digital model defines the shape and position of the branch conduit opening relative to the main conduit.

[0348] Aspect 4. Based on the method of Aspect 1, this includes performing machine learning to determine the path.

[0349] Aspect 5. A method for scanning and cutting a liner in a pipe, the method comprising: creating a digital map of the environment using multiple sensors; identifying a first location of a service based on a scanning reference frame; navigating a robot to the first location using the digital map and based on a robot reference frame; calculating a cutting path based on the digital map; and performing the cutting at the first location.

[0350] Aspect 6. The method according to aspect 5 further includes: identifying a second location for the second operation; and navigating the robot to the second location and performing the cutting at the second location.

[0351] Aspect 7. According to the method of aspect 5, identifying the first location includes determining the first location based on the distance from the origin.

[0352] Aspect 8. According to the method of aspect 5, the calculation of the cutting path includes considering at least one of the type of cutting, the drill bit selected for cutting, and the cutting speed.

[0353] Aspect 9. According to the method of Aspect 5, wherein the step of creating a digital map is performed by a first robot, and the steps of identifying a first location, navigating the robot, and calculating a cutting path are performed by a second robot, the method further includes determining a second reference frame for the sensors of the second robot, and converting data from the digital map created by the sensors of the first robot to the second reference frame of the second robot, thereby compensating for construction differences between the first robot and the second robot.

[0354] Aspect 10. The method according to aspect 1 further includes defining a scan reference frame relative to a location in a global reference frame, and wherein the step of creating a digital map includes locating features within the pipeline relative to the scan reference frame.

[0355] Aspect 11. A system for cutting an opening in a liner of a main conduit to establish fluid communication with a branch conduit, the system comprising a robot including a cutting tool extendable from the robot to cut the liner, the robot being sized to be received in and move along the main conduit after the liner is received in the main conduit, the robot being operatively connected to a controller configured to calculate a cutting path for the cutting tool using data about the shape and location of the opening in the branch conduit leading to the main conduit, and to move the cutting tool along the calculated cutting path to cut the liner to re-establish fluid communication between the branch conduit and the main conduit.

[0356] Aspect 12. The system according to aspect 11, wherein the controller is configured to reference a digital model of the branch conduit opening created in a scan of the main conduit prior to lining with a liner, and wherein the controller calculates a cutting path to avoid contact between the cutting tool and the modeled branch conduit opening.

[0357] Aspect 13. According to the system of aspect 12, the digital model defines the shape and position of the branch conduit opening relative to the main conduit.

[0358] Aspect 14. According to the system of aspect 11, the controller is programmed to use machine learning to determine the cutting path.

[0359] Aspect 15. A system for scanning and cutting a liner in a pipe, the system comprising: a robot having a cutting tool operable to cut through the liner; and a controller for controlling the operation of the robot, the controller being configured to reference a digital map of the environment created during scanning of the pipe using multiple sensors prior to lining the pipe with the liner, the controller being configured to identify a first location of the operation based on a scanning reference frame, and to navigate the robot to the first location using the digital map and based on a robot reference frame, the controller being programmed to calculate a cutting path based on the digital map, and to control the cutting tool to perform cutting of the liner at the first location.

[0360] Aspect 16. According to the system of aspect 15, the controller is configured to identify the second location of the second job, navigate the robot to the second location, and perform cutting at the second location using a cutting tool.

[0361] Aspect 17. According to the system of aspect 15, the controller identifies the first location by determining the first location based on the distance from the origin.

[0362] Aspect 18. According to the system of aspect 15, the controller calculates the cutting path by considering at least one of the type of cutting, the selection of the drill bit for cutting, and the cutting speed.

[0363] Aspect 19. According to the system of aspect 15, wherein the controller receives digital maps created by different robots, the controller is configured to determine a second reference frame of the second robot's sensors, and converts data from the digital map created by the first robot's sensors to the second reference frame of the second robot, thereby compensating for construction differences between the first robot and the second robot.

[0364] Aspect 20. A method comprising: capturing sensor data from a plurality of sensors located within an environment; performing an artificial intelligence algorithm based on the sensor data to locate features of the environment, the located features including one or more work locations within the environment; creating a digital map of the environment based on the located features of the environment; creating a scanning reference frame for each of the one or more work locations; generating an optimized path using the digital map for a robot to traverse the environment to avoid obstacles while navigating to the one or more work locations; navigating the robot to a first work location adjacent to the one or more work locations using the optimized path, the robot including a cutting tool, wherein the robot has a robot reference frame and the navigation steps are based on a comparison between the scanning reference frame and the robot reference frame; initiating a first cutting operation at the first work location; generating sensor data during the first cutting operation; and adjusting other planned cutting operations at the one or more work locations based on the generation steps.

[0365] Aspect 21. A method for re-establishing fluid communication between a main pipe and a branch pipe, the branch pipe extending from the main pipe after it has been lined with a liner, the method comprising: moving a robot supporting a cutting tool along the lined main pipe to a location near the branch pipe, the cutting tool being selectively extended from the robot to cut the liner; extending the cutting tool from the robot toward the liner at a location where the branch pipe has an opening on the opposite side of the liner leading to the main pipe; cutting the liner with the cutting tool to form an initial cut; scanning the opening after the initial cut; and cutting the liner a second time with the cutting tool using data from the scan of the opening to form a second cut.

[0366] Aspect 22. According to the method of aspect 21, wherein cutting the liner to form the initial cut comprises moving the cutting tool in one of a helical and zigzag motions.

[0367] Aspect 23. A system for re-establishing fluid communication between a main pipe and a branch pipe, the branch pipe extending from the main pipe after being lined with a liner, the system comprising a robot, a cutting tool supported by the robot and extendable from the robot to cut the liner, sensors supported on the robot and configured to sense the environment in which the robot is located, and a controller operatively connected to the robot to control the robot to move along the lined main pipe to a location near the branch pipe, the controller being configured to extend the cutting tool from the robot toward the liner at a location where the branch pipe has an opening on the opposite side of the liner leading to the main pipe, and to activate the cutting tool to cut the liner to form an initial cut, the controller being further configured to activate the sensors after the initial cut to scan the opening and to reactivate the cutting tool to cut the liner a second time using data from the scan of the opening to form a second cut.

[0368] Information technology, hardware and software

[0369] While examples of the systems envisioned herein have been described in conjunction with various computing devices / processors, their core principles can be applied to any computing device, processor, or system capable of implementing such a monitoring system. The various techniques described herein can be implemented in combination with hardware or software, or, where appropriate, with a combination of both. Therefore, methods and apparatus can take the form of program code (i.e., instructions) embodied in a specific tangible storage medium having a specific tangible physical structure. Examples of tangible storage media include floppy disks, optical disc read-only memory devices (CD-ROMs), digital multifunction disks or digital video disks (DVDs), hard disk drives, or any other tangible machine-readable storage medium (computer-readable storage medium). Therefore, a computer-readable storage medium is not a signal. A computer-readable storage medium is not a transient signal. Furthermore, a computer-readable storage medium is not a propagating signal. As described herein, a computer-readable storage medium is an article of manufacture. When the program code is loaded into and executed by a machine such as a computer, that machine becomes the device of the disclosed system. In the case of executing program code on a programmable computer, the computing device will typically include a processor, a processor-readable storage medium (including volatile or non-volatile memory or storage elements), at least one input device, and at least one output device. The program can be implemented in assembly or machine language, if desired. The language can be a compiled or interpreted language, and can be combined with hardware implementations.

[0370] The methods and apparatus associated with the disclosed system can also be practiced via communication embodied in program code, transmitted through some transmission medium, such as via wires or cables, via optical fibers or via any other form of transmission, via over-the-air (OTA) or via firmware over the air (FOTA). When the program code is received and loaded into a machine such as an erasable programmable read-only memory (EPROM), gate array, programmable logic device (PLD), client computer, etc., and executed by the machine, the machine becomes an apparatus for implementing the telecommunications as described herein. When implemented on a general-purpose processor, the program code, combined with the processor, is the sole apparatus providing operation to invoke the functionality of the telecommunications system.

[0371] While this disclosure has described cloud-based networks, it should be understood that the systems and methods disclosed herein can be deployed in both cellular networks and IT infrastructure, supporting current and future use cases. Furthermore, operators or third-party vendors can use this architecture to extend networks at the edge.

[0372] The methods and apparatus associated with the telecommunications systems described herein can also be practiced via communication embodied in program code, transmitted through some transmission medium, such as by wires or cables, by optical fibers, or via any other form of transmission. When the program code is received and loaded into a machine such as an EPROM, gate array, programmable logic device (PLD), client computer, etc., and executed by the machine, the machine becomes an apparatus for implementing the telecommunications as described herein. When implemented on a general-purpose processor, the program code, combined with the processor, is the sole apparatus providing operation to invoke the functionality of the telecommunications system.

[0373] In describing preferred methods, systems, or apparatuses of the subject matter of this disclosure as shown in the figures, specific terminology has been used for clarity. However, the claimed subject matter is not intended to be limited to the specific terminology chosen so far, and it should be understood that each specific element includes all technical equivalents that operate in a similar manner to achieve similar purposes. Furthermore, the word "or" is generally used sexually unless otherwise provided herein.

[0374] This written description uses examples to enable those skilled in the art to practice the claimed subject matter, including making and using any device or system and performing any incorporated methods. The patentable scope of the disclosed subject matter is defined by the claims and may include other examples that would occur to those skilled in the art (e.g., skipping steps, combining steps, or adding steps between the exemplary methods disclosed herein). Such other examples are intended to be within the scope of the claims if they have structural elements that differ from the literal language of the claims, or if they include equivalent structural elements that are not substantially different from the literal language of the claims.

Claims

1. A method for controlling the movement of a robot in an environment, wherein, The robot has multiple sensors associated with it, and the method includes: An initial scanning reference frame is determined for the robot, wherein the scanning reference frame is calculated based on an initial origin; The new origin is calculated based on readings from at least one of the sensors; and The initial scan reference frame is adjusted to determine a new scan reference frame based on the new origin. in, The adjustment steps are based on the difference between the initial scanning reference frame and the new scanning reference frame.

2. The method according to claim 1, wherein, The initial scanning reference frame for the robot includes: an initial scanning reference frame for each of the plurality of sensors based on the pose of each of the plurality of sensors, and wherein, The adjustment step includes: adjusting the initial scanning reference frame used for each of the plurality of sensors to a new scanning reference frame used for each of the plurality of sensors.

3. The method according to claim 2, further comprising: Calculate the correction factor, and in, The adjustment steps include: applying the correction factor to calculate the new scanning reference frame for each of the plurality of sensors.

4. The method according to claim 1, wherein, Modify the environment, and The step of calculating the new origin is initiated by modifying the environment.

5. The method according to claim 1, wherein, At least one of the sensors is an inertial measurement unit (IMU), and wherein, The IMU calculates the new origin based on its position within the environment.

6. The method according to claim 5, wherein, The IMU calculates the new origin while traversing the environment.

7. The method of claim 6, further comprising: The correction factor is calculated based on the difference between the initial origin and the new origin.

8. The method according to claim 7, wherein, The robot is moving within the modified environment, and The robot uses the correction factor to calculate the distance it will travel in the environment.

9. The method according to claim 8, wherein, The IMU was used to calculate the distance.

10. The method according to claim 8, wherein, The distance is calculated using a motor encoder.

11. The method of claim 7, further comprising: A table is used to compare the original distance in the environment with the modified distance in the modified environment based on the correction factor.

12. The method according to claim 7, wherein, The robot uses machine learning to gather and initiate steps.

13. The method according to claim 1, wherein, The robot is configured to: (1) Collect data from the multiple sensors, and (2) Based on the collection of data, initiate the adjustment steps.

14. A method for controlling the movement of a robot in a modified environment, wherein, The modified environment is based on the original environment, and the method includes: While the robot is traversing the modified environment, a scanning device is used to scan the modified environment; Monitor the position of the scanning device in the modified environment relative to a reference frame of the original environment; The reference frame of the scanning device is periodically adjusted relative to the reference frame of the original environment, depending on the position being monitored by the scanning device.

15. The method according to claim 13, wherein, The periodic adjustment of the reference frame includes: Adjustments are made based on the size difference between the original environment and the modified environment.

16. The method according to claim 15, wherein, The original environment was a pipe, and The modified environment is a liner placed inside the pipe.

17. A method for fusing sensor data associated with a robot in its environment, wherein, The robot has multiple sensors associated with it, and the method includes: Determine the initial scanning reference frame of the robot, wherein the scanning reference frame is based on the initial pose of the robot; An initial sensor reference frame is determined for each of the plurality of sensors, wherein the initial sensor reference frame is based on the corresponding initial attitude of each of the plurality of sensors; A new origin is established, which modifies the robot's initial pose and the corresponding initial pose of each of the plurality of sensors; An updated reference frame is calculated based on the difference between the robot's initial pose and its modified pose, and the difference between the corresponding initial pose of each of the plurality of sensors and the corresponding modified pose of each of the plurality of sensors; and Based on computational steps, sensor data is fused.

18. The method according to claim 17, wherein, The robot's initial frame of reference is based on the initial environment through which the robot is traversing, and The robot's updated frame of reference is based on the modified environment. in, The modified environment has at least one size metric that is different from the initial environment.

19. The method according to claim 18, wherein, The initial environment is a pipe, and The modified environment is a liner placed inside the pipe, and in, The different dimensional measures refer to the changing radius of the pipe.

Images