Pipeline robot positioning system and method based on visual inertial fusion

Abstract

The application discloses a pipeline robot positioning system and method based on visual inertial fusion. The method comprises the following steps: collecting pipeline environment images and pose data, offline training a neural radiance field prior model for representing the internal geometric structure and appearance distribution of the pipeline; collecting visual images and inertial data in real time and generating time-aligned multi-sensor synchronous data packets; adaptively triggering the projection of a coded structured light by a projector according to the pose estimation confidence ellipsoid volume, the number of ORB feature points and the feature point distribution entropy; inputting the image with the active illumination pattern and the inertial predicted rough pose into the neural radiance field prior model to generate a virtual feature point set and perform probabilistic association with real feature points; constructing a tight coupling graph optimization problem based on the comprehensive feature set and the inertial data to obtain the robot pose, and obtaining the positioning result through closed-loop detection and global graph optimization. The application can improve the positioning robustness and accuracy in low-texture, weak-light and repeated-texture pipeline environments.

Description

Technical Field

[0001] This invention relates to the field of autonomous positioning technology for pipeline robots, specifically to a positioning system and method for pipeline robots based on visual-inertial navigation fusion. Background Technology

[0002] Pipelines, underground utility tunnels, gas pipelines, water supply and drainage pipelines, and industrial transmission pipelines are typically enclosed or semi-enclosed spaces characterized by their narrow shape, lack of environmental visibility, difficulty in personnel access, and limited or unavailable wireless signals. To accomplish pipeline inspection, defect identification, maintenance, and digital modeling of pipeline networks, pipeline robots are usually required to autonomously move inside the pipeline, acquire images, perform 3D reconstruction, and spatial positioning.

[0003] Visual inertial odometry (VIO) estimates the position, attitude, velocity, and inertial sensor bias of a mobile robot by fusing camera images and inertial measurement unit (IMU) data, without relying on external positioning infrastructure. Typical VIO uses a tightly coupled optimization framework that models both the reprojection error of visual features and the pre-integration error of the IMU, achieving high positioning accuracy in well-lit, textured indoor and outdoor environments.

[0004] Existing patent CN110174136B discloses an intelligent inspection robot and method for underground pipelines. This method utilizes a robot equipped with several sensors that autonomously move within the underground pipeline space to collect multi-source data, including images, 3D laser point clouds, and depth images, and provides geospatial location references for the inspection data. This approach emphasizes multi-source sensor fusion and spatial labeling of pipeline inspection data.

[0005] Existing patent CN110766785B discloses a device and method for real-time positioning and three-dimensional reconstruction of underground pipelines. The device consists of a pipeline crawling robot, a processor, an RGB-D camera, and an inertial measurement unit. It collects data through the RGB-D camera and IMU, and uses methods such as visual feature tracking, IMU pre-integration, visual SFM, visual reprojection error, and IMU measurement residual to achieve real-time positioning and three-dimensional reconstruction of underground pipelines.

[0006] Existing patent CN112665582A discloses an underground pipeline detection system based on IMU and laser spot images. The system includes a flexible pipeline robot, an inertial measurement module, an odometer module, a laser image acquisition module, a GPS module, and a processing module. The laser image acquisition module includes a laser emitter, a CMOS camera, and an imaging screen. It extracts spot information by performing grayscale enhancement, filtering, and threshold segmentation on the laser spot image.

[0007] However, the aforementioned existing technologies still have the following shortcomings in the long-term positioning process of actual pipeline robots: Existing visual inertial localization schemes typically rely on natural texture features or RGB-D depth information. In long straight pipes, metal pipes, oil-covered pipes, and pipes in low light or complete darkness, the pipe wall surface often exhibits low texture, repetitive texture, high reflectivity, or partial occlusion, making it difficult for the visual front end to reliably extract a sufficient number of spatially distributed feature points. This leads to feature tracking failure, unstable scale constraints, and accumulated pose drift.

[0008] While existing multi-sensor pipeline inspection solutions can enhance data richness through sensors such as LiDAR, RGB-D cameras, odometers, IMUs, or GPS, they typically focus on multi-source data acquisition, 3D reconstruction, or labeling of inspection results, without establishing predictive, proactive sensing mechanisms to address the degradation process of visual observation. Especially when visual features are about to fail but not yet completely lost, existing solutions struggle to promptly assess the degradation state of the positioning system and enhance visual observation as needed.

[0009] Existing laser spot or active light source solutions are mostly used for image enhancement, ranging, or spot detection. They usually do not take into account the state uncertainty of visual inertial odometry, the number of image features, and the quality of feature spatial distribution to determine whether to enable the active light source. This can easily lead to problems such as increased power consumption due to continuous illumination, sensor overexposure, increased thermal noise, or failure to enhance observation in time at critical degradation moments.

[0010] Existing visual-inertial SLAM or pipe 3D reconstruction schemes generally rely on real feature points in the current image frame for data association and optimization. When there are insufficient real feature points, concentrated feature distribution, or repetitive pipe wall texture, even with the introduction of IMU pre-integration, pose divergence can only be alleviated in a short time, and it is difficult to compensate for the lack of visual observation from the overall geometric level of the scene.

[0011] Therefore, there is an urgent need for a pipeline robot localization system and method that can actively enhance visual observation in pipeline feature degradation environments and use learned geometric priors to supplement visual feature constraints, so as to improve the localization continuity, accuracy and robustness of pipeline robots in complex pipeline environments. Summary of the Invention

[0012] To address the above problems, this invention proposes a pipeline robot localization method based on visual-inertial fusion, the method comprising the following steps: Real-time images from the vision camera and inertial data from the inertial measurement unit are acquired to obtain multi-sensor synchronization data packets; The active projector is triggered to project coded structured light based on the state of the multi-sensor synchronization data packet, resulting in an image with an active illumination pattern. An image with an active lighting pattern and a coarse pose predicted by inertial data are input into a neural radiation field prior model to obtain a virtual feature point set. The virtual feature point set and inertial data are fused and optimized through tight coupling to obtain the pose, velocity and sensor bias of the pipeline robot. Loop closure detection and global graph optimization are performed based on pose and feature information to obtain a globally consistent pose trajectory. The localization result of the pipeline robot is obtained based on the pose and the globally consistent pose trajectory.

[0013] As a preferred embodiment of the pipeline robot localization method based on visual-inertial fusion described in this invention, the prior model for training neural radiation fields includes: The control system guides the pipeline robot to perform a spiral trajectory motion inside the pipeline, records the visual image sequence and inertial measurement data, and integrates the inertial measurement data to obtain a coarse pose sequence with accumulated error compared to each frame of visual image; The visual image sequence and the coarse pose sequence are input into the visual structure recovery process. Sparse three-dimensional point cloud is reconstructed through feature matching and triangulation to obtain the precise pose corresponding to each frame of the visual image sequence and generate precise image-pose-three-dimensional point correspondence data. By using accurate image-pose-3D point correspondence data, an improved neural radiation field prior model is trained. On the basis of standard volume rendering loss, a geometric consistency constraint loss provided by 3D point cloud is added, so that the neural radiation field prior model learns the internal distribution consistent with the real geometry of the pipe, and the trained neural radiation field prior model is obtained.

[0014] As a preferred embodiment of the pipeline robot localization method based on visual-inertial fusion described in this invention, the multi-sensor synchronization data packet includes: The vision camera acquires real-time pipeline images at a first frequency, and the inertial measurement unit acquires inertial data at a second frequency, stamping each frame of real-time pipeline image and each inertial data packet with a precise timestamp from the host clock. Based on the precise timestamp, inertial data packets from the inertial measurement unit within the time window before and after the corresponding time point of each frame of real-time pipeline image are associated and packaged with the real-time pipeline image; Temperature compensation and preliminary deviation correction are performed on the packaged inertial measurement unit inertial data, followed by distortion removal and histogram equalization to generate time-aligned multi-sensor synchronization data packets.

[0015] As a preferred embodiment of the pipeline robot localization method based on visual-inertial fusion described in this invention, wherein: the image with active illumination pattern includes, Based on the covariance propagation model of inertial measurement unit inertial data, the confidence ellipsoid volume of pose estimation within a fixed time period is predicted. The number of ORB feature points is extracted from the real-time pipeline image in the multi-sensor synchronization data packet, and the image entropy of the ORB feature point distribution is calculated. The confidence ellipsoid volume, ORB feature point quantity, and ORB feature point distribution image entropy of the pose estimation are input into a rule-based state machine. The state machine obtains discrete trigger levels based on the preset priority logic and the combined state of the confidence ellipsoid volume, ORB feature point quantity, and ORB feature point distribution image entropy. When the trigger level reaches the preset activation level, a trigger command is generated, which controls the active projector to emit coded structured light in the form of a pseudo-random speckle pattern within a preset short time. The trigger command controls the global shutter of the vision camera, aligning the exposure time with the emission time of the active projector, and exposing during the coded structured light illumination to obtain an image with an active illumination pattern.

[0016] As a preferred embodiment of the pipeline robot localization method based on visual-inertial fusion described in this invention, the pose, velocity, and sensor deviation of the pipeline robot include, The image with active illumination pattern and the rough pose predicted by the inertial measurement unit data in the multi-sensor synchronization data packet are input into the trained neural radiation field prior model. Through the inference of the neural radiation field prior model, virtual feature points with two-dimensional image coordinates and precise three-dimensional spatial coordinates are obtained, forming a set of virtual feature points. FAST corner points and BRIEF descriptors are extracted from images with active lighting patterns to form a set of real feature points. The virtual feature point set and the real feature point set are probabilistically correlated based on descriptor similarity and spatial distance to generate a comprehensive feature set of probabilistic data association. Using inertial measurement unit (IMU) inertial data, continuous measurements of linear acceleration and angular velocity are accumulated and encapsulated between adjacent image frames showing changes in visual features to form relative motion constraint terms. These relative motion constraint terms are based on the IMU measurements accumulated between adjacent time points. Based on relative motion constraints, this is a graph optimization problem that constructs visual reprojection error terms, inertial measurement unit pre-integration error terms, and prior 3D coordinate error terms of virtual feature points for a comprehensive feature set. The visual reprojection error term is used to constrain the pose of the pipeline robot and the three-dimensional coordinates of the feature points in the integrated feature set, so that the two-dimensional coordinates of its back projection are consistent with the image observation coordinates. Among them, the pre-integration error term of the inertial measurement unit is used to constrain the pose, velocity and sensor deviation of the pipeline robot at adjacent time points, satisfy the inertial kinematic relationship described by the relative motion constraint term, and transform the inertial data of the continuous inertial measurement unit into tightly coupled constraints connecting discrete pose nodes. The prior error term for the three-dimensional coordinates of virtual feature points is used to constrain the three-dimensional coordinates of the virtual feature point set, ensuring consistency with the prior geometric information obtained from the neural radiation field prior model; By solving the graph optimization problem, minimizing the visual reprojection error term, the inertial measurement unit pre-integration error term, and the prior error term of the three-dimensional coordinates of the virtual feature points, the pose, velocity, and sensor deviation of the pipeline robot are obtained.

[0017] As a preferred embodiment of the pipeline robot localization method based on visual-inertial fusion described in this invention, the globally consistent pose trajectory includes: Based on the pose and BRIEF descriptor in the comprehensive feature set of the pipeline robot, inverted index retrieval is performed to obtain loop detection candidate frames; Random sampling consistency verification is performed on the comprehensive feature set of the loop detection candidate frame and the current frame to obtain the relative pose transformation, and the relative pose transformation is added to the global pose graph as a closed loop constraint edge. The global pose graph with closed-loop constraint edges is optimized to obtain a globally consistent pose trajectory.

[0018] As a preferred embodiment of the pipeline robot localization method based on visual-inertial fusion described in this invention, the localization result of the pipeline robot includes: The pose of the pipeline robot is timestamped and aligned with the globally consistent pose trajectory to obtain the aligned pose sequence. The aligned pose sequence is then fused with the globally consistent pose trajectory to obtain the localization result of the pipeline robot.

[0019] Based on the above-mentioned visual-inertial fusion method for pipeline robot localization, a pipeline robot localization system based on visual-inertial fusion is provided, including the following modules and functions: The training module collects images and pose data of the pipeline environment, and trains the neural radiation field prior model offline based on the images and pose data of the pipeline environment. The acquisition module acquires real-time images from the vision camera and inertial data from the inertial measurement unit to obtain multi-sensor synchronous data packets. The triggering module triggers the active projector to project coded structured light based on the status of the multi-sensor synchronization data packet, thereby obtaining an image with an active illumination pattern. The fusion module inputs an image with an active lighting pattern and a coarse pose predicted by inertial data into a neural radiation field prior model to obtain a virtual feature point set. The virtual feature point set and inertial data are fused and optimized through tight coupling to obtain the pose, velocity and sensor bias of the pipeline robot. The localization module performs closed-loop detection and global graph optimization based on pose and feature information to obtain a globally consistent pose trajectory. Based on the pose and the globally consistent pose trajectory, the localization result of the pipeline robot is obtained.

[0020] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the pipeline robot localization method based on visual-inertial fusion as described in the present invention.

[0021] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the pipeline robot localization method based on visual-inertial fusion as described in the present invention.

[0022] Compared with the prior art, the beneficial effects of the present invention include: (1) This invention enables the system to learn the continuous geometry and appearance distribution inside the pipe by training the neural radiation field prior model offline, and provides scene prior constraints for visual inertial odometry during real-time positioning.

[0023] (2) This invention does not continuously project an active light source, but adaptively triggers coded structured light based on the uncertainty of pose estimation, the number of feature points and the quality of the spatial distribution of feature points. It can make predictive interventions before visual observation degrades, taking into account both positioning robustness and energy consumption control.

[0024] (3) The present invention introduces a matching high-contrast visual texture to the inner wall of a low-texture or dark pipe by using coded structured light in the form of pseudo-random speckle pattern, thereby increasing the number of real feature points extracted and the uniformity of their distribution, and reducing the probability of visual front-end tracking failure.

[0025] (4) The present invention generates virtual feature points with two-dimensional image coordinates and three-dimensional spatial coordinates through a prior model of neural radiation field, so that the system can still obtain two-dimensional-three-dimensional geometric constraints that can be used for localization optimization when natural features are insufficient.

[0026] (5) The present invention probabilistically associates virtual feature points with real feature points, which can use prior geometric information to filter and enhance real observations, and improve the reliability of data association under the conditions of repeated pipe textures, local occlusion and illumination changes. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] Figure 1 This is a flowchart of a pipeline robot localization method based on visual-inertial fusion.

[0029] Figure 2 This is a schematic diagram of a pipeline robot positioning system based on visual-inertial fusion. Detailed Implementation

[0030] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0031] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0032] Secondly, the term "one embodiment" or "example" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. The appearance of an embodiment in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that mutually excludes other embodiments.

[0033] Example 1 Reference Figure 1 and Figure 2 This is one embodiment of the present invention, which provides a pipeline robot localization method based on visual-inertial fusion, including the following steps: S1. Collect images and pose data of the pipeline environment, and train the neural radiation field prior model offline based on the images and pose data of the pipeline environment.

[0034] S1.1 Control the pipeline robot to perform a spiral trajectory motion inside the pipeline, record the visual image sequence and inertial measurement data, and integrate from the inertial measurement data to obtain a rough pose sequence with cumulative error compared to each frame of visual image.

[0035] Furthermore, after the pipeline robot enters the pipeline, the control unit drives the robot to move along the pipeline axis in a spiral trajectory, so that the field of view of the vision camera covers the inner circumference of the pipeline from multiple angles. During this movement, the vision camera continuously acquires images of the inner wall of the pipeline at a preset frequency to form a sequence of visual images. The inertial measurement unit simultaneously acquires angular velocity and linear acceleration at a higher frequency to form inertial measurement data. The inertial measurement data is numerically integrated to obtain the relative displacement and rotation of the robot between adjacent visual image acquisition moments. The relative motion is accumulated to obtain a coarse pose sequence that is temporally aligned with each frame of visual image. This coarse pose sequence has errors due to the zero bias of the inertial measurement unit and the accumulation of integration.

[0036] S1.2 Input the visual image sequence and the coarse pose sequence into the visual structure recovery process, and reconstruct a sparse three-dimensional point cloud through feature matching and triangulation to obtain the precise pose corresponding to each frame of the visual image sequence, and generate precise image-pose-three-dimensional point correspondence data.

[0037] Furthermore, feature point detection and descriptor matching are performed between adjacent frames of the visual image sequence to establish cross-frame trajectories of feature points. Using the initial relative pose provided by the coarse pose sequence as a constraint, the trajectories of successfully matched feature points are triangulated to reconstruct the positions of feature points in three-dimensional space, forming a sparse three-dimensional point cloud. Based on the three-dimensional coordinates of the sparse three-dimensional point cloud and the two-dimensional observations of feature points in the visual image sequence, bundle adjustment is used to jointly optimize the precise pose of the visual image sequence and the three-dimensional coordinates of the sparse three-dimensional point cloud, minimizing the reprojection error of all feature points. Finally, the precise pose corresponding to each frame of the visual image sequence is obtained, generating precise image-pose-three-dimensional point correspondence data. Each set of data is clearly associated with a visual image, its optimized precise pose, and the three-dimensional coordinates of feature points in the sparse three-dimensional point cloud visible in that image.

[0038] S1.3. Using precise image-pose-3D point correspondence data, train an improved neural radiation field prior model. On the basis of standard volume rendering loss, add geometric consistency constraint loss provided by 3D point cloud, so that the neural radiation field prior model learns the internal distribution consistent with the real geometry of the pipe, and obtain the trained neural radiation field prior model.

[0039] Furthermore, an improved neural radiation field prior model is trained using precise image-pose-3D point correspondence data. During training, data samples are extracted from the image-pose-3D point correspondence data, including a visual image, its precise pose, and the corresponding 3D point coordinates. The camera light determined by the precise pose passes through the scene, and the neural radiation field prior model is consulted to obtain the predicted color and density. The predicted image is synthesized through volume rendering, and the color difference between the predicted image and the real visual image is used as the standard volume rendering loss. The spatial positions of the sampling points along the light rays are extracted from the neural radiation field prior model, and the distances from these positions to the nearest points in the corresponding 3D point cloud provided by the image-pose-3D point correspondence data are calculated to construct the geometric consistency constraint loss. The standard volume rendering loss and the geometric consistency constraint loss are weighted to obtain the total loss. The parameters of the neural radiation field prior model are updated through the backpropagation algorithm. After multiple rounds of iterative training on all image-pose-3D point correspondence data, the parameters of the neural radiation field prior model are optimized, so that the scene geometry and appearance distribution learned by the model are consistent with the real geometry of the pipeline, resulting in the trained neural radiation field prior model.

[0040] S2. Acquire real-time images from the vision camera and inertial data from the inertial measurement unit to obtain a multi-sensor synchronization data packet.

[0041] S2.1 The vision camera acquires real-time pipeline images at a first frequency, and the inertial measurement unit acquires inertial data at a second frequency, stamping each frame of real-time pipeline image and each inertial data packet of the inertial measurement unit with a precise timestamp from the host clock.

[0042] Furthermore, the vision camera periodically acquires images of the inner wall of the pipe at a preset first frequency, outputting frames of real-time pipe images. The inertial measurement unit acquires three-dimensional angular velocity and three-dimensional linear acceleration at a preset second frequency, outputting inertial measurement unit inertial data packets of the measured values. When the real-time pipe image is read from the image sensor of the vision camera, the current host clock time is recorded as the precise timestamp of that frame of real-time pipe image. When the inertial measurement unit inertial data packet is read from the interface of the inertial measurement unit, the current host clock time is recorded as the precise timestamp of that inertial measurement unit inertial data packet, thus marking the precise acquisition time for each frame of real-time pipe image and each inertial measurement unit inertial data packet.

[0043] S2.2. Based on the precise timestamp, associate and package the inertial data packets of the inertial measurement unit within the time window before and after the corresponding time point of each frame of real-time pipeline image with the real-time pipeline image.

[0044] Furthermore, based on the recorded precise timestamps, for each frame of real-time pipeline image, an equal-length time window is defined on the time axis, both forward and backward. Inertial measurement unit (IMU) data packets whose precise timestamps fall within this time window are filtered out. These filtered IMU data packets are then combined with the real-time pipeline image of that frame to form a data packet. This data packet contains the visual observation at that moment as well as the IMU observations in the adjacent time periods, thus completing the association and packaging of the real-time pipeline image and the IMU data packets.

[0045] S2.3 Perform temperature compensation and preliminary deviation correction on the packaged inertial measurement unit inertial data, and perform distortion removal and histogram equalization processing to generate time-aligned multi-sensor synchronization data packets.

[0046] Furthermore, based on the temperature sensor readings of the inertial measurement unit (IMU), a preset temperature-zero bias mapping table is consulted to perform temperature compensation on the angular velocity and acceleration measurements in the IMU's inertial data. Using pre-calibrated gyroscope and accelerometer zero biases, preliminary deviation correction is performed on the compensated measurements. For the packaged real-time pipeline image, the image is dedistorted using the pre-calibrated intrinsic parameter matrix and distortion coefficients of the vision camera to correct the radial and tangential distortion of the lens. Then, histogram equalization is performed on the dedistorted image to enhance the image contrast. After temperature compensation and preliminary deviation correction of the IMU's inertial data, as well as distortion correction and histogram equalization of the real-time pipeline image, a multi-sensor synchronization data packet is generated.

[0047] S3. Based on the status of the multi-sensor synchronization data packet, the active projector is triggered to project coded structured light to obtain an image with an active illumination pattern.

[0048] S3.1. Based on the covariance propagation model of inertial measurement unit inertial data, predict the confidence ellipsoid volume of pose estimation within a fixed time period in the future, extract the number of ORB feature points from the real-time pipeline image in the multi-sensor synchronization data packet, and calculate the image entropy of the ORB feature point distribution.

[0049] Furthermore, inertial measurement unit (IMU) inertial data is parsed from the multi-sensor synchronization data packet. Using the current error covariance matrix and the IMU's state transition Jacobian matrix, covariance propagation calculation is performed over a fixed-length future time window to obtain the pose covariance matrix for future moments. The determinant of this covariance matrix is ​​calculated and substituted into the three-dimensional ellipsoid volume formula to obtain the confidence ellipsoid volume representing the uncertainty of future pose. Real-time pipeline images are acquired from the multi-sensor synchronization data packet, and ORB feature detection and descriptor calculation are performed on the images to statistically analyze the detected ORB features. The total number of B feature points is used as the ORB feature point count. The real-time pipeline image is spatially divided into K non-overlapping grid regions. The number of ORB feature points falling into each grid region is counted, and the proportion of the ORB feature points in each grid region to the total number of ORB feature points is calculated to obtain the probability of that region. Finally, the entropy value of the probability distribution of all regions is calculated according to the information entropy formula to obtain the image entropy of the ORB feature point distribution. This generates three quantitative indicators for evaluating the health of the visual-inertial state: confidence ellipsoid volume, ORB feature point count, and image entropy of the ORB feature point distribution.

[0050] Specifically, proactive perception triggers are constructed from two orthogonal dimensions: the inherent uncertainty of state estimation and the quality of external information in visual perception, enabling predictive rather than reactive resource scheduling. Calculating the confidence ellipsoid volume through covariance propagation essentially quantifies the expected value of future short-term pure inertial navigation errors, making trigger decisions predictable and allowing intervention before actual visual tracking failure. Calculating the image entropy of ORB feature point distribution extends the evaluation of visual information from a purely quantitative dimension to a spatial distribution structure dimension, effectively identifying quality issues such as feature point clustering and uneven distribution. These three indicators, from kinematic, observation quantity, and observation geometry perspectives, respectively, construct an evaluation system that comprehensively reflects the robustness degradation of visual inertial odometry.

[0051] The expression for the confidence ellipsoid volume is: ; in, To determine the volume of the ellipsoid, Let be the determinant of the covariance matrix. To predict future time points, Let be the covariance matrix at future times; ; in, Based on the current time, Let Jacobian be the state transition matrix. For inertial prediction duration, For propagation noise covariance; The image entropy expression for the ORB feature point distribution is: ; in, The information entropy of the feature distribution, The total number of image regions to be divided. For image region indexing, The probability of feature point region distribution; S3.2 Input the confidence ellipsoid volume, ORB feature point quantity, and ORB feature point distribution image entropy of the pose estimation into a rule-based state machine. The state machine obtains discrete trigger levels based on the preset priority logic and the combined state of the confidence ellipsoid volume, ORB feature point quantity, and ORB feature point distribution image entropy.

[0052] Furthermore, the confidence ellipsoid volume, the number of ORB feature points, and the image entropy of the ORB feature point distribution are input into a finite state machine with predefined logical rules. The state machine internally defines multiple discrete trigger levels and transition conditions from the current state to the next state, outputting a specific trigger level. These transition conditions are written with predefined priority logic. For example, a high-level transition condition can be set as follows: if the confidence ellipsoid volume is greater than a volume threshold, the state machine immediately transitions to the highest trigger level state, regardless of the values ​​of the number of ORB feature points and the image entropy of the ORB feature point distribution. If the confidence ellipsoid volume does not exceed the volume threshold, the state machine further evaluates secondary conditions, such as whether the number of ORB feature points is less than a quantity threshold and whether the image entropy of the ORB feature point distribution is less than an entropy threshold. If both conditions are met, the state machine transitions to a medium-level trigger level state; otherwise, it remains in a low or no-trigger-level state. Based on the input triplet values, the state machine iterates through and matches the predefined transition conditions, outputting a discrete trigger level corresponding to the current comprehensive state evaluation result.

[0053] Specifically, finite state machines are used for decision-making, simulating human decision-making wisdom when dealing with multi-objective conflicts, and prioritizing responses to the most pressing risks. The pre-defined priority logic of the state machine, such as unconditionally prioritizing excessive inertial uncertainty over insufficient visual quality, ensures at the algorithmic level that, when resources are limited, inertial divergence risks that could lead to the collapse of the entire system are contained first. The nonlinear condition-triggered decision-making paradigm can more precisely characterize the real needs of the system under different degradation modes than fixed thresholds or weighted summations, avoiding wasting resources on secondary issues or being slow to respond to critical risks. Its implementation is based on the modeling theory of discrete event systems. S3.3 When the trigger level reaches the preset activation level, a trigger command is generated. The trigger command controls the active projector to emit coded structured light in the form of a pseudo-random speckle pattern within a preset short time. Furthermore, when the discrete trigger levels reach or exceed the preset activation level, a digital electrical pulse signal is generated as a trigger command. This trigger command is sent to the driving circuit of the active projector. Upon receiving the trigger command, the driving circuit immediately controls the light source of the active projector to emit a preset pseudo-random speckle pattern within a very short preset duration. This pattern is the coded structured light. The trigger command generation logic ensures that the active projector is activated only at the specific moment when it is determined that enhanced perception is needed, and it operates in an instantaneous pulse mode.

[0054] Specifically, pseudo-random speckle patterns are used as the coded structured light, transforming the projector from a uniform illumination source into a spatial information encoder. The high spatial frequency and non-repetitive characteristics of the speckle pattern provide an easily matched optical fingerprint for the pipe's inner wall, which lacks natural texture, solving the data association problem in feature-scarce environments. Meanwhile, the preset short-duration pulsed emission mode enables on-demand energy management, minimizing average power consumption while avoiding thermal noise, sensor oversaturation, and continuous environmental interference caused by constantly lit light sources.

[0055] S3.4 The trigger command controls the global shutter of the vision camera to align the exposure time with the emission time of the active projector, and exposes during the coded structured light illumination to obtain an image with an active illumination pattern.

[0056] Furthermore, the trigger command is sent to the shutter control unit of the vision camera simultaneously with the active projector. Based on the trigger command, the shutter control unit precisely controls the opening time and duration of the global shutter of the vision camera, ensuring that the entire exposure period of the global shutter falls entirely within the emission period of the coded structured light emitted by the active projector. Through this synchronous control, the vision camera only exposes and captures images during the period when the coded structured light is uniformly illuminating the inner wall of the pipe, capturing an image with a high-contrast active illumination pattern formed mainly by the illumination of the coded structured light.

[0057] Specifically, the potential of active sensing is maximized by strictly separating signal and noise in the time domain to obtain the highest quality observation images. An optical sampling window is constructed by precisely aligning the global shutter exposure time window with the projection light pulse time window. Within this window, the camera sensor receives and integrates almost exclusively the coded light signal from the active projector, effectively treating temporally asynchronous ambient stray light as noise and significantly suppressing it. This greatly improves the signal-to-noise ratio of the image, and the extremely short synchronous exposure time is equivalent to freezing high-speed motion on the time axis, eliminating motion blur.

[0058] S4. Input the image with active lighting pattern and the coarse pose predicted by inertial data into the neural radiation field prior model to obtain a virtual feature point set. Fuse the virtual feature point set and inertial data through tight coupling optimization to obtain the pose, velocity and sensor bias of the pipeline robot.

[0059] S4.1 Input the image with active illumination pattern and the coarse pose predicted by the inertial measurement unit in the multi-sensor synchronization data packet into the trained neural radiation field prior model. Through the inference of the neural radiation field prior model, virtual feature points of two-dimensional image coordinates and precise three-dimensional spatial coordinates are obtained, forming a virtual feature point set.

[0060] Furthermore, the coarse pose of the pipeline robot at the current moment is calculated from the inertial measurement unit inertial data in the multi-sensor synchronization data packet through inertial navigation. This coarse pose includes the coarse position and coarse attitude. The image with the active lighting pattern and this coarse pose are input into the trained neural radiation field prior model. The neural radiation field prior model uses its encoder to extract the depth features of the image with the active lighting pattern. Combined with the coarse pose information, the model performs three-dimensional spatial coordinate query and two-dimensional image coordinate regression in the decoder network. The network outputs a vector containing two-dimensional pixel coordinates and corresponding precise three-dimensional spatial coordinates. Each vector represents a virtual feature point, and the output virtual feature points constitute a virtual feature point set.

[0061] Specifically, the neural radiation field prior model is transformed into an online, scene-understanding visual prior generator. Images with active lighting patterns ensure the richness and quality of the input information. The input is a coarse pose predicted from inertial data, providing the model with the viewpoint context of the current observation. This allows the model to perform scene queries based on the current view and approximate location, inferring feature points with stable 3D coordinates that should exist in the pipe geometry from the current perspective. Generating visual observations with precise 3D geometric meaning solves the feature source problem.

[0062] S4.2 Extract FAST corner points and BRIEF descriptors from images with active lighting patterns to form a set of real feature points. Probabilistically associate the virtual feature point set with the real feature point set based on descriptor similarity and spatial distance to generate a comprehensive feature set of probabilistic data association.

[0063] Furthermore, the FAST corner detection algorithm is run on an image with an active illumination pattern. A BRIEF descriptor is calculated for each detected corner point, forming a set of real feature points consisting of two-dimensional coordinates and descriptors. The two-dimensional image coordinates of each virtual feature point are read from the set of virtual feature points. Based on the neural radiation field prior model, the intermediate layer features of the network used to generate the virtual feature points are generated, or a descriptor is calculated for them through a lightweight network. Hamming distance or cosine similarity is calculated for each pair of descriptors in the set of virtual feature points and the set of real feature points. The two-dimensional image Euclidean distance between the corresponding feature point pairs is used. A joint probability model is constructed based on the descriptor similarity and spatial distance, such as through Mahalanobis distance or Gaussian mixture model, to evaluate the probability that each pair of virtual and real feature points is a correct match. A correlation is established for feature point pairs with probabilities higher than a threshold. All the correlated feature points are then included in the data list to form a comprehensive feature set of probabilistic data correlation.

[0064] Specifically, the strong 3D prior information inherent in virtual feature points serves as anchor points to guide the matching of real features. The 2D coordinates and descriptors of these virtual feature points are generated by a prior model, inherently containing an understanding of the pipeline structure; therefore, their position and appearance descriptions exhibit higher predictive consistency in space. By probabilistically associating descriptor similarity with spatial distance, virtual feature points carrying prior knowledge are used to vote on or verify the reliability of real feature points. This improves the association accuracy under complex conditions such as texture repetition and dynamic occlusion. S4.3. Using inertial measurement unit (IMU) inertial data, the continuous measurements of linear acceleration and angular velocity are accumulated and encapsulated between adjacent image frames with changing visual features to form relative motion constraint terms. The relative motion constraint terms are based on the IMU measurements accumulated between adjacent time points.

[0065] Furthermore, inertial data from all inertial measurement units (IMUs) between key frame moments of visual feature changes from the multi-sensor synchronization data packet are obtained. The angular velocity measurements from these IMUs are subtracted from the currently estimated gyroscope zero bias, and the acceleration measurements are subtracted from the currently estimated accelerometer zero bias and gravity is compensated. The corrected angular velocity is continuously integrated on the manifold to obtain the relative rotation change between the two key frames. The corrected acceleration is integrated twice in the rotated coordinate system to obtain the relative velocity change and relative position change. Throughout the integration process, median integration or the Runge-Kutta method is used to approximate continuous-time motion, and the covariance of measurement noise is propagated synchronously. Finally, a relative motion constraint term containing relative rotation, relative velocity, relative displacement, and their uncertainties is encapsulated.

[0066] Specifically, pre-integration theory is used to construct tightly coupled inertial constraint terms, decoupling inertial data processing from state estimation. By pre-integrating IMU data between two keyframes, a relative motion constraint is generated that is independent of the absolute pose of the two keyframes and depends only on the IMU measurements between the two frames. This relative motion constraint term is a well-encapsulated observation with accurate noise characteristics. In subsequent optimization, when the keyframe pose is iteratively adjusted, it is not necessary to repeatedly integrate a large amount of original IMU data; only this invariant pre-integration constraint term needs to be used, improving computational efficiency and providing a physically kinematically correct strong constraint edge connecting the two pose nodes.

[0067] S4.4. Based on relative motion constraints, construct a graph optimization problem for the visual reprojection error term, the inertial measurement unit pre-integration error term, and the prior error term of the three-dimensional coordinates of virtual feature points for the comprehensive feature set.

[0068] Furthermore, a nonlinear least squares optimization problem is established, with the objective function being the sum of squares of three error terms. Each error term is composed of its residual vector and the corresponding information matrix weighted together. The visual reprojection error term is based on the three-dimensional coordinates of each feature point in the comprehensive feature set and the corresponding two-dimensional observation coordinates of the image, as well as the camera pose to be optimized. The inertial measurement unit pre-integration error term is based on the relative motion constraint term and the state to be optimized of the two connected keyframes. The virtual feature point three-dimensional coordinate prior error term is calculated based on the optimized three-dimensional coordinates of the virtual feature points in the comprehensive feature set and the precise three-dimensional spatial coordinate prior value of the point output by the neural radiation field prior model. These three error terms are added together to construct the graph optimization problem.

[0069] Specifically, a third type of constraint source scene geometric prior is introduced and formalized as an optimizable error term. The addition of the 3D coordinate prior error term for virtual feature points ensures that state estimation not only satisfies multi-view geometric consistency and inertial dynamic consistency, but also conforms to the consistency of the macroscopic scene structure learned offline. This is equivalent to adding a shape-preserving regularization term to the optimization problem, enhancing its observability. Especially in long, straight, textureless pipelines, where visual constraints are extremely weak, the prior error term can provide crucial directional and shape constraints.

[0070] S4.5 The visual reprojection error term is used to constrain the pose of the pipeline robot and the three-dimensional coordinates of the feature points in the comprehensive feature set, so that the two-dimensional coordinates of its back projection are consistent with the image observation coordinates.

[0071] Furthermore, for each feature point in the comprehensive feature set, whether it is a real feature point or a virtual feature point, its current optimized 3D coordinates are taken. Using the camera pose of the corresponding keyframe, i.e., the rotation matrix and translation vector, the 3D coordinates are transformed into the camera coordinate system. The coordinates are then projected onto the image plane through the camera intrinsic model to obtain the theoretical 2D projection coordinates. The difference between the theoretical projection coordinates and the actual 2D coordinates observed by the feature point in the image is taken to obtain the 2D residual vector. The square of the magnitude of the residual vector is the visual reprojection error contributed by the point. The sum of the visual reprojection errors of all feature points constitutes the visual reprojection error term.

[0072] Specifically, the semantics and scope of the visual reprojection error term are expanded. This error term is redefined as a bridge connecting pose and composite feature points, where the composite feature points include virtual feature points. This brings about a fundamental change: the 3D coordinates of the virtual feature points are themselves used as optimization variables, but they have strong initial values ​​from the prior model. Minimizing the reprojection error that includes virtual feature points is actually jointly optimizing pose and scene geometry, and forcing them to be consistent with the view predicted by the prior model. The 3D coordinates given by the prior model are not regarded as fixed truth values, but rather used as the starting point for optimization, allowing for fine-tuning under multi-view geometric constraints. This achieves closed-loop optimization of prior knowledge and multi-view observation: the prior provides the initial geometry, and the reprojection error utilizes multi-view... Figure 1 Consistency refines the geometry, which in turn generates more accurate pose constraints. This improves adaptability to environmental changes and enhances the consistency of the overall geometric reconstruction.

[0073] S4.6, where the inertial measurement unit pre-integration error term is used to constrain the pose, velocity and sensor deviation of the pipeline robot at adjacent time points, satisfying the inertial kinematic relationship described by the relative motion constraint term, and transforming the inertial data of continuous inertial measurement units into tightly coupled constraints connecting discrete pose nodes.

[0074] Furthermore, the calculation of the inertial measurement unit (IMU) pre-integration error term requires the optimized states of two adjacent keyframes, including the pose, velocity, and sensor bias of the IMU in these two keyframes. Using these states, the theoretical relative motion change from the initial state to the final state is calculated through the IMU dynamics model. This theoretical relative motion is compared with the relative motion constraint term obtained by pre-integration of pure IMU data. The rotational residual is obtained on the manifold, and the velocity and position residuals are obtained in Euclidean space, which constitute the IMU pre-integration error term.

[0075] Specifically, the continuous IMU dynamics model is discretized and represented as an edge in graph optimization. This edge connects two pose nodes, two velocity nodes, and a shared sensor bias node. The error term directly measures the inconsistency between the relative motion predicted by the state nodes and the relative motion observed by the IMU. By minimizing this error, the optimization process forces changes between adjacent state nodes to conform to the physical motion observed by the IMU, which is not only data fusion but also a model constraint. This effectively complements the high-frequency, short-term accurate but drift-prone characteristics of the IMU with the low-frequency, absolute but potentially unreliable characteristics of vision, suppressing the cumulative drift of vision and providing observability for estimating the time-varying sensor bias of the IMU, thus achieving the fundamental goal of six-degree-of-freedom estimation.

[0076] S4.7 The prior error term for the three-dimensional coordinates of virtual feature points is used to constrain the three-dimensional coordinates of the virtual feature point set, ensuring consistency with the prior geometric information obtained from the neural radiation field prior model.

[0077] Furthermore, the prior error term for the three-dimensional coordinates of virtual feature points is evaluated for each virtual feature point in the comprehensive feature set. The current estimated value of the three-dimensional coordinates of the virtual feature point during the optimization process is read, and the precise three-dimensional spatial coordinates output by the neural radiation field prior model for the virtual feature point are read. This precise three-dimensional spatial coordinates are used as the prior value. The three-dimensional Euclidean distance residual between the current optimized estimate and the prior value is the square of the prior error contributed by the virtual feature point. The sum of the prior errors of all virtual feature points constitutes the prior error term for the three-dimensional coordinates of virtual feature points.

[0078] Specifically, a learning-based scene geometry prior constraint is introduced, adding an anchor point to the 3D coordinates of each virtual feature point to pull it towards the reasonable position predicted by the neural radiation field prior model. This essentially injects the knowledge of the pipelined continuous scene representation learned offline into the online optimization in the form of point cloud priors. A structured regularization is provided to prevent virtual feature points from moving to physically impossible locations due to erroneous observations or insufficient information.

[0079] S4.8 By solving the graph optimization problem, the visual reprojection error term, the inertial measurement unit pre-integration error term, and the prior error term of the three-dimensional coordinates of the virtual feature points are minimized to obtain the pose of the pipeline robot, the speed of the pipeline robot, and the sensor deviation of the pipeline robot.

[0080] Furthermore, the constructed graph optimization problem is input into a nonlinear optimization solver, such as a library of Gauss-Newton or Levenberg-Marquardt methods. The optimization solver uses the initial pose, initial velocity, initial sensor bias, and initial 3D coordinates of feature points in the integrated feature set of the pipeline robot as the initial values ​​of the optimization variables. The optimization objective is to minimize the weighted sum of the visual reprojection error term, the inertial measurement unit pre-integration error term, and the prior error term of the 3D coordinates of the virtual feature points. The solution is iterative. In each iteration, the solver calculates the Jacobian matrix of all error terms with respect to all optimization variables, constructs the incremental equation, solves the state increment, and updates the optimization variables until the convergence condition is met. The output is the optimal estimate of the pose, velocity, and sensor bias of the pipeline robot that minimizes the overall objective function.

[0081] Specifically, within the joint Bayesian inference framework for heterogeneous error sources, visual observation, inertial observation, and prior knowledge are jointly modeled as conditional probability distributions for hidden states (pose, velocity, bias, map). By minimizing the weighted sum of the three errors, the posterior probability of the state is essentially maximized, and the confidence allocation of each constraint source is automatically achieved through the information matrix. The information matrix of the visual reprojection error term reflects the uncertainty of feature point localization; the information matrix of the inertial measurement unit pre-integration error term encapsulates the noise characteristics of the IMU; and the information matrix of the virtual feature point prior error term expresses the confidence of the prior model. The optimization solver propagates and fuses these uncertainties through the Jacobian matrix, outputting the statistically optimal state estimate. This framework enables optimal adaptive resource allocation by trusting vision when texture is rich, trusting the IMU during rapid movement, and relying on prior knowledge when features are missing, achieving accurate localization even in the extremely variable environment of pipelines.

[0082] The expression for the visual reprojection error term is: ; in, This is due to visual reprojection error. Let be the transformation matrix from the camera to the world coordinate system. These are the 3D coordinates of the feature points in the world coordinate system. These are the 2D pixel coordinates of the feature points observed in the image. The expression for the IMU pre-integration error term is: ; in, For IMU pre-integration error, From the start time By the end time IMU pre-integrated measurements, Let be the state vector at the initial moment. The state vector at the end time. At the starting time, The end time; The expression for the prior error term of virtual feature points is: ; in, This represents the prior error of the virtual feature point coordinates.

[0083] S5. Based on pose and feature information, perform closed-loop detection and global graph optimization to obtain a globally consistent pose trajectory. Based on the pose and the globally consistent pose trajectory, obtain the localization result of the pipeline robot.

[0084] S5.1. Based on the pose and BRIEF descriptor in the comprehensive feature set of the pipeline robot, an inverted index retrieval is performed to obtain loop detection candidate frames.

[0085] Furthermore, position information is extracted from the pose of the pipeline robot, and combined with all BRIEF descriptors in the comprehensive feature set, a visual bag-of-words model database is constructed. The database stores the visual bag-of-words vectors of historical keyframes and their corresponding poses. When a new pose of the pipeline robot and the comprehensive feature set are generated, the BRIEF descriptor of the current comprehensive feature set and the visual bag-of-words vector of the current frame are extracted. This visual bag-of-words vector is compared with the historical bag-of-words vectors in the visual bag-of-words model database. Using an inverted index structure, multiple historical keyframes with a similarity exceeding a preset threshold with the current bag-of-words vector are quickly retrieved. These historical keyframes are used as loop detection candidate frames, quickly narrowing the loop closure search range and efficiently filtering out candidate frames that may close loops from a large number of historical frames.

[0086] S5.2 Perform random sampling consistency verification on the comprehensive feature set of the loop detection candidate frame and the current frame, obtain the relative pose transformation, and add the relative pose transformation as a closed loop constraint edge to the global pose graph.

[0087] Furthermore, for each loop detection candidate frame, the historical comprehensive feature set corresponding to the loop detection candidate frame and the comprehensive feature set of the current frame are obtained. Brute-force matching or fast approximate nearest neighbor matching is performed on the BRIEF descriptors in the two comprehensive feature sets to obtain a preliminary set of feature matching pairs. The random sampling consensus algorithm is used to process this preliminary set of matching pairs. The random sampling consensus algorithm randomly selects the minimum sample set to estimate a relative pose transformation model. Under the current model, the projection error of all matching pairs is calculated, and the number of inliers that meet the error threshold is counted. After multiple iterations, the relative pose transformation model corresponding to the maximum number of inliers is selected as the optimal estimate. The relative pose transformation of this optimal estimate, i.e., the rotation matrix and translation vector, is obtained. If the inlier ratio exceeds the quality threshold, the loop closure is considered valid. This valid relative pose transformation is added as a constraint edge to the global pose graph. This constraint edge connects the pose node of the current frame with the pose node of the loop detection candidate frame.

[0088] S5.3. Perform pose graph optimization on the global pose graph with added closed-loop constraint edges to obtain a globally consistent pose trajectory.

[0089] Furthermore, the global pose graph is a graph structure containing nodes and edges. Nodes represent the poses of the pipeline robot across all historical keyframes. Edges include two types: sequential edges generated by visual inertial odometry, connecting pose nodes at adjacent time points, with constraints defined by inertial measurement unit pre-integration and visual constraints; and closed-loop constraint edges detected and added, connecting non-adjacent pose nodes. A nonlinear least-squares optimization problem is constructed for all nodes and edges. The optimization objective is to minimize the constraint errors of all edges in the graph, minimize the error between the relative pose corresponding to the sequential edge and the relative pose calculated from the node pose, and minimize the error between the relative pose transformation corresponding to the closed-loop constraint edge and the relative pose calculated from the node pose. This optimization problem is solved using the Gauss-Newton method or the Levenberg-Marquardt method, iteratively updating the position and orientation of all pose nodes to achieve global consistency of the entire pose graph while satisfying all constraints. The trajectory of all pose nodes output after optimization is the globally consistent pose trajectory.

[0090] S5.4. The pose of the pipeline robot is timestamped with the globally consistent pose trajectory to obtain the aligned pose sequence. The aligned pose sequence is then fused with the globally consistent pose trajectory to obtain the localization result of the pipeline robot.

[0091] Furthermore, the pose of the pipeline robot is output in real time from the front end of the visual inertial odometry system. This pose is timestamped. A series of globally optimized poses with timestamped values ​​are obtained from the globally consistent pose trajectory. These two sets of pose data are aligned based on the timestamps. For each timestamp of the pipeline robot's pose output in real time, the closest global pose in time is found in the globally consistent pose trajectory through linear interpolation. A global pose sequence that is strictly aligned with the real-time pose sequence in time is generated. The aligned real-time pipeline robot pose sequence and the aligned global pose sequence are fused. The fusion method can be a sliding window filter or a weighted average with certain weights. For example, in the time period after loop closure is detected and optimization is completed, the weight of the global pose sequence is increased. The fused pose is output as the final localization result of the pipeline robot at each moment.

[0092] Example 2 A pipeline robot localization system based on vision-inertial navigation fusion includes: The training module collects images and pose data of the pipeline environment, and trains the neural radiation field prior model offline based on the images and pose data of the pipeline environment. The acquisition module acquires real-time images from the vision camera and inertial data from the inertial measurement unit to obtain multi-sensor synchronous data packets. The triggering module triggers the active projector to project coded structured light based on the status of the multi-sensor synchronization data packet, thereby obtaining an image with an active illumination pattern. The fusion module inputs an image with an active lighting pattern and a coarse pose predicted by inertial data into a neural radiation field prior model to obtain a virtual feature point set. The virtual feature point set and inertial data are fused and optimized through tight coupling to obtain the pose, velocity and sensor bias of the pipeline robot. The localization module performs closed-loop detection and global graph optimization based on pose and feature information to obtain a globally consistent pose trajectory. Based on the pose and the globally consistent pose trajectory, the localization result of the pipeline robot is obtained.

[0093] This embodiment also provides a computer device applicable to the pipeline robot localization method based on visual-inertial navigation fusion, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the pipeline robot localization method based on visual-inertial navigation fusion as proposed in the above embodiment.

[0094] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0095] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the pipeline robot localization method based on vision-inertial fusion as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0096] In summary, this invention trains a prior model of neural radiation field offline by collecting images and pose data of the pipeline environment, collects real-time images and inertial data online, and adaptively triggers an active projector to project coded structured light to obtain an active illumination image based on the state of the multi-sensor data packets. This image and the inertial prediction pose are input into the prior model to generate a virtual feature point set. The robot's pose, velocity, and sensor deviation are obtained by tightly coupling optimization by fusing real features and inertial data. Combined with closed-loop detection and global graph optimization, a globally consistent pose trajectory is generated, achieving accurate and continuous positioning within the pipeline.

[0097] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for locating pipeline robots based on visual-inertial navigation fusion, characterized in that, include: S1. Collect images and pose data of the pipeline environment, and train the neural radiation field prior model offline based on the images and pose data of the pipeline environment. The neural radiation field prior model is used to characterize the geometric structure and appearance distribution inside the pipeline. S2. Acquire real-time pipeline images from the vision camera and inertial data from the inertial measurement unit, and generate time-aligned multi-sensor synchronization data packets based on timestamps; S3. Based on the image entropy of the confidence ellipsoid volume, the number of ORB feature points and the distribution of ORB feature points obtained from the multi-sensor synchronization data packet, trigger the active projector to project coded structured light to obtain an image with an active illumination pattern. S4. Input the image with the active lighting pattern and the coarse pose predicted by the inertial data into the neural radiation field prior model to obtain a virtual feature point set with two-dimensional image coordinates and three-dimensional spatial coordinates. Extract the real feature point set from the image with active lighting pattern, and probabilistically correlate the virtual feature point set with the real feature point set to obtain a comprehensive feature set; Based on the comprehensive feature set and inertial data, a tightly coupled optimization graph optimization problem is constructed and solved to obtain the pose, velocity and sensor bias of the pipeline robot. The graph optimization problem includes visual reprojection error term, inertial measurement unit pre-integration error term and virtual feature point three-dimensional coordinate prior error term. S5. Based on the pose and comprehensive feature set of the pipeline robot, perform closed-loop detection and global graph optimization to obtain a globally consistent pose trajectory. Based on the pose of the pipeline robot and the globally consistent pose trajectory, obtain the localization result of the pipeline robot.

2. The pipeline robot localization method based on visual-inertial fusion as described in claim 1, characterized in that, S1 specifically includes: S1.1 Control the pipeline robot to perform a spiral trajectory motion inside the pipeline, record the visual image sequence and inertial measurement data, and integrate from the inertial measurement data to obtain a coarse pose sequence corresponding to each frame of visual image; S1.2 Input the visual image sequence and the coarse pose sequence into the visual structure recovery process, reconstruct the sparse three-dimensional point cloud through feature matching and triangulation, and obtain the optimized pose corresponding to each frame of visual image through bundle adjustment, generating image-pose-three-dimensional point correspondence data. S1.

3. Train the neural radiation field prior model using image-pose-3D point correspondence data. The training loss includes volume rendering loss and geometric consistency constraint loss provided by sparse 3D point cloud, so that the neural radiation field prior model learns a continuous scene representation consistent with the real geometry of the pipeline.

3. The pipeline robot localization method based on visual-inertial fusion as described in claim 1, characterized in that, S2 specifically includes: S2.1 The vision camera acquires real-time images of the pipeline at a first frequency, and the inertial measurement unit acquires angular velocity and linear acceleration data at a second frequency. S2.2, mark the host clock timestamp for each frame of real-time pipeline image and each data packet of inertial measurement unit; S2.

3. Based on the timestamp, associate and package each frame of real-time pipeline image with its corresponding inertial measurement unit data packet within the time window; S2.4 Perform temperature compensation and preliminary deviation correction on the associated and packaged inertial measurement unit data, perform distortion removal and histogram equalization processing on the real-time pipeline image, and generate time-aligned multi-sensor synchronization data packets.

4. The pipeline robot localization method based on visual-inertial fusion as described in claim 1, characterized in that, S3 specifically includes: S3.1, Based on the covariance propagation model of inertial measurement unit data, predict the confidence ellipsoid volume of pose estimation within a fixed time period in the future; S3.2 Extract the number of ORB feature points from the real-time pipeline image in the multi-sensor synchronization data packet, and calculate the image entropy of the ORB feature point distribution; S3.3 Input the image entropy of the confidence ellipsoid volume, the number of ORB feature points, and the distribution of ORB feature points into a rule-based state machine. The state machine obtains discrete trigger levels based on the preset priority logic and the combined state of the image entropy of the confidence ellipsoid volume, the number of ORB feature points, and the distribution of ORB feature points. S3.4 When the trigger level reaches the preset activation level, a trigger command is generated, which controls the active projector to emit coded structured light in the form of a pseudo-random speckle pattern within a preset time. S3.5, trigger command controls the global shutter of the vision camera to align the exposure time with the emission time of the active projector, and expose during the coded structured light illumination to obtain an image with an active illumination pattern.

5. The pipeline robot localization method based on visual-inertial fusion as described in claim 1, characterized in that, The method for obtaining the virtual feature point set in S4 is as follows: the coarse pose of the pipeline robot at the current moment is obtained by inertial navigation calculation from the inertial measurement unit in the multi-sensor synchronization data packet, which includes coarse position and coarse attitude. The image with active lighting pattern and this coarse pose are input into the trained neural radiation field prior model. The neural radiation field prior model uses its encoder to extract the depth features of the image with active lighting pattern. Combined with the coarse pose information, the model performs three-dimensional spatial coordinate query and two-dimensional image coordinate regression in the decoder network. The network outputs a vector containing two-dimensional pixel coordinates and corresponding precise three-dimensional spatial coordinates. Each vector represents a virtual feature point.

6. The pipeline robot localization method based on visual-inertial fusion as described in claim 1, characterized in that, The graph optimization problems that are constructed and solved in S4 for tightly coupled optimization include: Using inertial measurement unit data, continuous measurements of linear acceleration and angular velocity are accumulated and encapsulated between adjacent image frames showing changes in visual features, forming relative motion constraint terms; Based on relative motion constraints, this is a graph optimization problem that constructs visual reprojection error terms, inertial measurement unit pre-integration error terms, and prior 3D coordinate error terms of virtual feature points for a comprehensive feature set. The visual reprojection error term is used to constrain the pose of the pipeline robot and the three-dimensional coordinates of the feature points in the integrated feature set, so that the two-dimensional coordinates of its back projection are consistent with the image observation coordinates. The pre-integration error term of the inertial measurement unit is used to constrain the pose, velocity and sensor deviation of the pipeline robot at adjacent time points, satisfying the inertial kinematic relationship described by the relative motion constraint term; The prior error term for the three-dimensional coordinates of virtual feature points is used to constrain the three-dimensional coordinates of the virtual feature point set to be consistent with the prior geometric information obtained from the neural radiation field prior model; By solving the graph optimization problem, minimizing the visual reprojection error term, the inertial measurement unit pre-integration error term, and the prior error term of the three-dimensional coordinates of the virtual feature points, the pose, velocity, and sensor deviation of the pipeline robot are obtained.

7. The pipeline robot localization method based on visual-inertial fusion as described in claim 1, characterized in that, S5 specifically includes: S5.

1. Based on the pose and comprehensive feature set of the pipeline robot, the BRIEF descriptor is used for inverted index retrieval to obtain loop detection candidate frames; S5.2 Perform random sampling consistency verification on the comprehensive feature set of the loop detection candidate frame and the current frame to obtain the relative pose transformation; S5.3 Add the relative pose transformation as a closed-loop constraint edge to the global pose graph; S5.4 Optimize the global pose graph with added closed-loop constraint edges to obtain a globally consistent pose trajectory. S5.

5. The pose of the pipeline robot is timestamped with the globally consistent pose trajectory to obtain the aligned pose sequence. The aligned pose sequence is then fused with the globally consistent pose trajectory to obtain the localization result of the pipeline robot.

8. A pipeline robot positioning system based on vision-inertial navigation fusion, characterized in that, It includes a visual camera, an inertial measurement unit, an active projector, a memory, a processor, and training, acquisition, triggering, fusion, and localization modules executed by the processor, wherein: The training module is used to collect images and pose data of the pipeline environment, and to train the neural radiation field prior model offline based on the images and pose data of the pipeline environment. The acquisition module is used to acquire real-time pipeline images through a vision camera, acquire inertial data through an inertial measurement unit, and generate time-aligned multi-sensor synchronization data packets based on timestamps. The triggering module is used to trigger the active projector to project coded structured light based on the confidence ellipsoid volume, the number of ORB feature points and the image entropy of the ORB feature point distribution obtained from the multi-sensor synchronization data packet, and to control the exposure of the vision camera during the coded structured light illumination to obtain an image with an active illumination pattern. The fusion module is used to input the image with active lighting pattern and the coarse pose predicted by inertial data into the neural radiation field prior model to obtain a virtual feature point set with two-dimensional image coordinates and three-dimensional spatial coordinates; extract the real feature point set from the image with active lighting pattern, and probabilistically correlate the virtual feature point set with the real feature point set to obtain a comprehensive feature set; and construct and solve a tightly coupled optimization graph optimization problem based on the comprehensive feature set and inertial data to obtain the pose, velocity and sensor bias of the pipeline robot. The localization module is used to perform closed-loop detection and global graph optimization based on the pose and comprehensive feature set of the pipeline robot to obtain a globally consistent pose trajectory. Based on the pose of the pipeline robot and the globally consistent pose trajectory, the localization result of the pipeline robot is obtained.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor enables the processor to implement the pipeline robot localization method based on visual-inertial fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the pipeline robot localization method based on visual-inertial fusion as described in any one of claims 1 to 7.

Images