Pipe network detection robot positioning method, device, equipment, medium and product

Abstract

The invention discloses a pipe network detection robot positioning method, device and equipment, a medium and a product. Images are obtained through a binocular camera arranged on the detection robot, and a left eye image and a right eye image are transmitted in a topic mode; extracting feature points and descriptors of the left eye image and the right eye image by using an acceleration feature network; generating an initialized pipeline map by using an SLAM system and determining an initial pose of a camera through a tracking thread; the initial pose is optimized through the feature points and the descriptors; constructing a cylinder according to the initialized pipeline map, and inserting key frames meeting conditions into a local mapping thread; generating map points and cylindrical points by using the local mapping thread and the received key frame, and executing local BA; and a closed-loop detection thread is utilized to search for candidate loopback frames through an XFat database, similarity transformation is calculated to obtain an accumulated error, and a global BA optimization map is executed to position the detection robot. According to the scheme, precise positioning of the pipe network detection robot in a pipeline complex scene can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of pipe network positioning, and in particular to a pipe network detection robot positioning method, device, equipment, medium and product. Background Art

[0002] With the rapid development of urban pipe networks, pipe network inspection robots are increasingly being used in pipeline inspections and fault location. Accurate positioning is a core requirement for autonomous inspections by pipe network inspection robots. Simultaneous Localization and Mapping (SLAM) technology, with its low cost and high information content, has become a mainstream solution for robot positioning.

[0003] The ORB-SLAM series of algorithms have outstanding performance in the field of traditional visual SLAM due to their efficient feature processing capabilities, multi-threaded architecture and loop detection functions, and are widely used in positioning tasks in various scenarios.

[0004] However, in special scenarios such as pipelines, the limitations of traditional feature point methods gradually become apparent: in pipeline network inspection scenarios, robots need to operate in 2 / 3 water, semi-water or no water environments, facing complex challenges such as sparse texture on the inner wall of the pipeline, uneven lighting, metal reflections, and repeated structures, which places extremely high demands on the robustness and positioning accuracy of SLAM technology. Summary of the Invention

[0005] The present invention provides a method, device, equipment, medium and product for positioning a pipe network inspection robot, which can achieve precise positioning of the pipe network inspection robot in complex pipeline scenarios.

[0006] According to one aspect of the present invention, a method for positioning a pipe network inspection robot is provided, comprising:

[0007] The detection robot acquires images through a binocular camera, and transmits the left image and the right image in the form of topics respectively; and extracts feature points and descriptors of the left image and the right image using an accelerated feature network;

[0008] Generate an initial pipeline map using the SLAM system and determine the initial pose of the camera through a tracking thread; optimize the initial pose using the feature points and descriptors;

[0009] Construct a cylinder according to the initialized pipeline map, and insert the keyframes that meet the conditions into the local mapping thread;

[0010] Using the local mapping thread to generate map points and cylinder points using the received key frames, and perform local BA;

[0011] The closed-loop detection thread is used to search for candidate loop frames through the XFeat database, calculate the similarity transformation to obtain the cumulative error, perform the global BA optimization map, and locate the detection robot through the optimized map.

[0012] Optionally, the left eye image and the right eye image acquired by the binocular camera are respectively transmitted through independent communication topics, the topic format adopts the Robot Operating System ROS standard message type, and the transmission frame rate is the same as the acquisition frame rate of the binocular camera.

[0013] Optionally, the acceleration feature network is an end-to-end network model based on deep learning, whose input is RGB data of left and right images, and whose output includes:

[0014] A key point heat map representing the probability distribution of the positions of feature points in the image in the form of a two-dimensional matrix, wherein a higher value of the first matrix element in the key point heat map indicates a greater possibility that a feature point exists at that position;

[0015] A reliability heatmap of the same size as the key point heatmap, wherein the second matrix element value of the reliability heatmap reflects the reliability score of the feature point at the corresponding position, and the score range is [0, 1], and the closer the value is to 1, the higher the reliability;

[0016] Descriptor set, each feature point corresponds to a 128-dimensional binary descriptor vector, which is used for matching between feature points.

[0017] Optionally, the extracting feature points of the left image and the right image using an accelerated feature network includes:

[0018] The confidence score of each feature point in the key point heat map and the reliability heat map is calculated using the following formula:

[0019] ;

[0020] in, is the normalized score of a feature point in the key point heat map, is the normalized score of a feature point in the key point heat map, and is the weight coefficient, and + =1;

[0021] The top N feature points with the highest scores are selected, and the evenly distributed feature points are screened by the grid division method. The image is divided into M×M grids, and no more than P feature points are retained in each grid.

[0022] Optionally, generating an initial pipeline map using a SLAM system and determining an initial position of the camera through a tracking thread includes:

[0023] Selecting a preset number of binocular images, calculating the basic matrix through feature point matching, decomposing the initial rotation matrix and translation vector, and combining the binocular camera intrinsic parameters and baseline distance to triangulate and generate a three-dimensional point cloud to form the initial pipeline map;

[0024] Determine whether there is a previous frame image. If so, predict the current pose of the binocular camera based on the constant speed motion model as the initial value, match the feature points of the current frame with the previous frame, use the RANSAC-PnP algorithm to solve the pose and iteratively optimize;

[0025] If the initial pose is determined for the first time after the SLAM system is started, the initial pose is obtained by filtering the inliers from the feature point matching pairs using a random sampling consistency algorithm, calculating the essential matrix and decomposing the matrix.

[0026] Optionally, optimizing the initial pose using the feature points and descriptors includes:

[0027] Selecting map points corresponding to a preset number of keyframes with the highest degree of co-viewing with the current frame from the local map, extracting their descriptors to form a local feature library, and associating each map point with observation information of at least a preset number of keyframes;

[0028] A two-way matching mechanism is used to first match the feature point descriptor of the current frame with the local feature library to obtain the initial matching pairs. Then, for each matching pair, the projection consistency between the feature point of the current frame and the map point in the corresponding key frame is verified, and matching pairs with projection errors exceeding the preset number of pixels are eliminated.

[0029] Taking the initial pose as the starting point of iteration, a pose optimization objective function based on Lie algebra is constructed. The objective function is the weighted sum of the reprojection errors of all matching points, where the weights are dynamically adjusted according to the number of observations of the map points.

[0030] The Gauss-Newton algorithm is used for iterative optimization. The Jacobian matrix is ​​calculated after each iteration to update the pose parameters. The optimization is stopped when the number of iterations reaches the preset number or the pose change between two adjacent iterations is less than 1e-6 radians and 1e-3 meters, and the optimized pose is output.

[0031] Optionally, constructing a cylinder according to the initialized pipeline map and inserting a key frame that meets the conditions into a local mapping thread includes:

[0032] Based on the point cloud data of the initialized pipeline map, a random sampling consistency algorithm is used to fit the cylinder parameters, which include the unit direction vector of the cylinder axis, the coordinates of the axis midpoint, and the cylinder radius. The distance from each point in the point cloud to the candidate axis is calculated using the point-to-line distance formula. Inliers with a distance less than a preset threshold are selected. When the proportion of inliers exceeds a preset proportion of the total number of point clouds, the candidate cylinder is determined to be a valid model.

[0033] The current frame is determined to be a keyframe and inserted into the local mapping thread when it meets the following conditions:

[0034] The translation distance from the previous keyframe exceeds 1 / 5 of the pipe diameter or the rotation angle exceeds 5 degrees;

[0035] The number of matching pairs between the feature points extracted in the current frame and the local map feature points is no less than 80, and the distribution of the matching points in the image covers at least 8 preset grid areas;

[0036] The root mean square value of the reprojection error after pose optimization is less than 1.5 pixels;

[0037] The keyframe inserted into the local mapping thread contains the following information: optimized camera pose, feature point set and corresponding descriptors, image acquisition timestamp, co-viewing relationship with adjacent keyframes, and projection position parameters of the keyframe in the cylindrical model.

[0038] Optionally, the using the local mapping thread to generate map points and cylinder points using the received key frames and perform local BA, including:

[0039] For the matching feature point pairs between key frames and adjacent key frames, the 3D coordinates are calculated through binocular triangulation. Points with a reprojection error of less than 2 pixels and a parallax angle greater than 1 degree are selected as valid map points. Each map point is associated with its observed key frame index and observed pixel coordinates.

[0040] Based on the cylindrical model, the cylindrical constraint verification is performed on the valid map point, and the distance from the map point to the cylinder axis is calculated. If the deviation between the distance and the cylinder radius is within a preset value, it is marked as a cylinder point, and its parametric coordinates in the circumferential and axial directions of the cylinder are recorded;

[0041] Construct an optimization window containing the current keyframe, its co-view keyframes and corresponding map points, express the camera pose with Lie algebra, the vertices include the keyframe poses and map point coordinates, and the edges are observation constraints; add additional cylindrical constraint edges to the cylindrical points, constraining their distance to the axis to be equal to the cylinder radius, and setting the weight to 1.5 times that of the ordinary observation edge; use the g2o optimization library for sparse BA solver, iterate until the root mean square of the reprojection error is less than 1.2 pixels or the number of iterations reaches 30, and output the optimized pose and map point coordinates.

[0042] Optionally, the loop closure detection thread searches for candidate loop frames through the XFeat database, calculates similarity transformation to obtain cumulative error, and performs global BA optimization map, including:

[0043] The feature point descriptor of the current key frame is input into the XFeat database, and the top 20 historical key frames with the highest matching degree with the current frame features are found as candidate loop frames through the approximate nearest neighbor search algorithm;

[0044] For the candidate loop frame, the similarity transformation matrix is ​​estimated from the feature matching pairs using the RANSAC algorithm, where the inlier threshold is set to 3 pixels. When the number of inliers exceeds 30% of the total number of matching pairs and the transformation matrix satisfies the homography constraint, the candidate frame is confirmed to be a valid loop frame. The cumulative error is calculated based on the similarity transformation matrix, and the error value is the root mean square of the reprojection error of all inliers.

[0045] A global optimization graph containing all keyframe poses and map points is constructed, and the similarity transformation obtained by loop closure detection is added to the graph as a strong constraint. The constraint weights are dynamically adjusted according to the accumulated error. A sparse BA algorithm is used for optimization, with the weighted sum of the reprojection error and the loop closure constraint error as the objective function. During the iterative process, a marginalization operation is performed every 10 iterations to remove redundant variables. The optimization is stopped when the change in the objective function between two consecutive iterations is less than 1e-6 or the number of iterations reaches 80.

[0046] According to another aspect of the present invention, a pipe network inspection robot positioning device is provided, comprising:

[0047] An image acquisition unit is configured to acquire images using a binocular camera provided on the detection robot, transmit a left image and a right image in the form of topics, and extract feature points and descriptors of the left and right images using an accelerated feature network;

[0048] An initial pose determination unit is used to generate an initial pipeline map using a SLAM system and determine the initial pose of the camera through a tracking thread; and optimize the initial pose using the feature points and descriptors;

[0049] A cylinder construction unit, configured to construct a cylinder according to the initialized pipeline map and insert key frames that meet the conditions into a local mapping thread;

[0050] A local BA unit, configured to generate map points and cylinder points using the received keyframes using the local mapping thread, and perform local BA;

[0051] The global BA unit is used to use the closed-loop detection thread to find candidate loop frames through the XFeat database, calculate the similarity transformation to obtain the cumulative error, perform global BA to optimize the map, and locate the detection robot through the optimized map.

[0052] According to another aspect of the present invention, an electronic device is provided, comprising:

[0053] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the pipe network inspection robot positioning method described in any embodiment of the present invention.

[0054] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the pipe network inspection robot positioning method described in any embodiment of the present invention when executed.

[0055] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the pipe network inspection robot positioning method according to any embodiment of the present invention is implemented.

[0056] The technical solution of this embodiment, which leverages deep learning and prior information about pipe geometry, addresses the challenges of traditional visual SLAM methods, which suffer from the serious limitations of positioning accuracy and robustness due to issues such as repetitive pipe structures, sparse pipe wall textures, and uneven internal lighting. It achieves centimeter-level positioning in complex pipe environments, reducing positioning errors and improving positioning accuracy compared to traditional positioning methods. This approach is suitable for widespread deployment and promotion in urban pipeline systems, and can be expanded to similar scenarios such as diversion tunnels, culverts, and tunnels.

[0057] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 This is a flow chart of a pipe network inspection robot positioning method provided in Example 1 of the present invention;

[0060] Figure 2 This is a flow chart of a method for determining an initial posture provided by the second embodiment of the present invention;

[0061] Figure 3 This is a flow chart of a local BA method provided in Example 3 of the present invention;

[0062] Figure 4 This is a flow chart of a global BA method provided in Example 3 of the present invention;

[0063] Figure 5 This is a structural diagram of a pipe network inspection robot positioning device provided by a fourth embodiment of the present invention;

[0064] Figure 6 It is a structural diagram of an electronic device for implementing the positioning method of a pipe network inspection robot according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0066] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0067] Example 1

[0068] Figure 1 This is a flow chart of a method for positioning a pipe network inspection robot provided by the first embodiment of the present invention. This embodiment is applicable to the situation where a pipe network inspection robot is positioned in a complex pipeline scene. The method can be executed by a pipe network inspection robot positioning device. The pipe network inspection robot positioning device can be implemented in the form of hardware and / or software. The pipe network inspection robot positioning device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0069] S110, acquiring images through a binocular camera set by the detection robot, transmitting the left eye image and the right eye image in the form of topics respectively; and extracting feature points and descriptors of the left eye image and the right eye image using an accelerated feature network.

[0070] The binocular camera carried by the detection robot synchronously collects left and right images of the internal environment of the pipeline, similar to human binocular vision. Through the communication mechanism of the robot operating system, the left and right images are encapsulated as independent topics for real-time transmission.

[0071] Feature points are representative key pixel points identified from the image, such as seams, protrusions and other significant features on the inner wall of the pipe. These points remain stable under different viewing angles and lighting; descriptor generation: a high-dimensional vector descriptor is generated for each feature point to describe the local image features of the point, such as texture, gradient, etc., so that the same physical points in different images can be associated through descriptor matching.

[0072] In an embodiment of the present invention, the left eye image and the right eye image acquired by the binocular camera are transmitted through independent communication topics respectively. The topic format adopts the standard message type of the Robot Operating System ROS, and the transmission frame rate is the same as the acquisition frame rate of the binocular camera.

[0073] The left and right images are transmitted via two independent ROS topics. The core of binocular vision is to calculate depth information through the parallax of the left and right images. Independent topics ensure that the left and right images are read separately by the front-end processing module of the SLAM system, avoiding data aliasing.

[0074] Images are transmitted using the sensor_msgs / Image standard message format defined by ROS, which can be directly parsed by the image processing library in the ROS ecosystem. This ensures compatibility between camera driver nodes and SLAM algorithm nodes without the need for custom data formats.

[0075] The transmission frame rate is consistent with the camera acquisition frame rate, ensuring the real-time and time synchronization of the image data: on the one hand, it avoids image backlog or loss due to too low a transmission frame rate, ensuring that the SLAM system can perform pose estimation based on the latest image; on the other hand, the left and right images are strictly aligned in time, reducing the left and right image synchronization error caused by robot movement.

[0076] In this embodiment of the present invention, the acceleration feature network is an end-to-end network model based on deep learning. Its input is the RGB data of the left and right images, and its output includes:

[0077] A key point heat map representing the probability distribution of the locations of feature points in the image in the form of a two-dimensional matrix, wherein the higher the value of the first matrix element in the key point heat map, the greater the possibility that a feature point exists at that location;

[0078] The reliability heatmap has the same size as the key point heatmap. The second matrix element value of the reliability heatmap reflects the reliability score of the feature point at the corresponding position. The score range is [0,1]. The closer the value is to 1, the higher the reliability.

[0079] Descriptor set, each feature point corresponds to a 128-dimensional binary descriptor vector, which is used for matching between feature points.

[0080] Specifically, the accelerated feature network is an end-to-end deep learning model that does not require manual design of feature extraction rules. It directly uses RGB color images captured by the left and right cameras as input and adapts to the extraction of color information under complex lighting conditions in the pipeline.

[0081] A keypoint heatmap is a two-dimensional probability matrix related to the input image size. For example, if the input image is 1280×720, the heatmap might be scaled to 320×180 or similar. The value of each element in the matrix represents the probability of the presence of a keypoint at that image location, such as a pipe seam, a bump, or a stain. A higher value indicates a more likely location to be a stable keypoint.

[0082] The reliability heatmap completely corresponds to the size of the key point heatmap. Its element value (range 0-1) represents the reliability of the feature point at the corresponding position. The closer the value is to 1, the more stable the feature point can remain in the face of changes in viewing angle, lighting fluctuations, slight occlusion, etc., which can effectively reduce the risk of feature mismatch caused by interference such as reflections and water stains in the pipeline.

[0083] A 128-dimensional binary descriptor set generates a binary vector for each filtered feature point. By encoding local information around the feature point, such as texture, gradient, and color distribution, this allows for rapid matching of feature points across different images. Binary descriptors are more lightweight than floating-point descriptors, accelerating the real-time matching efficiency of SLAM systems and meeting the real-time demands of robotic inspections.

[0084] In an embodiment of the present invention, extracting feature points of the left and right images using an accelerated feature network includes:

[0085] The confidence score of each feature point in the key point heat map and the reliability heat map is calculated using the following formula:

[0086] ;

[0087] in, is the normalized score of a feature point in the key point heat map, is the normalized score of a feature point in the key point heat map, and is the weight coefficient, and + =1.

[0088] The top N feature points with the highest scores are selected, and the evenly distributed feature points are screened by the grid division method. The image is divided into M×M grids, and no more than P feature points are retained in each grid.

[0089] It is the normalized score of the feature point extracted from the key point heat map, reflecting the probability of the position being a feature point. Correspondingly, It is the normalized score of the feature point extracted from the reliability heat map, reflecting the stability of the feature point in a complex environment. and Used to balance the importance of key point probability and reliability. For example, in a scene with sparse pipeline textures, , giving priority to retaining feature points with high reliability.

[0090] Select the top N scoring feature points, ensuring that the highest-scoring candidate points are retained. Divide the image into a uniform M×M grid (e.g., 16×16), retaining a maximum of P feature points within each grid. This prevents feature points from being concentrated in localized areas, such as pipe joints, and ensures that they are evenly distributed across the image. Evenly distributed feature points provide more comprehensive spatial constraints for subsequent pose estimation, reducing positioning errors caused by missing local features. This uniformity improves the stability of rotation and translation estimates, especially in long, symmetrical structures like pipes.

[0091] S120, using the SLAM system to generate an initial pipeline map and determining the initial pose of the camera through the tracking thread; optimizing the initial pose through feature points and descriptors.

[0092] After the SLAM system is started, the first few frames of images captured by the binocular camera are used to calculate the three-dimensional point cloud through binocular stereo matching. Combined with the prior knowledge of the pipeline structure, an initial pipeline map containing three-dimensional point coordinates and preliminary geometric constraints is constructed.

[0093] The tracking thread obtains the initial pose in the following way:

[0094] If this is the first startup and there is no historical data, the base matrix / essential matrix between images is calculated through feature point matching, decomposing the camera's rotation and translation parameters. A world coordinate system is then established with the camera coordinate system of the first frame as the origin. If the system is restarted or relocated, the current camera's position in the world coordinate system is quickly determined by feature matching with the existing initial map.

[0095] The feature points and descriptors extracted from the current frame are matched with the 3D points in the initialization map. By comparing the similarity of the descriptors, the correspondence between the image feature points and the 3D points in space is found. A reprojection error function is constructed based on the matched pairs. This error is minimized using a nonlinear optimization algorithm, and the camera's rotation and translation parameters are iteratively corrected. Ultimately, the reprojection error is controlled within a preset threshold, ensuring the accuracy of the initial pose.

[0096] S130: Construct a cylinder according to the initialized pipeline map, and insert the key frames that meet the conditions into the local mapping thread.

[0097] From the 3D point cloud used to initialize the pipeline map, an algorithm is used to fit the cylinder's parameters, including the spatial orientation of the cylinder's axis, the coordinates of the axis's midpoint, and the cylinder's radius. During the fitting process, points that meet the cylinder's geometric constraints are selected, such as those whose distance from the axis is close to the pipeline's radius. Noise points, such as those caused by impurities, bubbles, and other interference within the pipeline, are eliminated. The cylindrical model adds structural constraints to the map, and newly generated map points are verified for conformance to cylindrical characteristics, thereby reducing cumulative error. This is particularly suitable for scenarios with long, repetitive structures such as pipelines.

[0098] Keyframes are key image frames used to construct maps in SLAM systems. They contain core information such as camera pose and feature points. Not all image frames are selected as keyframes.

[0099] Screening criteria typically include:

[0100] Movement distance: Compared with the previous keyframe, the robot's movement distance exceeds 1 / 5 of the pipe diameter or the rotation angle exceeds 5° to avoid redundant frames and ensure mapping efficiency;

[0101] Feature matching quality: The number of feature matching pairs between the current frame and the local map is sufficient, such as no less than 80 pairs, and is evenly distributed, covering multiple areas of the image;

[0102] Pose accuracy: The optimized camera pose reprojection error is small, such as root mean square <1.5 pixels, ensuring pose reliability.

[0103] Qualified keyframes will be transmitted to the local mapping thread to generate new 3D map points, optimize the local map structure, and merge with the cylindrical model to gradually build a more accurate pipeline environment map.

[0104] S140: Use the local mapping thread to generate map points and cylinder points using the received key frames and perform local BA.

[0105] After receiving a new keyframe, the local mapping thread matches it with neighboring keyframes. Using binocular vision triangulation, it calculates the 3D coordinates of the matching feature points and generates new map points. These map points are then screened, eliminating those with excessive reprojection errors or small parallax angles to ensure map point accuracy.

[0106] Combined with the constructed cylindrical pipe model, the newly generated map points are geometrically constrained and the distance from each map point to the cylindrical axis is calculated. If the deviation between this distance and the theoretical pipe radius is within the allowable range, the point is marked as a cylindrical point and its parameters in the cylindrical coordinate system are recorded. The significance of the cylindrical point is to strengthen the cylindrical structural constraints of the pipe and reduce map drift caused by sparse features or noise.

[0107] BA is a nonlinear least-squares algorithm used to optimize camera pose and 3D point coordinates. Its core goal is to improve pose consistency with the map by minimizing reprojection error. Unlike global BA, which optimizes the entire map, local BA constructs an optimization window only for the current keyframe, its co-viewed neighboring keyframes, and the map points observed by these keyframes. This avoids the high computational cost of global optimization and ensures real-time performance.

[0108] S150, using the closed-loop detection thread to search for candidate loop frames through the XFeat database, calculating the similarity transformation to obtain the cumulative error, performing the global BA optimization map, and positioning the detection robot through the optimized map.

[0109] After moving a certain distance in a pipeline, the robot may return to an area it previously traversed. Loop closure detection is necessary to identify this phenomenon and correct accumulated positioning errors. XFeat is a highly efficient feature extraction and matching algorithm that builds a database that stores feature descriptors of historical keyframes. The loop closure detection thread compares the feature descriptors of the current keyframe with the historical data in the database and uses an approximate nearest neighbor search to find the top N historical keyframes with the highest feature matches as candidate loop frames. To avoid repeated matches within a short period of time, candidate frames that are too close to the current frame are filtered out to ensure the effectiveness of loop closure detection.

[0110] Global BA is a global optimization algorithm that minimizes the overall reprojection error, eliminates cumulative errors, and ensures the global consistency of the map by adjusting the poses and map point coordinates of all keyframes. A global optimization model containing all keyframes and map points is constructed, and the similarity transformation obtained by loop closure detection is added to the model as a strong constraint. The sparse BA algorithm is used for iterative optimization until the error converges or the maximum number of iterations is reached. The optimized globally consistent map eliminates scale drift and pose deviation caused by long-distance movement. The globally optimized map provides an accurate three-dimensional model of the pipeline environment. The robot can match features with the map through real-time collected images, and combine algorithms such as PnP to solve the current camera pose to achieve precise positioning based on the globally consistent map. This positioning method is not affected by cumulative errors and can maintain high positioning accuracy even in long-distance pipeline inspections.

[0111] Example 2

[0112] Figure 2 This is a flow chart of a method for determining an initial posture provided by the second embodiment of the present invention. Figure 2 As shown, the method includes:

[0113] S210, select a preset number of binocular images, calculate the basic matrix through feature point matching, decompose to obtain the initial rotation matrix and translation vector, combine the binocular camera internal parameters and baseline distance, and triangulate to generate a three-dimensional point cloud to form the initial pipeline map.

[0114] After the SLAM system is activated, it first collects a preset number of binocular images. These images must meet certain motion constraints to ensure the reliability of subsequent feature matching and geometric calculations. Although a single-frame binocular image can generate a local point cloud, the lack of motion information makes it difficult to construct a globally consistent coordinate system. The motion relationship between multiple frames can provide a scale and orientation reference.

[0115] For the selected image sequence, the feature points and descriptors of the left and right eye images of each frame are extracted through the accelerated feature network, and then the feature points are matched between adjacent frames to find the corresponding pixel position of the same physical point in space in different images.

[0116] Based on the matched feature point pairs, the RANSAC algorithm is used to estimate the fundamental matrix. The fundamental matrix is ​​a crucial geometric constraint in binocular vision, describing the projective relationship between the left and right eye images. Essentially, it encodes the mathematical relationship between camera intrinsics, relative pose, and spatial point coordinates, and can be used to verify the validity of feature matching. The fundamental matrix contains information about the relative pose between the corresponding cameras in the two image frames. By performing SVD on the fundamental matrix, the initial camera rotation matrix and translation vector can be obtained.

[0117] Using the binocular camera's intrinsic parameters and baseline distance, feature point coordinates in the pixel coordinate system are converted to ray directions in the camera coordinate system. For each matched feature point, its 3D coordinates in the world coordinate system are calculated using geometric triangulation principles based on the relative poses of adjacent frames. This process is repeated for all matched points, generating a dense 3D point cloud. This 3D point cloud is organized by spatial position and, combined with the cylindrical shape prior of the pipe, forms an initial 3D map of the pipe's inner wall, encompassing key features.

[0118] S220. Determine whether there is a previous frame image. If so, predict the current pose of the binocular camera based on the constant speed motion model as the initial value, match the feature points of the current frame with the previous frame, use the RANSAC-PnP algorithm to solve the pose and iteratively optimize.

[0119] The SLAM system first checks whether there is image data from the previous frame that has been processed. If there is a previous frame, it means that the robot is in continuous motion, and the temporal continuity of motion can be used to predict the pose. In a short time interval, the robot's motion can be approximated as uniform linear motion and uniform rotational motion. Prediction process: Based on the camera pose of the previous frame and the motion speed of the previous frames, the predicted pose of the current frame is inferred. This predicted value serves as the initial value for subsequent pose solutions, which can significantly narrow the search range of the optimization algorithm, improve the solution speed and convergence, and avoid falling into local optimality. For the images of the current frame and the previous frame, the similarity of the feature descriptors is compared to find the feature point pairs corresponding to the same physical point in the two frames.

[0120] Given a 3D point and its 2D projection in the current image, the PnP algorithm is used to determine the camera pose. Its core is to establish and solve a nonlinear system of equations based on the projection relationship between spatial points and image points.

[0121] The random sampling consensus RANSAC algorithm is used to handle mismatches within matching pairs. It solves the pose by randomly sampling a subset of matching pairs multiple times. It then counts the number of inliers that meet the projection constraints for that pose and retains the pose with the most inliers as the optimal solution, significantly improving the robustness of the pose solution. The pose obtained by RANSAC is used as the initial value to construct a reprojection error function. This error is iteratively minimized using a nonlinear optimization algorithm, and the pose parameters are further refined until the error converges.

[0122] S230: If the initial pose is determined for the first time after the SLAM system is started, the internal points are selected from the feature point matching pairs through a random sampling consistency algorithm, and the essential matrix is ​​calculated and decomposed to obtain the initial pose.

[0123] When the SLAM system is first started, there is no historical pose data or map information, making it impossible to predict pose using motion information from the previous frame, as is done during the continuous tracking phase. At this point, the camera's initial pose must be calculated from scratch, relying entirely on the initially acquired image data. For the first two binocular image frames acquired after the SLAM system is started, a feature extraction network is used to extract their respective feature points and descriptors. Descriptor matching is then used to obtain matching pairs of feature points between the two frames. The random sampling consensus algorithm, RANSAC, is used to filter inliers.

[0124] Because mismatches are inevitable during feature matching, the RANSAC algorithm is used to filter inliers. This algorithm repeatedly randomly selects a small number of matching pairs, assumes these matching pairs are inliers, and calculates a geometric model. The number of matching pairs that meet the model's constraints is counted, and the matching pairs corresponding to the model with the most inliers are retained as valid data. Outliers that do not meet the constraints are removed.

[0125] Based on the selected inliers, the essential matrix corresponding to the two image frames is calculated. The essential matrix is ​​a key geometric constraint in binocular vision that describes the relative pose between two camera coordinate systems. Its mathematical expression encodes the relationship between the rotation matrix and translation vector between the two frames and is related to the camera intrinsic parameters. The essential matrix contains the relative pose information between the two frames. By performing a singular value decomposition on it, possible rotation matrix and translation vector combinations can be solved. Combined with actual physical constraints, a single reasonable rotation matrix and translation vector are selected from these solutions, which is the initial pose.

[0126] In an embodiment of the present invention, optimizing the initial pose by using feature points and descriptors includes:

[0127] Selecting map points corresponding to a preset number of keyframes with the highest degree of co-viewing with the current frame from the local map, extracting their descriptors to form a local feature library, and associating each map point with observation information of at least a preset number of keyframes;

[0128] A two-way matching mechanism is used to first match the feature point descriptor of the current frame with the local feature library to obtain the initial matching pairs. Then, for each matching pair, the projection consistency between the feature point of the current frame and the map point in the corresponding key frame is verified, and matching pairs with projection errors exceeding the preset number of pixels are eliminated.

[0129] Taking the initial pose as the starting point of iteration, a pose optimization objective function based on Lie algebra is constructed. The objective function is the weighted sum of the reprojection errors of all matching points, where the weights are dynamically adjusted according to the number of observations of the map points.

[0130] The Gauss-Newton algorithm is used for iterative optimization. The Jacobian matrix is ​​calculated after each iteration to update the pose parameters. The optimization is stopped when the number of iterations reaches the preset number or the pose change between two adjacent iterations is less than 1e-6 radians and 1e-3 meters, and the optimized pose is output.

[0131] A preset number of keyframes with the highest degree of co-viewing with the current frame are selected from the local map. The corresponding 3D map points and their descriptors are extracted from these keyframes to form a local feature library. The degree of co-viewing refers to the number of shared feature points. A higher degree of co-viewing indicates a stronger spatial correlation between these keyframes and the current frame, and thus a greater reference value for optimizing the current pose. Each map point must be associated with observations from at least the preset number of keyframes to ensure multi-viewpoint verification and higher accuracy.

[0132] The bidirectional matching mechanism first matches the descriptors of the feature points in the current frame with the descriptors of the map points in the local feature library to obtain initial matching pairs. Each initial matching pair is then verified for projection consistency. The 3D map points are projected back to the keyframe image based on their associated keyframe poses, and the deviation between the projected positions and the original observed feature points in the keyframe is checked. If the projection error exceeds a preset number of pixels, the match is considered a mismatch and is discarded. Only matching pairs with a satisfactory error are retained.

[0133] The camera pose is converted into a Lie algebraic form, transforming the pose optimization problem into a linearized vector space solution, simplifying iterative computations. The optimization objective is the weighted sum of the reprojection errors of all matching points. This is the deviation between the pixel position of a 3D map point projected onto the image using the current camera pose and the actual position of the corresponding feature point in the current frame. The more times a map point has been observed, the greater its weight is, as its spatial coordinates are more reliable and its contribution to optimization is expected to be greater.

[0134] Starting from the initial pose, the Gauss-Newton algorithm is used for iterative optimization. In each iteration, the Jacobian matrix of the objective function with respect to the Lie algebra variables is calculated, reflecting the sensitivity of the error to pose changes. The incremental equation is solved based on the Jacobian matrix, and the pose parameters are updated to reduce the objective function. When the number of iterations reaches a preset value, or the pose change between two consecutive iterations is less than a threshold, the iterations are terminated and the current pose is output as the optimization result. A numerical optimization algorithm is used to gradually correct the pose, minimizing the reprojection error, ultimately yielding a highly accurate camera pose.

[0135] In an embodiment of the present invention, a cylinder is constructed according to the initialized pipeline map, and key frames that meet the conditions are inserted into the local mapping thread, including:

[0136] Based on the point cloud data of the initialized pipeline map, a random sampling consistency algorithm is used to fit the cylinder parameters. The cylinder parameters include the unit direction vector of the cylinder axis, the coordinates of the axis midpoint, and the cylinder radius. The distance from each point in the point cloud to the candidate axis is calculated using the point-to-line distance formula. Inliers with distances less than a preset threshold are selected. When the proportion of inliers exceeds a preset ratio of the total number of point clouds, the candidate cylinder is determined to be a valid model.

[0137] The current frame is determined to be a keyframe and inserted into the local mapping thread when it meets the following conditions:

[0138] The translation distance from the previous keyframe exceeds 1 / 5 of the pipe diameter or the rotation angle exceeds 5 degrees;

[0139] The number of matching pairs between the feature points extracted in the current frame and the local map feature points is no less than 80, and the distribution of the matching points in the image covers at least 8 preset grid areas;

[0140] The root mean square value of the reprojection error after pose optimization is less than 1.5 pixels;

[0141] Among them, the keyframe inserted into the local mapping thread contains the following information: the optimized camera pose, the feature point set and the corresponding descriptor, the image acquisition timestamp, the co-viewing relationship with the adjacent keyframes, and the projection position parameters of the keyframe in the cylindrical model.

[0142] Pipelines have a typical cylindrical structure, which can be converted into geometric constraints through algorithms to improve map accuracy. From the 3D point cloud used to initialize the pipeline map, key cylinder parameters are extracted, including the unit direction vector of the axis, the coordinates of the axis midpoint, and the cylinder radius.

[0143] By randomly sampling a small number of points from the point cloud, a candidate cylinder axis is fitted. The distance from all points in the point cloud to the candidate axis is then calculated, and points with distances less than a preset threshold are selected as inliers. When the proportion of inliers to the total number of points in the point cloud exceeds a preset value, the candidate cylinder is considered a good fit for the pipeline structure and is considered a valid model. This step is used to remove noise points from the point cloud and enhance the pipeline's structural features.

[0144] Keyframes are the core image frames used to build and optimize maps. They must meet strict conditions to ensure mapping efficiency and accuracy. Keyframes are considered only when the translation distance between the current frame and the previous one exceeds 1 / 5 of the pipe diameter, or the rotation angle exceeds 5 degrees. This prevents repeated insertion of keyframes within a short distance and ensures sufficient parallax between adjacent keyframes to facilitate triangulation and generate high-precision map points.

[0145] The number of matching pairs between the current frame's feature points and the local map's feature points must be at least 80 to ensure sufficient spatial constraints for subsequent optimization. Matching points must cover at least eight pre-set grid areas in the image to avoid concentrating matching points in a single area and ensure comprehensive spatial constraints. The root mean square error of the optimized camera pose reprojection must be less than 1.5 pixels to ensure the pose estimate of the current frame is sufficiently accurate to avoid introducing errors into the map.

[0146] The keyframes inserted into the local mapping thread must carry multi-dimensional information to meet the needs of map construction and optimization: the optimized camera pose serves as the spatial reference for triangulation of map points; feature points and descriptors are used to match with other keyframes to establish cross-frame spatial associations; timestamps ensure the time synchronization of multi-sensor data; common view relationships record shared feature point information with other keyframes to construct the topological structure of the local map; cylindrical model projection parameters represent the axial position and circumferential angle of the current frame in the pipeline cylindrical model, strengthening the geometric association between the keyframe and the pipeline structure.

[0147] Example 3

[0148] Figure 3 This is a flow chart of a local BA method provided by the third embodiment of the present invention. Figure 3 As shown, the method includes:

[0149] S310. For the matching feature point pairs between the key frame and the adjacent key frame, calculate the 3D coordinates through binocular triangulation, select points with a reprojection error less than 2 pixels and a parallax angle greater than 1 degree as valid map points, and associate each map point with its observed key frame index and observed pixel coordinates.

[0150] A keyframe is matched with its adjacent keyframes using feature descriptors to obtain feature point pairs. Using the intrinsic parameters of the binocular camera and the relative pose between the two keyframes, the 2D pixel coordinates of a pair of matched feature points are converted to 3D spatial coordinates.

[0151] The calculated 3D map points are reprojected back to the original keyframe image according to the camera pose. If the pixel deviation between the projected position and the original feature point is less than 2 pixels, the 3D coordinate calculation is accurate; otherwise, it may be a mismatch or calculation error and needs to be eliminated. The parallax angle refers to the angle formed by the 3D point and the line connecting the optical centers of the two keyframe cameras. The larger the angle, the more significant the difference between the two perspectives, and the higher the depth accuracy of the triangulated calculation. If the angle is less than 1 degree, the depth calculation is susceptible to noise. Using these two conditions, low-precision and unreliable 3D points can be filtered out, retaining valid map points that can stably reflect the pipeline structure.

[0152] The observed keyframe index is used to establish a many-to-many association between map points and keyframes. The observed pixel coordinates store the pixel position of the map point in each observed keyframe, providing a basis for calculating the reprojection error for pose optimization.

[0153] S320. Based on the cylinder model, perform cylinder constraint verification on the valid map point and calculate the distance from the map point to the cylinder axis. If the deviation between the distance and the cylinder radius is within the preset value, mark it as a cylinder point and record its parameterized coordinates in the circumferential and axial directions of the cylinder.

[0154] Based on the previously fitted cylindrical model, the valid map points generated by binocular triangulation are verified twice to determine whether they conform to the cylindrical geometric features of the inner wall of the pipe. The straight-line distance from each valid map point to the cylindrical axis is calculated, and this distance is compared with the radius of the cylindrical model to screen cylindrical or non-cylindrical points. For map points marked as cylindrical points, their parameterized coordinates in the cylindrical coordinate system must be recorded: circumferential angle θ: the angle between the map point and the reference direction in the plane perpendicular to the axis, with the cylindrical axis as the central axis, describing the position of the point in the circumferential direction of the pipe; axial distance s: the distance from the map point along the cylindrical axis to the midpoint of the axis, describing the position of the point in the length direction of the pipe.

[0155] S330. Construct an optimization window containing the current keyframe, its co-view keyframes and the corresponding map points, and express the camera pose with Lie algebra. The vertices include the keyframe poses and map point coordinates, and the edges are observation constraints. Add additional cylindrical constraint edges to the cylindrical points, constraining their distance to the axis to be equal to the cylinder radius, and setting the weight to 1.5 times that of the ordinary observation edge. Use the g2o optimization library for sparse BA solution, iterate until the root mean square of the reprojection error is less than 1.2 pixels or the number of iterations reaches 30, and output the optimized pose and map point coordinates.

[0156] The current keyframe, several keyframes with the highest degree of co-viewing with it, and the map points observed by these keyframes are selected to form a local optimization window.

[0157] Define the vertices and edges of the optimization model. The vertices are the variables that need to be solved during the optimization process, including: Keyframe pose: Use Lie algebra to represent it, convert the nonlinear rotation matrix into a vector in linear space, and simplify the optimization calculation. Map point coordinates: Expressed in three-dimensional space coordinates (x, y, z), as the spatial position parameters to be optimized. The edges describe the constraint relationship between vertices, that is, the source of the optimization objective function: Observation constraint edge: For each map point, its projected position in the keyframe image should be consistent with the actual observed pixel position of the feature point. This constraint is quantified by the reprojection error. Cylinder constraint edge: An additional structural constraint that forces the distance from the cylinder point to the cylinder axis to be equal to the pipe radius. The weight of this constraint is set to 1.5 times that of the ordinary observation edge, which means that the geometric structure constraints of the pipeline are given priority in the optimization, and the contribution of structured features to accuracy is enhanced.

[0158] The g2o optimization library is used for sparse BA solution. Sparse BA takes advantage of the fact that map points are only observed by a few keyframes. It greatly reduces the amount of computation through sparse matrix operations, making it suitable for real-time systems. Taking the current pose and map point coordinates as the initial values, the pose and map point parameters are continuously corrected by iteratively minimizing the sum of squared residuals of all constrained edges. Convergence judgment: The iteration stops when any of the following conditions are met: the root mean square of the reprojection error of all observation points is less than 1.2 pixels; the number of iterations reaches 30. The optimized keyframe poses and map point coordinates eliminate the accumulated errors in the local range, making the local map more accurate and self-consistent.

[0159] Figure 4 This is a flow chart of a global BA method provided by the third embodiment of the present invention. Figure 4 As shown, the method includes:

[0160] S410: Input the feature point descriptor of the current key frame into the XFeat database, and use the approximate nearest neighbor search algorithm to find the top 20 historical key frames with the highest matching degree with the current frame features as candidate loop frames.

[0161] Descriptors of feature points in the current keyframe are extracted and input into the XFeat feature database, which stores historical data. Using an approximate nearest neighbor search algorithm, the similarity between the current frame's features and those of all historical keyframes in the database is quickly calculated, identifying the top 20 historical keyframes with the highest matching scores. Historical keyframes are considered candidate loop frames, meaning the robot may have previously reached the corresponding locations in these frames, and are used to verify loop formation.

[0162] S420. For the candidate loop frame, estimate the similarity transformation matrix from the feature matching pairs using the RANSAC algorithm, where the inlier determination threshold is set to 3 pixels. When the number of inliers exceeds 30% of the total number of matching pairs and the transformation matrix satisfies the homography constraint, the candidate frame is confirmed to be a valid loop frame. Calculate the cumulative error based on the similarity transformation matrix, and the error value is the root mean square of the reprojection errors of all inliers.

[0163] For candidate loop frames, the RANSAC algorithm is used to estimate the similarity transformation matrix from feature matching pairs, with a threshold of 3 pixels set as the inlier threshold. When the number of inliers exceeds 30% of the total matching pairs and the transformation matrix conforms to the homography geometric constraints, the candidate frame is considered a valid loop frame, indicating that the robot has indeed returned to the position it previously reached. Based on the similarity transformation matrix, the root mean square of the reprojection error of all inliers is calculated as the cumulative error before and after the loop, reflecting the degree of deviation in the pose estimate.

[0164] S430. Construct a global optimization graph containing all keyframe poses and map points, add the similarity transformation obtained by loop closure detection to the graph as a strong constraint, and dynamically adjust the constraint weight according to the accumulated error; use the sparse BA algorithm for optimization, and use the weighted sum of the reprojection error and the loop constraint error as the objective function. During the iterative process, perform a marginalization operation every 10 iterations to remove redundant variables; stop the optimization when the change in the objective function between two consecutive iterations is less than 1e-6 or the number of iterations reaches 80.

[0165] A global optimization graph containing all keyframe poses and map points is constructed, and the similarity transformation obtained by loop closure detection is added as a strong constraint. The constraint weight is dynamically adjusted with the accumulated error.

[0166] The sparse BA algorithm is used for optimization, with the weighted sum of the reprojection error and the loop closure constraint error as the optimization target. The redundant variables are eliminated by marginalization every 10 iterations to reduce the amount of calculation.

[0167] The algorithm stops when the objective function change between two consecutive iterations is less than 1e-6 or when the iteration reaches 80 times, and finally outputs a globally consistent pose and map to eliminate the accumulated error.

[0168] Example 4

[0169] Figure 5 This is a schematic diagram of the structure of a pipe network inspection robot positioning device provided by the fourth embodiment of the present invention. Figure 5 As shown, the device includes:

[0170] The image acquisition unit 510 is used to acquire images through the binocular camera set by the detection robot, transmit the left eye image and the right eye image in the form of topics respectively, and use the accelerated feature network to extract feature points and descriptors of the left eye image and the right eye image.

[0171] The initial pose determination unit 520 is configured to generate an initial pipeline map using the SLAM system and determine the initial pose of the camera through a tracking thread; and optimize the initial pose using the feature points and descriptors.

[0172] The cylinder construction unit 530 is configured to construct a cylinder according to the initialized pipeline map and insert key frames that meet the conditions into a local mapping thread.

[0173] The local BA unit 540 is configured to generate map points and cylinder points using the received key frames by using the local mapping thread, and perform local BA.

[0174] The global BA unit 550 is used to use the closed-loop detection thread to search for candidate loop frames through the XFeat database, calculate the similarity transformation to obtain the cumulative error, perform global BA to optimize the map, and locate the detection robot through the optimized map.

[0175] The pipe network inspection robot positioning device provided in the embodiment of the present invention can execute the pipe network inspection robot positioning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0176] Example 5

[0177] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0178] like Figure 6 As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0179] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0180] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the pipeline network inspection robot positioning method.

[0181] In some embodiments, the pipe network inspection robot positioning method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the pipe network inspection robot positioning method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the pipe network inspection robot positioning method via any other suitable means (e.g., via firmware).

[0182] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0183] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0184] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0187] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0188] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0189] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A pipe network inspection robot positioning method, characterized in that: include: The binocular camera set up by the detection robot acquires images and transmits the left eye image and the right eye image in the form of topics respectively; Accelerating the feature network to extract feature points and descriptors of the left and right images; Use the SLAM system to generate the initial pipeline map and determine the initial position of the camera through the tracking thread; Optimizing the initial pose using the feature points and descriptors; Construct a cylinder according to the initialized pipeline map, and insert the keyframes that meet the conditions into the local mapping thread; Using the local mapping thread to generate map points and cylinder points using the received key frames, and perform local BA; The closed-loop detection thread is used to search for candidate loop frames through the XFeat database, calculate the similarity transformation to obtain the cumulative error, perform the global BA optimization map, and locate the detection robot through the optimized map.

2. The method according to claim 1, characterized in that The left eye image and the right eye image acquired by the binocular camera are transmitted through independent communication topics respectively. The topic format adopts the standard message type of the Robot Operating System ROS, and the transmission frame rate is the same as the acquisition frame rate of the binocular camera.

3. The method according to claim 1, characterized in that The acceleration feature network is an end-to-end network model based on deep learning. Its input is the RGB data of the left and right images, and its output includes: A key point heat map representing the probability distribution of the positions of feature points in the image in the form of a two-dimensional matrix, wherein a higher value of the first matrix element in the key point heat map indicates a greater possibility that a feature point exists at that position; A reliability heatmap of the same size as the key point heatmap, wherein the second matrix element value of the reliability heatmap reflects the reliability score of the feature point at the corresponding position, and the score range is [0, 1], and the closer the value is to 1, the higher the reliability; Descriptor set, each feature point corresponds to a 128-dimensional binary descriptor vector, which is used for matching between feature points.

4. The method according to claim 3, characterized in that The extracting feature points of the left image and the right image by using an accelerated feature network includes: The confidence score of each feature point in the key point heat map and the reliability heat map is calculated using the following formula: ； in, is the normalized score of a feature point in the key point heat map, is the normalized score of a feature point in the reliability heat map, and is the weight coefficient, and + =1; The top N feature points with the highest scores are selected, and the evenly distributed feature points are screened by the grid division method. The image is divided into M×M grids, and no more than P feature points are retained in each grid.

5. The method according to claim 1, wherein The method of generating an initial pipeline map using the SLAM system and determining the initial position of the camera through the tracking thread includes: Selecting a preset number of binocular images, calculating the basic matrix through feature point matching, decomposing the initial rotation matrix and translation vector, and triangulating the generated three-dimensional point cloud to form the initialized pipeline map by combining the binocular camera intrinsic parameters and baseline distance; Determine whether there is a previous frame image. If so, predict the current pose of the binocular camera based on the constant speed motion model as the initial value, match the feature points of the current frame with the previous frame, use the RANSAC-PnP algorithm to solve the pose and iteratively optimize; If the initial pose is determined for the first time after the SLAM system is started, the initial pose is obtained by filtering the inliers from the feature point matching pairs using a random sampling consistency algorithm, calculating the essential matrix and decomposing the matrix.

6. The method according to claim 1, characterized in that Optimizing the initial pose by using the feature points and descriptors includes: Selecting map points corresponding to a preset number of keyframes with the highest degree of co-viewing with the current frame from the local map, extracting their descriptors to form a local feature library, and associating each map point with observation information of at least a preset number of keyframes; A two-way matching mechanism is used to first match the feature point descriptor of the current frame with the local feature library to obtain the initial matching pairs. Then, for each matching pair, the projection consistency between the feature point of the current frame and the map point in the corresponding key frame is verified, and matching pairs with projection errors exceeding the preset number of pixels are eliminated. Taking the initial pose as the starting point of iteration, a pose optimization objective function based on Lie algebra is constructed. The objective function is the weighted sum of the reprojection errors of all matching points, where the weights are dynamically adjusted according to the number of observations of the map points. The Gauss-Newton algorithm is used for iterative optimization. The Jacobian matrix is ​​calculated after each iteration to update the pose parameters. The optimization is stopped when the number of iterations reaches the preset number or the pose change between two adjacent iterations is less than 1e-6 radians and 1e-3 meters, and the optimized pose is output.

7. The method according to claim 1, characterized in that The step of constructing a cylinder according to the initialized pipeline map and inserting a key frame that meets the conditions into a local mapping thread includes: Based on the point cloud data of the initialized pipeline map, a random sampling consistency algorithm is used to fit the cylinder parameters, which include the unit direction vector of the cylinder axis, the coordinates of the axis midpoint, and the cylinder radius. The distance from each point in the point cloud to the candidate axis is calculated using the point-to-line distance formula. Inliers with a distance less than a preset threshold are selected. When the proportion of inliers exceeds a preset proportion of the total number of point clouds, the candidate cylinder is determined to be a valid model. The current frame is determined to be a keyframe and inserted into the local mapping thread when it meets the following conditions: The translation distance from the previous keyframe exceeds 1 / 5 of the pipe diameter or the rotation angle exceeds 5 degrees; The number of matching pairs between the feature points extracted in the current frame and the local map feature points is no less than 80, and the distribution of the matching points in the image covers at least 8 preset grid areas; The root mean square value of the reprojection error after pose optimization is less than 1.5 pixels; The keyframe inserted into the local mapping thread contains the following information: optimized camera pose, feature point set and corresponding descriptors, image acquisition timestamp, co-viewing relationship with adjacent keyframes, and projection position parameters of the keyframe in the cylindrical model.

8. The method according to claim 7, characterized in that The local mapping thread generates map points and cylinder points using the received key frames and performs local BA, including: For the matching feature point pairs between key frames and adjacent key frames, the 3D coordinates are calculated through binocular triangulation. Points with a reprojection error of less than 2 pixels and a parallax angle greater than 1 degree are selected as valid map points. Each map point is associated with its observed key frame index and observed pixel coordinates. Based on the cylindrical model, the cylindrical constraint verification is performed on the valid map point, and the distance from the map point to the cylinder axis is calculated. If the deviation between the distance and the cylinder radius is within a preset value, it is marked as a cylinder point, and its parametric coordinates in the circumferential and axial directions of the cylinder are recorded; Construct an optimization window containing the current keyframe, its co-view keyframes and corresponding map points, express the camera pose with Lie algebra, the vertices include the keyframe poses and map point coordinates, and the edges are observation constraints; add additional cylindrical constraint edges to the cylindrical points, constraining their distance to the axis to be equal to the cylinder radius, and setting the weight to 1.5 times that of the ordinary observation edge; use the g2o optimization library for sparse BA solver, iterate until the root mean square of the reprojection error is less than 1.2 pixels or the number of iterations reaches 30, and output the optimized pose and map point coordinates.

9. The method according to claim 1, characterized in that The loop closure detection thread searches for candidate loop frames through the XFeat database, calculates similarity transformation to obtain cumulative error, and performs global BA optimization map, including: The feature point descriptor of the current key frame is input into the XFeat database, and the top 20 historical key frames with the highest matching degree with the current frame features are found as candidate loop frames through the approximate nearest neighbor search algorithm; For the candidate loop frame, the similarity transformation matrix is ​​estimated from the feature matching pairs using the RANSAC algorithm, where the inlier threshold is set to 3 pixels. When the number of inliers exceeds 30% of the total number of matching pairs and the transformation matrix satisfies the homography constraint, the candidate frame is confirmed to be a valid loop frame. The cumulative error is calculated based on the similarity transformation matrix, and the error value is the root mean square of the reprojection error of all inliers. A global optimization graph containing all keyframe poses and map points is constructed, and the similarity transformation obtained by loop closure detection is added to the graph as a strong constraint. The constraint weights are dynamically adjusted according to the accumulated error. A sparse BA algorithm is used for optimization, with the weighted sum of the reprojection error and the loop closure constraint error as the objective function. During the iterative process, a marginalization operation is performed every 10 iterations to remove redundant variables. The optimization is stopped when the change in the objective function between two consecutive iterations is less than 1e-6 or the number of iterations reaches 80.

10. A pipe network inspection robot positioning device, characterized in that: include: The image acquisition unit is used to acquire images through the binocular camera set by the detection robot, and transmit the left eye image and the right eye image in the form of topics respectively; Accelerating the feature network to extract feature points and descriptors of the left and right images; An initial pose determination unit, used to generate an initial pipeline map using a SLAM system and determine the initial pose of the camera through a tracking thread; Optimizing the initial pose using the feature points and descriptors; A cylinder construction unit, configured to construct a cylinder according to the initialized pipeline map and insert key frames that meet the conditions into a local mapping thread; A local BA unit, configured to generate map points and cylinder points using the received keyframes using the local mapping thread, and perform local BA; The global BA unit is used to use the closed-loop detection thread to find candidate loop frames through the XFeat database, calculate the similarity transformation to obtain the cumulative error, perform global BA to optimize the map, and locate the detection robot through the optimized map.

11. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the pipe network inspection robot positioning method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the pipe network inspection robot positioning method according to any one of claims 1 to 9 when executed.

13. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the pipe network inspection robot positioning method according to any one of claims 1 to 9.

Images