A pipe network detection robot positioning method, device, equipment, medium and product

Abstract

The application discloses a kind of pipe network detection robot positioning method, device, equipment, medium and product.The image is obtained by detecting the binocular camera of robot setting, respectively with topic form transmission left eye image and right eye image;Utilize acceleration feature network to extract the feature point and descriptor of left eye image and right eye image;Utilize SLAM system to generate initialization pipeline map and determine the initial pose of camera by tracking thread;Optimize initial pose by feature point and descriptor;According to initialization pipeline map, construct cylinder, and insert the key frame meeting the condition into local mapping thread;Utilize local mapping thread to generate map point and cylindrical point using the received key frame, execute local BA;Utilize loop detection thread to find candidate loop frame by XFeat database, calculate similar transformation to obtain cumulative error, execute global BA to optimize map and position detection robot.The scheme of the application can realize the accurate positioning of pipe network detection robot in complex pipeline scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pipe network positioning, and in particular to a pipe network detection robot positioning method, device, equipment, medium and product. BACKGROUND

[0002] With the rapid development of urban pipe network systems, pipe network detection robots are increasingly widely used in pipe inspection, fault positioning and other fields. Precise positioning is the core requirement for pipe network detection robots to achieve autonomous inspection, and the visual simultaneous localization and mapping (SLAM) technology has become the mainstream solution for robot positioning due to its low cost and high information capacity.

[0003] The ORB-SLAM series algorithm has outstanding performance in the traditional visual SLAM field due to its efficient feature processing capability, multi-thread architecture and loop detection function, and is widely used in positioning tasks in various scenarios.

[0004] However, in the special scenario of a pipe, the limitations of the traditional feature point method gradually appear: in the pipe network detection scenario, the robot needs to operate in a 2 / 3 water, half water or water-free environment, and faces complex challenges such as sparse texture on the inner wall of the pipe, uneven lighting, metal reflection and repeated structure, which puts high requirements on the robustness and positioning accuracy of the SLAM technology. SUMMARY

[0005] The present application provides a pipe network detection robot positioning method, device, equipment, medium and product, which can realize precise positioning of a pipe network detection robot in a complex pipe scenario.

[0006] According to an aspect of the present application, a pipe network detection robot positioning method is provided, comprising:

[0007] An image is acquired by a binocular camera provided by the detection robot, and the left eye image and the right eye image are transmitted in the form of a topic respectively; the feature points and descriptors of the left eye image and the right eye image are extracted using an accelerated feature network;

[0008] An initial pipe map is generated using a SLAM system, and the initial pose of the camera is determined through a tracking thread; the initial pose is optimized using the feature points and descriptors;

[0009] A cylinder is constructed according to the initial pipe map, and key frames that meet the conditions are inserted into a local mapping thread;

[0010] The local mapping thread generates map points and cylinder points using the received key frames, and performs local BA;

[0011] A closed-loop detection thread searches for a candidate loop frame through an XFeat database, calculates a cumulative error through a similar transformation, performs a global BA optimization map, and positions the detection robot through the optimized map.

[0012] Optionally, the left-eye image and the right-eye image acquired by the binocular camera are transmitted through independent communication topics respectively, the topic format adopts a robot operating system (ROS) standard message type, and the transmission frame rate is the same as the acquisition frame rate of the binocular camera.

[0013] Optionally, the accelerated feature network is an end-to-end network model based on deep learning, the input of the network is RGB data of the left-eye image and the right-eye image, and the output of the network includes:

[0014] A keypoint heat map in a two-dimensional matrix form represents a position probability distribution of a feature point in an image, wherein a higher value of a first matrix element in the keypoint heat map indicates a higher possibility that a feature point exists at the position;

[0015] A reliability heat map of the same size as the keypoint heat map, a second matrix element value of the reliability heat map reflects a reliability score of a feature point at a corresponding position, the score range is [0, 1], and a value closer to 1 indicates higher reliability;

[0016] A descriptor set, each feature point corresponds to a 128-dimensional binary descriptor vector, and is used for matching between feature points.

[0017] Optionally, the accelerated feature network is an end-to-end network model based on deep learning, the input of the network is RGB data of the left-eye image and the right-eye image, and the output of the network includes:

[0018] A confidence score of each feature point in the keypoint heat map and the reliability heat map is calculated through the following formula:

[0019] ;

[0020] wherein, is a normalized score of a feature point in the keypoint heat map, is a normalized score of a feature point in the keypoint heat map, and is a weight coefficient, and + = 1.

[0021] The top N feature points are selected according to the scores, the grid division method is used to select uniformly distributed feature points, and the image is divided into MxM grids, and no more than P feature points are reserved in each grid.

[0022] Optionally, the SLAM system generates an initialization pipeline map and determines an initial pose of the camera through a tracking thread, including:

[0023] selecting preset frame number of binocular images, calculating a fundamental matrix through feature point matching, decomposing to obtain an initial rotation matrix and a translation vector, combining binocular camera intrinsic parameters and baseline distance, triangulation to generate a three-dimensional point cloud to constitute the initial pipeline map;

[0024] determining whether there is a previous frame image currently, if there is, predicting the current pose of the binocular camera as an initial value based on a constant speed motion model, solving the pose by using RANSAC-PnP algorithm through feature point matching of the current frame and the previous frame and iterative optimization;

[0025] if the initial pose is determined for the first time after the SLAM system is started, filtering inliers from feature point matching pairs by using a random sample consensus algorithm, calculating an essential matrix and decomposing to obtain the initial pose.

[0026] Optionally, the initial pose is optimized by the feature points and the descriptors, including:

[0027] selecting preset number of map points corresponding to key frames with the highest degree of co-visibility from a local map, extracting their descriptors to form a local feature library, and each map point is associated with observation information of at least preset number of key frames;

[0028] using a bidirectional matching mechanism, first matching the current frame feature point descriptors with the local feature library to obtain initial matching pairs, then verifying the projection consistency of the current frame feature points and the map points in the corresponding key frames for each matching pair, and eliminating matching pairs with projection errors exceeding a preset number of pixels;

[0029] taking the initial pose as an iterative starting point, constructing a pose optimization objective function based on Lie algebra, and the objective function is a weighted sum of all matching point re-projection errors, wherein the weight is dynamically adjusted according to the observation times of the map points;

[0030] using a Gauss-Newton algorithm for iterative optimization, calculating a Jacobian matrix to update the pose parameters after each iteration, stopping when the number of iterations reaches a preset number or the pose change between adjacent two iterations is less than 1e-6 radian and 1e-3 meters, and outputting the optimized pose.

[0031] Optionally, the initial pipeline map is used to construct a cylinder, and key frames meeting the conditions are inserted into a local mapping thread, including:

[0032] Based on the point cloud data of the initialized pipeline map, a random sample consensus algorithm is used to fit a cylindrical parameter, the cylindrical parameter including a unit direction vector of a cylindrical axis, a coordinate of a midpoint of the axis and a cylindrical radius; wherein, a distance of each point in the point cloud to a candidate axis is calculated through a point-to-line distance formula, and an inner point with a distance less than a preset threshold is screened, when a proportion of the inner points exceeds a preset proportion value of the total number of the point cloud, the candidate cylindrical is determined as an effective model;

[0033] The current frame is determined as a key frame and inserted into a local mapping thread when the current frame satisfies the following conditions:

[0034] A translation distance between the current frame and a previous key frame exceeds 1 / 5 of a diameter of the pipeline or a rotation angle exceeds 5 degrees;

[0035] A number of matching pairs of feature points in the current frame and local map feature points is not less than 80, and a distribution of the matching points in an image covers at least 8 preset grid regions;

[0036] A root mean square value of a re-projection error after pose optimization is less than 1.5 pixels;

[0037] The key frame inserted into the local mapping thread includes the following information: an optimized camera pose, a feature point set and a corresponding descriptor, an image acquisition timestamp, a co-view relationship with a neighboring key frame, and a projection position parameter of the key frame in the cylindrical model.

[0038] Optionally, the local mapping thread generates a map point and a cylindrical point by using the received key frame, and performs local BA, including:

[0039] For matching feature point pairs of a key frame and a neighboring key frame, three-dimensional coordinates are calculated through binocular triangulation, points with a re-projection error less than 2 pixels and a parallax angle greater than 1 degree are screened as effective map points, and an observed key frame index and observed pixel coordinates of each map point are associated;

[0040] Based on the cylindrical model, cylindrical constraint verification is performed on the effective map points, a distance of the map point to the cylindrical axis is calculated, if a deviation of the distance from the cylindrical radius is within a preset value, the map point is marked as a cylindrical point, and parameterized coordinates of the cylindrical point in a circumferential direction and an axial direction are recorded;

[0041] An optimization window including a current key frame, a co-view key frame and a corresponding map point is constructed, a camera pose is represented by Lie algebra, vertices include key frame poses and map point coordinates, and edges are observation constraints; a cylindrical constraint edge is additionally added for a cylindrical point, the cylindrical constraint edge constrains a distance of the cylindrical point to the axis to be equal to the cylindrical radius, and a weight is set to be 1.5 times of a weight of a normal observation edge; a g2o optimization library is used for sparse BA solving, and iteration is performed until a root mean square of a re-projection error is less than 1.2 pixels or an iteration number reaches 30 times, and an optimized pose and map point coordinates are output.

[0042] Optionally, the closed loop detection thread searches for candidate loop frames through the XFeat database, calculates the accumulated error of the similarity transformation, and performs global BA optimization of the map, including:

[0043] The feature point descriptors of the current key frame are input into the XFeat database, and the first 20 historical key frames with the highest feature matching degree with the current frame are searched as candidate loop frames through the approximate nearest neighbor search algorithm.

[0044] For the candidate loop frame, a similarity transformation matrix is estimated from the feature matching pairs through the RANSAC algorithm, wherein the inlier determination threshold is set to 3 pixels; when the number of inliers exceeds 30% of the total number of matching pairs and the transformation matrix satisfies the homography constraint, the candidate frame is confirmed as an effective loop frame; the accumulated error is calculated based on the similarity transformation matrix, and the error value is the root mean square of the re-projection error of all inliers.

[0045] A global optimization graph containing all key frame poses and map points is constructed, and the similarity transformation obtained by loop detection is added as a strong constraint to the graph, and the constraint weight is dynamically adjusted according to the accumulated error; a sparse BA algorithm is used for optimization, and the weighted sum of the re-projection error and the loop constraint error is used as the objective function, and an edge operation is performed every 10 iterations in the iteration process to remove redundant variables; when the change of the objective function of two consecutive iterations is less than 1e-6 or the number of iterations reaches 80, the optimization is stopped.

[0046] According to another aspect of the present application, a pipe network detection robot positioning device is provided, comprising:

[0047] An image acquisition unit is configured to acquire images through a binocular camera arranged by the detection robot, and transmit left eye images and right eye images in the form of topics respectively; and extract feature points and descriptors of the left eye images and the right eye images using an accelerated feature network.

[0048] An initial pose determination unit is configured to generate an initial pipe map using a SLAM system and determine an initial pose of a camera through a tracking thread; and optimize the initial pose through the feature points and the descriptors.

[0049] A cylinder construction unit is configured to construct a cylinder according to the initial pipe map, and insert key frames meeting a condition into a local mapping thread.

[0050] A local BA unit is configured to generate map points and cylinder points using the local mapping thread using the received key frames, and perform local BA.

[0051] A global BA unit is configured to search for candidate loop frames through an XFeat database using a loop detection thread, calculate the accumulated error of the similarity transformation, perform global BA optimization of the map, and position the detection robot through the optimized map.

[0052] According to another aspect of the present application, there is provided an electronic device comprising:

[0053] at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the pipe network detection robot positioning method according to any one of the embodiments of the present application.

[0054] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to implement the pipe network detection robot positioning method according to any one of the embodiments of the present application when executed by the processor.

[0055] According to another aspect of the present application, there is provided a computer program product comprising a computer program for implementing the pipe network detection robot positioning method according to any one of the embodiments of the present application when executed by a processor.

[0056] The technical solution of the embodiments of the present application combines deep learning with pipe geometry prior information, and solves the problem that the positioning accuracy and robustness of the traditional visual SLAM method are seriously affected due to the repeated pipe structure, sparse pipe wall texture, and uneven pipe light. In a complex pipe environment, the positioning accuracy can be improved to centimeter level, the positioning error is reduced, and the positioning accuracy is improved compared with the traditional positioning method. The embodiments of the present application are suitable for wide deployment and promotion of urban pipe systems, and can be applied to similar scenes such as water diversion tunnels, culverts, and tunnels.

[0057] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0059] Figure 1 is a flowchart of a pipe network detection robot positioning method provided by the first embodiment of the present application;

[0060] Figure 2 is a flowchart of an initial pose determination method provided by the second embodiment of the present application;

[0061] Figure 3 is a flow chart of a local BA method provided by Embodiment Three of the present application;

[0062] Figure 4 is a flow chart of a global BA method provided by Embodiment Three of the present application;

[0063] Figure 5 is a structural schematic diagram of a pipe network detection robot positioning device provided by Embodiment Four of the present application;

[0064] Figure 6 is a structural schematic diagram of an electronic device for implementing a pipe network detection robot positioning method according to an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts should fall within the scope of the present application.

[0066] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product, or device.

[0067] Embodiment One

[0068] Figure 1 is a flow chart of a pipe network detection robot positioning method provided by Embodiment One of the present application. The present embodiment can be applicable to the positioning of a pipe network detection robot in a complex pipe scene. The method can be executed by a pipe network detection robot positioning device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. As shown in the figure, the method comprises: Figure 1

[0069] ​S110, acquire images by detecting binocular cameras arranged by the robot, respectively transmit left eye images and right eye images in the form of topics, and use an accelerated feature network to extract feature points and descriptors of the left eye images and the right eye images.

[0070] The binocular camera carried by the detection robot synchronously collects left eye and right eye images of the internal environment of the pipeline, similar to human binocular vision, and through the communication mechanism of the robot operating system, the left eye and right eye images are respectively encapsulated as independent topics for real-time transmission.

[0071] The feature points are representative key pixel points identified from the images, such as joints, protrusions and other significant features of the inner wall of the pipeline, which remain stable under different viewing angles and illumination; the descriptor generation generates a high-dimensional vector descriptor for each feature point, which is used to describe the local image features of the point, such as texture, gradient, etc., so that the same physical points in different images can be associated through descriptor matching.

[0072] In the embodiment of the application, the left eye images and the right eye images obtained by the binocular camera are transmitted through independent communication topics respectively, the topic format adopts the ROS standard message type of the robot operating system, and the transmission frame rate is the same as the acquisition frame rate of the binocular camera.

[0073] The left eye images and the right eye images are transmitted through two independent ROS topics. The core of binocular vision is to calculate the depth information through the disparity of left and right images, and independent topics can ensure that the left and right images are read by the front-end processing module of the SLAM system respectively, avoiding data aliasing.

[0074] The sensor_msgs / Image standard message format defined by ROS is used to transmit images, which can be directly parsed by the image processing library in the ROS ecosystem, and at the same time ensures the compatibility between the camera driving node and the SLAM algorithm node, without the need to define the data format.

[0075] The transmission frame rate is consistent with the camera acquisition frame rate, ensuring the real-time and time synchronization of the image data: on the one hand, it avoids image accumulation or loss due to low transmission frame rate, ensuring that the SLAM system can estimate the pose based on the latest image; on the other hand, the left and right images are strictly aligned in time, reducing the error caused by the asynchronization of left and right images due to robot movement.

[0076] In the embodiment of the application, the accelerated feature network is an end-to-end network model based on deep learning, and the input of the network is the RGB data of the left and right eye images, and the output of the network includes:

[0077] The key point heat map in the form of a two-dimensional matrix represents the position probability distribution of the feature points in the image, wherein the higher the value of the first matrix element in the key point heat map, the greater the possibility of the existence of the feature point at that position.

[0078] A reliability heat map with the same size as the key point heat map, the second matrix element value of the reliability heat map reflects the reliability score of the corresponding position feature point, the score range is [0, 1], and the value closer to 1 indicates higher reliability;

[0079] A descriptor set, each feature point corresponds to a 128-dimensional binary descriptor vector, which is used for matching between feature points.

[0080] Specifically, the accelerated feature network is an end-to-end deep learning model, which does not need to manually design feature extraction rules, directly takes the RGB color images collected by the left and right cameras as inputs, and is suitable for color information extraction under complex illumination in the pipeline.

[0081] The key point heat map is a two-dimensional probability matrix related to the size of the input image. When the input image is 1280*720, the heat map may be 320*180 after scaling. The numerical value of each element in the matrix represents the probability of the existence of a feature point, such as a pipeline joint, a protrusion, a stain, or other notable structures that can be used for positioning, at the corresponding image position. The higher the value, the more likely it is that the position will become a stable feature point.

[0082] The reliability heat map corresponds completely in size to the key point heat map, and the element value (0-1 range) thereof represents the reliability of the feature point at the corresponding position. The closer the value is to 1, the more stable the feature point is under changes in viewing angle, fluctuations in illumination, and slight occlusions, and the feature mismatch risk caused by pipeline internal reflections, water stains, and other disturbances can be effectively reduced.

[0083] The 128-dimensional binary descriptor set generates a binary vector for each feature point screened out, encodes local information such as texture, gradient, and color distribution around the feature point, and realizes fast matching of feature points between different images. Binary descriptors are more lightweight than floating-point descriptors, can accelerate the real-time matching efficiency of the SLAM system, and are suitable for the real-time requirements of robot inspection.

[0084] In the embodiment of the application, the accelerated feature network is used to extract feature points of the left eye image and the right eye image, comprising:

[0085] Each feature point in the key point heat map and the reliability heat map is scored by confidence through the following formula:

[0086] ;

[0087] Wherein, is the normalized score of a feature point in the key point heat map, is the normalized score of a feature point in the key point heat map, and is a weight coefficient, and + = 1.

[0088] Select the top N feature points according to the score, and select evenly distributed feature points by grid division method. Divide the image into MxM grids, and retain no more than P feature points in each grid.

[0089] is the normalized score of the feature point extracted from the key point heat map, reflecting the probability of the position as a feature point. Correspondingly, is the normalized score of the feature point extracted from the reliability heat map, reflecting the stability of the feature point in a complex environment. and is used to balance the importance of key point probability and reliability. For example, in a sparse scene of pipeline texture, can be increased , and feature points with high reliability are preferentially retained.

[0090] Select the top N feature points according to the score, and ensure that the highest scoring candidate points are retained. Divide the image into MxM uniform grids (such as 16x16), and retain at most P feature points in each grid to avoid feature points concentrating in local areas, such as feature points may be densely present at pipeline joints, and ensure that feature points are uniformly distributed in the image. Uniformly distributed feature points can provide more comprehensive spatial constraints for subsequent pose estimation, reduce positioning deviation caused by lack of local features, and especially in long and symmetrical structures such as pipelines, uniformity can improve the stability of rotation and translation estimation.

[0091] S120, using the SLAM system to generate an initial pipeline map and determine the initial pose of the camera through a tracking thread; optimizing the initial pose through feature points and descriptors.

[0092] After the SLAM system is started, the first few frames of images captured by the binocular camera are used to calculate the three-dimensional point cloud through binocular stereo matching, and an initial pipeline map containing three-dimensional point coordinates and preliminary geometric constraints is constructed in combination with pipeline structure prior knowledge.

[0093] The tracking thread obtains the initial pose in the following way:

[0094] If it is the first start without historical data, the fundamental matrix / essential matrix between images is calculated through feature point matching, the rotation and translation parameters of the camera are obtained by decomposition, and the world coordinate system is established with the camera coordinate system of the first frame image as the origin. If the system is restarted or relocated, the feature matching with the existing initial map is used to quickly solve the pose of the current camera in the world coordinate system.

[0095] The feature points and descriptors extracted from the current frame are matched with the three-dimensional points in the initialized map. By comparing the similarity of the descriptors, the corresponding relationship between the image feature points and the spatial three-dimensional points is found. Based on the matching pairs, a re-projection error function is constructed, and the rotation and translation parameters of the camera are iteratively corrected by minimizing the error through a non-linear optimization algorithm. Finally, the re-projection error is controlled within a preset threshold, ensuring the accuracy of the initial pose.

[0096] S130, according to the initialization pipeline map, a cylinder is constructed, and the key frames meeting the conditions are inserted into the local mapping thread.

[0097] From the three-dimensional point cloud of the initialization pipeline map, the cylinder parameters are fitted by algorithm, including the spatial direction of the cylinder axis, the midpoint coordinates of the axis, the cylinder radius, etc. During fitting, points that meet the geometric constraints of the cylinder, such as the distance from the point to the axis being close to the diameter of the pipe, are selected, and noise points, such as abnormal points formed by impurities, bubbles, etc. in the pipe, are removed. The cylinder model adds a structured constraint to the map, and the newly generated map points are verified for compliance with the cylinder characteristics, thereby reducing cumulative errors, especially suitable for long-distance repetitive structure scenes such as pipelines.

[0098] Key frames are key image frames used to construct maps in SLAM systems, containing camera poses, feature points, and other core information. Not all image frames will be selected as key frames.

[0099] The screening conditions usually include:

[0100] Motion distance: compared with the last key frame, the robot moves more than 1 / 5 of the diameter of the pipeline or rotates more than 5°, avoiding redundant frames and ensuring mapping efficiency;

[0101] Feature matching quality: the number of feature matching pairs between the current frame and the local map is sufficient, such as at least 80 pairs, and the distribution is uniform, covering multiple areas of the image;

[0102] Pose accuracy: the re-projection error of the optimized camera pose is small, such as the root mean square <1.5 pixels, ensuring the reliability of the pose.

[0103] Key frames that meet the conditions will be transmitted to the local mapping thread for generating new three-dimensional map points, optimizing the local map structure, and fusing with the cylinder model, gradually constructing a more accurate pipeline environment map.

[0104] S140, using the local mapping thread to generate map points and cylinder points using the received key frames, and performing local BA.

[0105] After receiving the new key frame, the local mapping thread matches it with the neighboring key frames, calculates the 3D coordinates of the matching feature points through the principle of binocular vision triangulation, and generates new map points. These map points are screened to remove points with too large re-projection error or too small parallax angle, ensuring the accuracy of the map points.

[0106] In combination with the constructed pipe cylinder model, the newly generated map points are subjected to geometric constraint verification, and the distance of each map point to the cylinder axis is calculated. If the distance deviates from the theoretical radius of the pipe within the allowable range, it is marked as a cylindrical point and its parameters in the cylindrical coordinate system are recorded. The significance of the cylindrical point lies in strengthening the cylindrical structure constraint of the pipe and reducing the map drift caused by sparse features or noise.

[0107] BA is a nonlinear least squares algorithm for optimizing camera pose and 3D point coordinates. The core is to minimize the re-projection error to improve the consistency of the pose and the map. Unlike global BA that optimizes the entire map, local BA only constructs an optimization window for the current key frame, the neighboring key frames that share the view with it, and the map points observed by these key frames, avoiding the high computational cost of global optimization and ensuring real-time performance.

[0108] S150, use the loop closure detection thread to find candidate loop frames through the XFeat database, calculate the cumulative error by similar transformation, execute global BA optimization of the map, and position the robot through the optimized map.

[0109] When the robot moves a distance in the pipe, it may return to the area it has passed before. At this time, loop closure detection is needed to identify this phenomenon to correct the accumulated positioning error. XFeat is a high-efficiency feature extraction and matching algorithm, and the database constructed by it stores the feature descriptors of historical key frames. The loop closure detection thread compares the feature descriptors of the current key frame with the historical data in the database, finds the top N historical key frames with the highest feature matching degree through approximate nearest neighbor search, and uses them as candidate loop frames. To avoid repeated matching in a short period of time, candidate frames with too close intervals from the current frame are filtered out to ensure the effectiveness of loop closure detection.

[0110] Global BA is a global optimization algorithm that minimizes the overall reprojection error by adjusting the poses of all keyframes and the coordinates of map points, eliminates accumulated errors, and ensures the global consistency of the map. A global optimization model containing all keyframes and map points is constructed, and the similar transformation obtained by loop detection is added as a strong constraint to the model. Sparse BA algorithm is used for iterative optimization until the error converges or the maximum number of iterations is reached. The globally optimized map eliminates the scale drift and pose deviation caused by long-distance movement. The globally optimized map provides an accurate three-dimensional model of the pipeline environment. The robot can match features between the real-time captured image and the map, and solve the current camera pose using PnP algorithm, to achieve accurate positioning based on the globally consistent map. This positioning method is not affected by accumulated errors, and can maintain high positioning accuracy even in long-distance pipeline inspection.

[0111] Embodiment Two

[0112] Figure 2 is a flowchart of an initial pose determination method provided by Embodiment Two of the present application. As shown in Figure 2 , the method comprises:

[0113] S210, select a preset number of binocular images, calculate the fundamental matrix through feature point matching, decompose to obtain an initial rotation matrix and a translation vector, combine the binocular camera intrinsic parameters and the baseline distance, triangulate to generate a three-dimensional point cloud to form an initial pipeline map.

[0114] After the SLAM system is started, a preset number of binocular images are first collected. These images need to meet certain motion constraints to ensure the reliability of subsequent feature matching and geometric calculations. Although a single binocular image can generate a local point cloud, it lacks motion information and is difficult to construct a globally consistent coordinate system. The motion relationship of multiple images can provide a scale and direction reference.

[0115] For the selected image sequence, the feature points and descriptors of each left and right image are extracted through the accelerated feature network, and then feature point matching is performed between adjacent frames to find the corresponding pixel positions of the same physical point in different images in space.

[0116] Based on the matched feature point pairs, the fundamental matrix is estimated by the RANSAC algorithm. The fundamental matrix is an important geometric constraint in binocular vision, which describes the projection relationship between the left and right images. Its essence is to encode the mathematical relationship of camera intrinsic parameters, relative pose, and spatial point coordinates, which can be used to verify the rationality of feature matching. The fundamental matrix contains the relative pose information between the corresponding cameras of two images. By performing SVD on the fundamental matrix, the initial rotation matrix and translation vector of the camera can be solved.

[0117] The feature point coordinates in the pixel coordinate system are converted into the ray direction in the camera coordinate system by using the internal parameters and the baseline distance of the binocular camera. For the matched feature points, the three-dimensional coordinates of the feature points in the world coordinate system are calculated according to the relative poses of adjacent frames through the geometric triangulation principle. This process is performed on all matched points to generate a dense three-dimensional point cloud. The three-dimensional point cloud is organized according to the spatial position, and an initial three-dimensional map containing the key features of the inner wall of the pipeline is formed in combination with the cylindrical prior of the pipeline.

[0118] In S220, it is determined whether the previous frame image exists currently. If the previous frame image exists, the current pose of the binocular camera is predicted as an initial value based on the constant speed motion model. The pose is solved by using the RANSAC-PnP algorithm and iteratively optimized through the feature point matching between the current frame and the previous frame.

[0119] The SLAM system first checks whether the image data of the previous frame processed exists. If the previous frame exists, it indicates that the robot is in continuous motion, and the pose prediction can be performed by using the time continuity of the motion. In a short time interval, the motion of the robot can be approximated as uniform linear motion and uniform rotational motion. The prediction process is as follows: the predicted pose of the current frame is calculated according to the camera pose of the previous frame and the motion speed of the previous frames. The predicted value is used as the initial value of the subsequent pose solving, which can greatly reduce the search range of the optimization algorithm, improve the solving speed and convergence, and avoid falling into a local optimum. For the images of the current frame and the previous frame, the feature point pairs corresponding to the same physical point in the two frames are found by comparing the similarity of the feature descriptors.

[0120] The PnP algorithm is used to solve the camera pose by using the three-dimensional point and the two-dimensional projection point of the three-dimensional point in the current image. The core of the algorithm is to establish a nonlinear equation set and solve the equation set by using the projection relationship between the space point and the image point.

[0121] The RANSAC algorithm is used to process the false matching in the matching pairs. The pose is solved by randomly sampling part of the matching pairs for multiple times, and then the number of inliers that meet the projection constraint of the pose is counted. The pose with the most inliers is retained as the optimal solution, which greatly improves the robustness of the pose solving. The pose obtained by the RANSAC is used as the initial value, a re-projection error function is constructed, and the pose parameters are further corrected by iteratively minimizing the error through the nonlinear optimization algorithm until the error converges.

[0122] In S230, if the initial pose is determined for the first time after the SLAM system is started, the inliers are screened from the feature point matching pairs by using the RANSAC algorithm, and the essential matrix is calculated and decomposed to obtain the initial pose.

[0123] When the SLAM system is just started, there is no any historical pose data or map information, and the motion information of the previous frame cannot be used to predict the pose like the continuous tracking stage. At this time, it is necessary to completely rely on the initial image data to calculate the initial pose of the camera from zero. For the first two frames of binocular images collected after the SLAM system is started, the feature extraction network is used to obtain the feature points and descriptors of each frame, and then the feature point matching pairs between the two frames are obtained through descriptor matching. The random sample consensus (RANSAC) algorithm is used to screen the inliers.

[0124] Due to the inevitable existence of false matching in the feature matching process, the RANSAC algorithm is used to screen the inliers. A small number of matching pairs are randomly selected several times, and it is assumed that these matching pairs are inliers and a geometric model is calculated. The number of matching pairs that meet the constraint of the model is counted, and the matching pairs corresponding to the model with the most inliers are reserved as valid data, and the outliers that do not meet the constraint are removed.

[0125] Based on the screened inliers, the essential matrix corresponding to the two frames of images is calculated. The essential matrix is an important geometric constraint in binocular vision for describing the relative pose between the coordinate systems of two cameras. Its mathematical expression encodes the relationship between the rotation matrix and the translation vector between the two frames, and is related to the camera intrinsic parameters. The essential matrix contains the relative pose information between the two frames, and by singular value decomposition, the possible combination of rotation matrix and translation vector can be obtained. Combined with the actual physical constraint, the unique reasonable rotation matrix and translation vector are selected from these solutions, which is the initial pose.

[0126] In the embodiment of the application, the initial pose is optimized by the feature points and descriptors, including:

[0127] A preset number of map points corresponding to the key frames with the highest degree of co-visibility with the current frame are selected from the local map, and their descriptors are extracted to form a local feature library. Each map point is associated with at least a preset number of observation information of key frames;

[0128] A bidirectional matching mechanism is adopted. First, the current frame feature point descriptors are matched with the local feature library to obtain initial matching pairs. Then, the projection consistency of the current frame feature points and the map points in the corresponding key frames is verified for each matching pair, and the matching pairs with projection errors exceeding a preset number of pixels are removed.

[0129] Taking the initial pose as the starting point of iteration, a pose optimization objective function based on Lie algebra is constructed, and the objective function is the weighted sum of the re-projection errors of all matching points, wherein the weight is dynamically adjusted according to the number of observations of the map points.

[0130] The Gauss-Newton algorithm is used for iterative optimization. The Jacobian matrix is calculated after each iteration to update the pose parameters. The iteration is stopped when the number of iterations reaches a preset number or the pose change between adjacent two iterations is less than 1e-6 radians and 1e-3 meters, and the optimized pose is output.

[0131] The preset number of key frames with the highest degree of co-visibility with the current frame are screened from the local map, and the three-dimensional map points and their descriptors corresponding to the key frames are extracted to form a local feature library. The degree of co-visibility refers to the number of shared feature points. The higher the co-visibility, the stronger the spatial correlation between the key frames and the current frame, and the greater the reference value of the map points to the current pose optimization. Each map point needs to be associated with the observation information of at least a preset number of key frames to ensure that the map point is verified by multiple perspectives and has higher accuracy.

[0132] The bidirectional matching mechanism first matches the descriptors of the feature points of the current frame with the descriptors of the map points in the local feature library to obtain initial matching pairs. Then, the projection consistency of each initial matching pair is verified, the three-dimensional map points are projected back to the key frame image according to the pose of the key frame associated with the three-dimensional map points, and the deviation of the projection position from the original observed feature points in the key frame image is checked. If the projection error exceeds a preset pixel number, it is determined as a false match and is removed, and only the matching pairs with errors meeting the requirements are retained.

[0133] The camera pose is converted into Lie algebra form, the optimization problem of the pose is converted into linearized solving in vector space, and iterative calculation is simplified. The weighted sum of the re-projection errors of all matching points is taken as the optimization objective. The pixel position of the three-dimensional map point projected onto the image by the current camera pose is deviated from the actual position of the corresponding feature point in the current frame. The more times the map point is observed, the greater the weight, because the spatial coordinates of such points are more reliable and should contribute more to the optimization.

[0134] Starting from the initial pose, the pose is iteratively optimized by the Gauss-Newton algorithm. In each iteration, the Jacobian matrix of the objective function with respect to the Lie algebra variable is calculated, which reflects the sensitivity of the error to the change of the pose. Based on the Jacobian matrix, the incremental equation is solved to update the pose parameters, so that the objective function is reduced. When the number of iterations reaches a preset value, or the change of the pose between adjacent iterations is less than a threshold value, the iteration is stopped, and the current pose is output as the optimization result. The pose is gradually corrected by the numerical optimization algorithm to minimize the re-projection error, and finally a high-precision camera pose is obtained.

[0135] In the embodiment of the present application, a cylinder is constructed according to an initialization pipeline map, and key frames meeting the conditions are inserted into a local mapping thread, including:

[0136] Based on the point cloud data of the initialization pipeline map, a random sample consensus algorithm is used to fit the cylinder parameters, including the unit direction vector of the cylinder axis, the midpoint coordinates of the axis and the cylinder radius. Among them, the distance of each point in the point cloud to the candidate axis is calculated by the point-to-line distance formula, and the inliers with a distance less than a preset threshold are screened. When the proportion of the inliers exceeds a preset proportion value of the total number of point clouds, the candidate cylinder is determined as an effective model.

[0137] The current frame is determined as a key frame and inserted into the local mapping thread when the following conditions are met:

[0138] The translation distance between the current frame and the previous key frame exceeds 1 / 5 of the pipe diameter or the rotation angle exceeds 5 degrees;

[0139] The number of matching pairs between the feature points extracted from the current frame and the local map feature points is not less than 80, and the distribution of the matching points in the image covers at least 8 preset grid regions;

[0140] The root mean square value of the re-projection error after pose optimization is less than 1.5 pixels;

[0141] The key frame inserted into the local mapping thread contains the following information: optimized camera pose, feature point set and corresponding descriptor, image acquisition timestamp, co-view relationship with adjacent key frames, and projection position parameters of the key frame in the cylindrical model.

[0142] The pipe has a typical cylindrical structure, which can be converted into geometric constraints through algorithms to improve map accuracy. From the three-dimensional point cloud of the initialized pipe map, the key parameters of the cylinder are extracted, including the unit direction vector of the axis, the midpoint coordinate of the axis, and the radius of the cylinder.

[0143] A candidate cylinder axis is fitted by randomly sampling a small number of points in the point cloud. Then, the distances of all points in the point cloud to the candidate axis are calculated, and the points with distances less than a preset threshold are selected as inliers. When the proportion of inliers in the total number of points exceeds a preset value, it indicates that the candidate cylinder can well fit the pipe structure, and it is determined as a valid model. This step is used to remove noise points in the point cloud and strengthen the structured features of the pipe.

[0144] Key frames are core image frames for building and optimizing maps, and need to meet strict conditions to ensure mapping efficiency and accuracy. The translation distance between the current frame and the previous key frame exceeds 1 / 5 of the pipe diameter or the rotation angle exceeds 5 degrees, which is considered to avoid repeated insertion of key frames within a short distance, while ensuring sufficient parallax between adjacent key frames to facilitate triangulation to generate high-precision map points.

[0145] The number of matching pairs between the feature points of the current frame and the local map feature points is not less than 80, ensuring that there are enough spatial constraint relationships for subsequent optimization. The matching points need to cover at least 8 preset grid regions in the image to avoid the matching points being concentrated in a local area and to ensure the comprehensiveness of spatial constraints. The root mean square of the re-projection error of the optimized camera pose needs to be less than 1.5 pixels to ensure that the pose estimation of the current frame is accurate enough to avoid introducing errors into the map.

[0146] The key frame inserted into the local mapping thread needs to carry multi-dimensional information to meet the needs of map construction and optimization: the optimized camera pose serves as the spatial reference for the triangulation of map points; feature points and descriptors are used to match other key frames to establish cross-frame spatial correlation; the timestamp ensures the time synchronization of multi-sensor data; the co-view relationship records the shared feature point information with other key frames, which is used to construct the topology of the local map; the cylindrical model projection parameters represent the axial position and circumferential angle of the current frame in the pipe cylindrical model, which strengthens the geometric correlation between the key frame and the pipe structure.

[0147] Embodiment Three

[0148] Figure 3 is a flowchart of a local BA method provided by Embodiment Three of the present application. As shown in Figure 3 , the method comprises:

[0149] S310, for the matching feature point pairs of the key frame and the adjacent key frame, the three-dimensional coordinates are calculated through binocular triangulation, and the points with a re-projection error less than 2 pixels and a parallax angle greater than 1 degree are selected as effective map points, and the observed key frame index and observation pixel coordinates of each map point are associated.

[0150] The feature point pairs between the key frame and its adjacent key frame have been obtained through feature descriptor matching. The two-dimensional pixel coordinates of a pair of matching feature points are converted into three-dimensional space coordinates by using the internal parameters of the binocular camera and the relative pose between the two key frames.

[0151] The calculated three-dimensional map points are re-projected back to the original key frame image according to the camera pose. If the pixel deviation between the projection position and the original feature point is less than 2 pixels, it means that the three-dimensional coordinate calculation is accurate; otherwise, it may be a matching error or a calculation error, which needs to be excluded. The parallax angle refers to the angle formed by the connecting lines of the three-dimensional point and the optical centers of the two key frames. The larger the angle, the more significant the difference between the two perspectives, and the higher the depth precision of the triangulation calculation. If the angle is less than 1 degree, the depth calculation is easily affected by noise. Through these two conditions, low-precision and unreliable three-dimensional points can be filtered out, and effective map points that can stably reflect the pipe structure are retained.

[0152] The observed key frame index is used to establish the many-to-many association between the map points and the key frames. The observation pixel coordinates store the pixel position of the map point in each observed key frame, which provides the basis for the re-projection error calculation for pose optimization.

[0153] S320, based on the cylindrical model, the effective map points are verified for cylindrical constraint, the distance of the map points to the cylindrical axis is calculated, if the deviation between the distance and the radius of the cylinder is within a preset value, the map point is marked as a cylindrical point, and the parameterized coordinates of the map point in the circumferential and axial directions of the cylinder are recorded.

[0154] Based on the previously fitted cylindrical model, the effective map points generated by binocular triangulation are verified again to determine whether they conform to the cylindrical geometric characteristics of the pipe inner wall. The straight-line distance of each effective map point to the cylindrical axis is calculated, and the distance is compared with the radius of the cylindrical model to screen out cylindrical points or non-cylindrical points. For the map points marked as cylindrical points, the parameterized coordinates in the cylindrical coordinate system need to be recorded, including the circumferential angle θ: the angle between the map point and the reference direction in the plane perpendicular to the axis, describing the position of the point in the circumferential direction of the pipe; and the axial distance s: the distance of the map point along the cylindrical axis to the midpoint of the axis, describing the position of the point in the length direction of the pipe.

[0155] S330, an optimization window containing the current key frame, its co-visibility key frame and corresponding map points is constructed to represent the camera pose in Lie algebra, the vertices include key frame pose and map point coordinates, and the edges are observation constraints; cylindrical constraint edges are additionally added for cylindrical points to constrain the distance to the axis equal to the cylindrical radius, and the weight is set to 1.5 times that of ordinary observation edges; the g2o optimization library is used for sparse BA solving, and the iteration is stopped when the root mean square of re-projection error is less than 1.2 pixels or the iteration number reaches 30 times, and the optimized pose and map point coordinates are output.

[0156] The current key frame, several key frames with the highest co-visibility degree and the map points observed by these key frames are selected to form a local optimization window.

[0157] The vertices and edges of the optimization model are defined, and the vertices are variables to be solved in the optimization process, including: key frame pose: represented by Lie algebra, which converts the nonlinear rotation matrix into a vector in the linear space to simplify the optimization calculation. Map point coordinates: represented by three-dimensional space coordinates (x, y, z) as spatial position parameters to be optimized. The edges describe the constraint relationship between the vertices, i.e., the source of the objective function of optimization: observation constraint edge: for each map point, its projection position in the key frame image should be consistent with the actual observed feature point pixel position, and this constraint is quantified by re-projection error. Cylindrical constraint edge: an additional structured constraint that forces the distance of a cylindrical point to the cylindrical axis to be equal to the pipe radius. The weight of this constraint is set to 1.5 times that of the ordinary observation edge, meaning that the geometric structure constraint of the pipe is prioritized in optimization, and the contribution of structured features to accuracy is strengthened.

[0158] The g2o optimization library is used for sparse BA solving. The sparse BA uses the feature that the map points are observed by only a few key frames, greatly reduces the calculation amount through sparse matrix operation, and is suitable for real-time systems. Taking the current pose and map point coordinates as initial values, the residual square sum of all constraint edges is minimized through iteration, and the pose and map point parameters are continuously corrected. Convergence criterion: stop iteration when any of the following conditions is met: the root mean square of the re-projection error of all observed points is less than 1.2 pixels; the number of iterations reaches 30 times. The optimized key frame pose and map point coordinates eliminate the cumulative error in the local range, making the local map more accurate and self-consistent.

[0159] Figure 4 is a flowchart of a global BA method provided by Embodiment Three of the present application, as shown in Figure 4 , the method comprises:

[0160] S410, input the feature point descriptor of the current key frame into the XFeat database, and find the top 20 historical key frames with the highest matching degree with the current frame features through the approximate nearest neighbor search algorithm as candidate loop frames.

[0161] The descriptor of the feature point in the current key frame is extracted and input into the XFeat feature database storing historical data. The approximate nearest neighbor search algorithm is used to quickly calculate the similarity of the current frame features and all historical key frame features in the database, and the top 20 historical key frames with the highest matching degree are found. The historical key frames as candidate loop frames mean that the robot may have reached the positions corresponding to these frames, which are used to verify the formation of the loop.

[0162] S420, for the candidate loop frames, estimate the similarity transformation matrix from the feature matching pairs through the RANSAC algorithm, wherein the inlier determination threshold is set to 3 pixels; when the number of inliers exceeds 30% of the total number of matching pairs and the transformation matrix satisfies the homography constraint, the candidate frame is confirmed as an effective loop frame; the cumulative error is calculated based on the similarity transformation matrix, and the error value is the root mean square of the re-projection error of all inliers.

[0163] For the candidate loop frames, estimate the similarity transformation matrix from the feature matching pairs using the RANSAC algorithm, and set 3 pixels as the inlier determination threshold. When the number of inliers exceeds 30% of the total matching pairs, and the transformation matrix meets the homography geometric constraint, the candidate frame is confirmed as an effective loop frame, i.e. the robot really returns to the position it has reached. Based on the similarity transformation matrix, the root mean square of the re-projection error of all inliers is calculated as the cumulative error before and after the loop, reflecting the degree of deviation of the pose estimation.

[0164] S430, a global optimization graph containing all key frame poses and map points is constructed, the similar transformation obtained by loop detection is added as a strong constraint to the graph, and the constraint weight is dynamically adjusted according to the cumulative error; a sparse BA algorithm is used for optimization, and the weighted sum of re-projection error and loop constraint error is taken as the objective function, and an edge operation is performed every 10 iterations in the iteration process to remove redundant variables; when the change of the objective function of two consecutive iterations is less than 1e-6 or the iteration number reaches 80, the optimization is stopped.

[0165] A global optimization graph containing all key frame poses and map points is constructed, and the similar transformation obtained by loop detection is added as a strong constraint, and the constraint weight is dynamically adjusted according to the cumulative error.

[0166] A sparse BA algorithm is used for optimization, and the weighted sum of re-projection error and loop constraint error is taken as the optimization objective, and every 10 iterations are removed by edge to reduce the calculation amount.

[0167] When the change of the objective function of two consecutive iterations is less than 1e-6 or the iteration reaches 80, the optimization is stopped, and the finally output global consistent pose and map are eliminated.

[0168] Embodiment four

[0169] Figure 5 It is a structural schematic diagram of a pipe network detection robot positioning device provided by the embodiment four of the present application. As shown in the figure, Figure 5 The device comprises:

[0170] The image acquisition unit 510 is used for acquiring images by the binocular camera arranged by the detection robot, and transmitting the left eye image and the right eye image in the form of topics respectively; and extracting the feature points and descriptors of the left eye image and the right eye image by using the accelerated feature network.

[0171] The initial pose determination unit 520 is used for generating an initial pipe map by using a SLAM system and determining the initial pose of the camera by a tracking thread; and optimizing the initial pose by using the feature points and descriptors.

[0172] The cylinder construction unit 530 is used for constructing a cylinder according to the initial pipe map, and inserting the key frame meeting the condition into a local mapping thread.

[0173] The local BA unit 540 is used for generating map points and cylinder points by using the local mapping thread by using the received key frame, and performing local BA.

[0174] The global BA unit 550 is used for searching for candidate loop frames by using the XFeat database by using a loop detection thread, calculating a similar transformation to obtain a cumulative error, performing global BA optimization of the map, and positioning the detection robot by using the optimized map.

[0175] The pipe network detection robot positioning device provided by the embodiments of the present application can execute the pipe network detection robot positioning method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0176] Embodiment five

[0177] Figure 6 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0178] As shown in Figure 6 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0179] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, a speaker, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0180] The processor 11 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the pipe network detection robot positioning method.

[0181] In some embodiments, the pipe network detection robot positioning method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the pipe network detection robot positioning method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the pipe network detection robot positioning method by any other suitable means, such as by means of firmware.

[0182] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0183] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a machine or a remote machine or a server.

[0184] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0186] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0187] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0188] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0189] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for locating a pipeline inspection robot, characterized in that, The method comprises the following steps: An image is acquired by a binocular camera arranged on a robot, and left-eye and right-eye images are transmitted in the form of topics respectively; An accelerated feature network is used to extract feature points and descriptors of the left-eye and right-eye images; An initial pipeline map is generated by a SLAM system, and an initial pose of the camera is determined by a tracking thread; The initial pose is optimized by the feature points and the descriptors; A cylinder is constructed according to the initial pipeline map, and key frames meeting a condition are inserted into a local mapping thread; The local mapping thread is used to generate map points and cylinder points by using the received key frames, and local BA is performed; A loop closure detection thread is used to find candidate loop closure frames through an XFeat database, to calculate a cumulative error by calculating a similarity transformation, to optimize a map by performing global BA, and to position the detection robot by using the optimized map; The accelerated feature network is an end-to-end network model based on deep learning, the input of which is RGB data of left-eye and right-eye images, and the output of which includes: A keypoint heat map in the form of a two-dimensional matrix representing a position probability distribution of feature points in an image, wherein a higher value of a first matrix element in the keypoint heat map indicates a higher possibility that a feature point exists at the position; A reliability heat map of the same size as the keypoint heat map, wherein a value of a second matrix element of the reliability heat map reflects a reliability score of a feature point at a corresponding position, and the score ranges from 0 to 1, and a value closer to 1 indicates higher reliability; A descriptor set, wherein each feature point corresponds to a 128-dimensional binary descriptor vector, which is used for matching between feature points; The accelerated feature network is used to extract feature points of the left-eye and right-eye images, which comprises the following steps: Confidence scores of feature points in the keypoint heat map and the reliability heat map are calculated by the following formula: ； wherein, is a normalized score of a feature point in the keypoint heat map, is a normalized score of a feature point in the reliability heat map, and is a weight coefficient, and + = 1. Top N feature points are selected according to the confidence scores, and feature points that are uniformly distributed are selected by a grid division method, wherein an image is divided into MxM grids, and no more than P feature points are reserved in each grid; The initial pose is optimized by the feature points and the descriptors, which comprises the following steps: Map points corresponding to a preset number of key frames with the highest degree of co-visibility with a current frame are selected from a local map, and descriptors of the map points are extracted to form a local feature library, and each map point is associated with observation information of at least a preset number of key frames; A bidirectional matching mechanism is adopted, wherein initial matching pairs are obtained by matching feature point descriptors of a current frame with a local feature library, and each matching pair is verified for projection consistency of a feature point of the current frame and a map point in a corresponding key frame, and matching pairs with a projection error exceeding a preset number of pixels are removed; An optimization objective function based on Lie algebra is constructed with the initial pose as an iteration starting point, and the objective function is a weighted sum of re-projection errors of all matching points, wherein the weights are dynamically adjusted according to the number of observations of the map points; A Gauss-Newton algorithm is used for iterative optimization, a Jacobian matrix is calculated after each iteration to update pose parameters, and the iteration is stopped when the number of iterations reaches a preset number or the pose change between adjacent iterations is less than 1e-6 radians and 1e-3 meters, and an optimized pose is output.

2. The method of claim 1, wherein, The left eye image and the right eye image acquired by the binocular camera are transmitted through independent communication topics respectively, the topic format adopts a robot operating system (ROS) standard message type, and a transmission frame rate is the same as a collection frame rate of the binocular camera.

3. The method of claim 1, wherein, The initial pipeline map is generated by using the SLAM system, and the initial pose of the camera is determined by using a tracking thread, and the method comprises the following steps: A preset number of binocular images are selected, a fundamental matrix is calculated by feature point matching, an initial rotation matrix and a translation vector are obtained by decomposition, a three-dimensional point cloud is generated by triangulation in combination with binocular camera internal parameters and a baseline distance, and the initial pipeline map is constructed. It is determined whether a previous frame image exists at present, if the previous frame image exists, a current pose of the binocular camera is predicted as an initial value based on a constant speed motion model, feature point matching of a current frame and the previous frame is performed, a pose is solved by using an RANSAC-PnP algorithm, and iterative optimization is performed. If the initial pose is determined for the first time after the SLAM system is started, in-point is screened from the feature point matching pair by using a random sample consensus (RANSAC) algorithm, an essential matrix is calculated, and the initial pose is obtained by decomposition.

4. The method of claim 1, wherein, The initial pipeline map is used to construct a cylinder, and a key frame meeting a condition is inserted into a local mapping thread, and the method comprises the following steps: Point cloud data of the initial pipeline map is used to fit cylinder parameters by using the RANSAC algorithm, the cylinder parameters comprise a unit direction vector of a cylinder axis, a coordinate of an axis midpoint and a cylinder radius, wherein distances of points in the point cloud to a candidate axis are calculated by using a point-to-line distance formula, in-points with distances less than a preset threshold value are screened, and when a proportion of the in-points exceeds a preset proportion value of a total number of the point cloud, the candidate cylinder is determined as an effective model; The current frame is determined as a key frame and is inserted into the local mapping thread when the current frame meets the following conditions: A translation distance between the current frame and a previous key frame exceeds 1 / 5 of a diameter of the pipeline or a rotation angle exceeds 5 degrees; A number of matching pairs of feature points in the current frame and local map feature points is not less than 80, and a distribution of the matching points in an image covers at least 8 preset grid regions; A root mean square value of a re-projection error after pose optimization is less than 1.5 pixels; The key frame inserted into the local mapping thread comprises the following information: an optimized camera pose, a feature point set and a corresponding descriptor, an image collection time stamp, a co-view relationship with a neighboring key frame, and a projection position parameter of the key frame in the cylinder model.

5. The method of claim 4, wherein, The local mapping thread generates map points and cylinder points by using the received key frame, and performs local BA, and the method comprises the following steps: Matching feature point pairs of a key frame and a neighboring key frame are used to calculate three-dimensional coordinates by binocular triangulation, points with a re-projection error less than 2 pixels and a parallax angle greater than 1 degree are screened as effective map points, and an index of a key frame observed by each map point and observed pixel coordinates are associated with each map point; The effective map points are verified by cylinder constraint based on the cylinder model, a distance of the map points to the cylinder axis is calculated, if a deviation between the distance and a cylinder radius is within a preset value, the map points are marked as cylinder points, and parameterized coordinates of the map points in a circumferential direction and an axial direction of the cylinder are recorded. An optimization window containing the current key frame, its co-view key frame and corresponding map points is constructed, camera poses are represented by Lie algebra, vertices include key frame poses and map point coordinates, and edges are observation constraints; a cylindrical constraint edge is additionally added for a cylindrical point to constrain the distance from the axis to the cylindrical point to be equal to the cylindrical radius, and the weight is set to be 1.5 times that of a normal observation edge; a g2o optimization library is used to solve sparse BA, and iteration is stopped when the root mean square of re-projection error is less than 1.2 pixels or the number of iterations reaches 30, and the optimized pose and map point coordinates are output.

6. The method of claim 1, wherein, The loop closure detection thread searches for candidate loop closure frames through the XFeat database, calculates the cumulative error by obtaining a similarity transformation, and performs global BA optimization of the map, including: The feature point descriptors of the current key frame are input into the XFeat database, and the first 20 historical key frames with the highest feature matching degree with the current frame are found as candidate loop closure frames through an approximate nearest neighbor search algorithm; For the candidate loop closure frames, a similarity transformation matrix is estimated from the feature matching pairs by the RANSAC algorithm, wherein the inlier determination threshold is set to 3 pixels; when the number of inliers exceeds 30% of the total number of matching pairs and the transformation matrix satisfies the homography constraint, the candidate frame is confirmed as an effective loop closure frame; the cumulative error is calculated based on the similarity transformation matrix, and the error value is the root mean square of the re-projection error of all inliers; A global optimization graph containing all key frame poses and map points is constructed, the similarity transformation obtained by loop closure detection is added as a strong constraint to the graph, and the constraint weight is dynamically adjusted according to the cumulative error; sparse BA algorithm is used for optimization, and the weighted sum of the re-projection error and the loop closure constraint error is used as the objective function, and an edge operation is performed every 10 iterations in the iteration process to remove redundant variables; the optimization is stopped when the change of the objective function in two consecutive iterations is less than 1e-6 or the number of iterations reaches 80.

7. A pipe network inspection robot positioning apparatus, characterized by, Including: An image acquisition unit is configured to acquire images by a binocular camera arranged on the robot, and transmit left-eye images and right-eye images in the form of topics respectively; An accelerated feature network is configured to extract feature points and descriptors of the left-eye images and the right-eye images; An initial pose determination unit is configured to generate an initialization pipeline map by using a SLAM system and determine an initial pose of the camera by a tracking thread; The initial pose is optimized by the feature points and the descriptors; A cylinder construction unit is configured to construct a cylinder according to the initialization pipeline map, and insert key frames meeting a condition into a local mapping thread; A local BA unit is configured to generate map points and cylindrical points by the local mapping thread using the received key frames, and perform local BA; A global BA unit is configured to search for candidate loop closure frames by a loop closure detection thread through an XFeat database, calculate a cumulative error by obtaining a similarity transformation, perform global BA optimization of a map, and position the robot by the optimized map. The accelerated feature network is an end-to-end network model based on deep learning, the input of which is RGB data of left-eye and right-eye images, and the output of which includes: A key point heat map representing a position probability distribution of feature points in an image in a two-dimensional matrix form, wherein a higher value of a first matrix element in the key point heat map represents a higher possibility of a feature point existing at the position; A reliability heat map of the same size as the key point heat map, a second matrix element value of the reliability heat map reflecting a reliability score of a feature point at a corresponding position, the score ranging from 0 to 1, and a value closer to 1 representing higher reliability; A descriptor set, each feature point corresponding to a 128-dimensional binary descriptor vector, for matching between feature points; The image acquisition unit, when extracting the feature points of the left and right images by using the accelerated feature network, performs: Confidence scores of each feature point in the key point heat map and the reliability heat map are calculated by the following formula: ； wherein, is a normalized score of a feature point in the keypoint heat map, is a normalized score of a feature point in the reliability heat map, and is a weight coefficient, and + = 1. Selecting top N feature points according to the scores, and selecting uniformly distributed feature points by grid division, dividing the image into MxM grids, and retaining no more than P feature points in each grid; The initial pose determination unit, when optimizing the initial pose by using the feature points and descriptors, performs: Selecting a preset number of map points corresponding to key frames with the highest co-visibility from the local map, extracting their descriptors to form a local feature library, and associating each map point with observation information of at least a preset number of key frames; Using a bidirectional matching mechanism, first matching the feature point descriptors of the current frame with the local feature library to obtain initial matching pairs, and then verifying the projection consistency of the feature points of the current frame and the map points in the corresponding key frames for each matching pair, and removing matching pairs with projection errors exceeding a preset number of pixels; Taking the initial pose as the starting point of iteration, constructing a pose optimization objective function based on Lie algebra, and the objective function being a weighted sum of the re-projection errors of all matching points, wherein the weights are dynamically adjusted according to the observation times of the map points; Using the Gauss-Newton algorithm for iterative optimization, calculating the Jacobian matrix to update the pose parameters after each iteration, stopping when the number of iterations reaches a preset number or the pose change between adjacent iterations is less than 1e-6 radians and 1e-3 meters, and outputting the optimized pose.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the pipe network detection robot positioning method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to execute the pipe network detection robot positioning method of any one of claims 1-6 when executed.

10. A computer program product, characterised in that, The computer program product comprises a computer program, which, when executed by the processor, implements the pipe network detection robot positioning method according to any one of claims 1-6.

Images