Robust binocular vision SLAM method and system for power distribution network unmanned aerial vehicle inspection
Through the robust binocular vision SLAM method, using binocular sensors and inertial measurement unit data, combined with the visual front-end and optimized back-end architecture, the problem of insufficient robustness of drone position estimation in the distribution network environment is solved, and the The efficiency and accuracy of autonomous drone inspection and power line mapping without RTK signals.
Patent Information
- Application Number
- CN202410991878.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-23
AI Technical Summary
The existing SLAM technology has insufficient robustness in the position estimation of drones in complex distribution network environments due to the dynamic disturbance between image frames, making it difficult to achieve independent patrols in the absence of RTK signals.
Using a robust binocular vision SLAM method, the image sequence and inertial measurement unit data are collected through binocular sensors, combined with the three-level architecture of the visual front-end and optimized back-end, time synchronization, ORB feature point extraction and matching are performed, and graph attention network removal is used. Dynamic feature points, IMU pre-integration and back-end BA optimization are performed to update pose information.
In the distribution network environment without RTK signal, the perception and positioning accuracy of the drone is improved, the inspection cost is reduced, the power line target construction is ensured, the equipment error is reduced, and the inspection is improved. efficiency.
Smart Images

Figure CN119007038A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to unmanned aerial vehicle inspection technology in the field of distribution networks, and in particular to a robust binocular vision SLAM method and system for unmanned aerial vehicle inspection of distribution networks. Background Art
[0002] The main task of the distribution network is to effectively transmit the power of the distribution network to end users and various electrical equipment. The inspection of traditional distribution network lines adopts the method of manual survey and inspection of abnormal routes. This method has the problems of huge workload, long time consumption and high experience requirements for operators. With the development of technology, the line inspection method based on the UAV platform is gradually becoming a new development direction in the field of power inspection. Simultaneous Localization and Mapping (SLAM) technology can accurately calculate the relative pose of the sensor and the surrounding environment through the sensors carried on the UAV, construct a sparse point cloud map, and estimate the instantaneous motion trajectory of the UAV at the same time. The emergence of this technology provides a reliable positioning and environmental perception means for UAV line inspection. In the distribution network environment, the decoding ability of the feature matching algorithm is subject to certain limitations. The current SLAM technology is easily affected by dynamic disturbances between image frames. In the complex distribution network environment, due to the interference of dynamic objects along the line on inspection targets such as poles, distribution boxes and wires, it is easy to cause the pose estimation to have a significant lack of robustness. Summary of the invention
[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a robust binocular vision SLAM method and system for distribution network UAV inspection is provided. The present invention aims to support autonomous inspection of UAVs in a distribution network environment lacking RTK signals based on a pure visual solution, eliminate some equipment and environmental interference introduced in the distribution network line acquisition process, and ensure the robustness of power line target mapping in such difficult tasks as autonomous UAV inspection.
[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A robust binocular vision SLAM method for distribution network UAV inspection, including: S1, using binocular sensors to collect the UAV inspection network environment to obtain binocular image sequences, obtaining the UAV's IMU data through the UAV's inertial measurement unit, transmitting the acquired binocular images to the visual front end, and transmitting the UAV's IMU data to the optimization back end; S2, the visual front end synchronizes the acquired binocular images and extracts ORB feature points, removes the extracted ORB feature points by optical flow, retains the ORB feature points with higher inspection modeling information, uses the graph attention network to match the ORB feature points, and transmits the matching valid ORB feature points to the optimization back end; S3, the optimization backend will match the valid ORB feature point pairs with the IMU data for time and space synchronization and complete the IMU pre-integration to obtain the pose information, and perform backend BA optimization on the ORB feature point pairs and IMU to update the pose information.
[0005] Optionally, the time synchronization of the acquired binocular images in step S2 includes: for two cache queues respectively used to cache the acquired left and right binocular images, whenever the acquired left and right binocular images are written into the two cache queues, the top timestamps are taken out from the two cache queues respectively to obtain the timestamp of the left eye image Time 0 and right eye image timestamp Time 1; Timestamp the left image Time 0 and right eye image timestamp Time 1Compare, if satisfied Time 0 <Time 1- a , then discard the left image; if Time 0 >Time 1+ a , then discard the right eye image, a It is the preset timestamp difference threshold.
[0006] Optionally, in step S2, the visual front end extracts ORB feature points from the acquired binocular image, including: S101, obtaining binocular image data of the current frame, using an image scale pyramid and using a fast feature detection algorithm FAST in each layer of the pyramid to extract feature points that are significantly different from surrounding pixels; S102, for the extracted feature points, the deviation between the corner point and the neighborhood centroid is calculated using the grayscale centroid method, and the main direction of spatial motion corresponding to the feature point is determined according to the offset vector; S103, performing Gaussian smoothing on the binocular image of the current frame, selecting a specified number of random point pairs in the vicinity of the extracted feature points in accordance with Gaussian distribution based on the BRIEF algorithm and rotating them according to the main direction of spatial motion of the feature points, comparing the pixels and sizes of the random point pair subwindows before and after the rotation, performing binary encoding to generate a feature descriptor with the same dimension as the number of the selected random point pairs as the feature descriptor of the extracted ORB feature points.
[0007] Optionally, the function expression for calculating the deviation between the corner point and the neighborhood centroid using the grayscale centroid method in step S102 is: , In the above formula, is the deviation between the corner point and the centroid of the neighborhood, is the coordinate of the center pixel of the image block, is the coordinate of the centroid, where the coordinate of the centroid is obtained by weighted average of the gray values of all pixels in the image window; The calculation function expression for determining the main direction of spatial motion corresponding to the feature point according to the offset vector is: , In the above formula, is the main direction of space motion, and are the weighted offset vector and the components in the x and y directions respectively, and we have: , , , , , In the above formula, is the offset vector magnitude, and Image blocks Offset vector in the x and y directions.
[0008] Optionally, in step S103, comparing the pixels and sizes of the random point pair subwindows before and after the rotation and performing binary encoding to generate a feature descriptor having a dimension consistent with the number of selected random point pairs includes: for each pair of random points before and after the rotation, if the pixel value of the random point before the rotation is less than the pixel value of the random point after the rotation, setting the corresponding bit to 1, otherwise setting the corresponding bit to 0, thereby generating a binary code consisting of 1 and 0 as a feature descriptor having a dimension consistent with the number of selected random point pairs.
[0009] Optionally, after the visual front end performs time synchronization and ORB feature point extraction on the acquired binocular image in step S2, it also includes detecting and removing target-irrelevant dynamic feature points from the extracted ORB feature points: S201, for the ORB feature point set formed by the ORB feature points of each frame of the binocular image, determine the feature point set Q of the current frame and the feature point set P of the next frame, calculate the motion prediction points corresponding to the ORB feature points in each frame in the feature point set Q of the current frame in the next frame based on the optical flow method, and obtain the motion feature point set corresponding to the feature point set Q of the current frame , according to the feature point set Q of the current frame and its corresponding motion feature point set Calculate the basis matrix , the fundamental matrix For a point x i and its predicted point x i+1 satisfy The matrix, and the basic matrix The calculation includes: first, the extracted ORB feature points and the corresponding motion prediction points are constructed with a size of The matrix A, according to Perform singular value decomposition on matrix A, where and are left singular matrices and right singular matrices, is a diagonal matrix; then according to the right singular matrix based on The solution size is The singular vectors of , and the singular vectors The following formula is shown The basic matrix : , In the above formula, ~ is a singular vector Elements in S202: First, for the ORB feature points in the feature point set Q of the current frame, q The epipolar line is calculated through the basic matrix, and the function expression for calculating the epipolar line corresponding to the ORB feature point is: , In the above formula, is the epipolar line corresponding to the ORB feature point, is the basic matrix, is the ORB feature point in the feature point set Q of the current frame q The coordinates of the ORB feature points in the feature point set Q of the current frame are then calculated q To the pole Euclidean distance; if the ORB feature point in the feature point set Q of the current frame q To the pole When the Euclidean distance is greater than a preset threshold m, the ORB feature point is judged to be a dynamic feature point; otherwise, the ORB feature point is judged to be a static feature point; S203, deleting the dynamic feature points and their feature descriptors, and retaining the static feature points and their feature descriptors.
[0010] Optionally, in step S2, using the graph attention network to perform ORB feature point matching includes: S301, for ORB feature points and their feature descriptors, a pre-trained classification network is used to calculate the matching ability of the ORB feature points, the classification network is composed of a multi-layer perceptron MLP, a linearization layer and a sigmoid activation function layer to output a matching ability with a value range of [0,1]; the matching ability refers to the probability of one ORB feature point matching another ORB feature point, and the allocation probability of all ORB feature points is obtained and the allocation probability matrix is constructed according to the following formula: , , In the above formula, is the allocation probability matrix between ORB feature points i and j, is the similarity between the feature descriptors of ORB feature points i and j, is the similarity between the feature descriptors of ORB feature points i and k, is the metric distance between feature descriptors of ORB feature points i and j, is an adjustment parameter for controlling the decay speed of similarity; S302, eliminating ORB feature points whose matching capabilities and the maximum values of the corresponding dimensions in the allocation matrix are lower than a preset threshold; S303: For the remaining ORB feature points, use the graph attention network to perform ORB feature point matching.
[0011] In addition, the present invention also provides a robust binocular vision SLAM system for applying the robust binocular vision SLAM method for distribution network drone inspection, comprising: The drone is used to collect binocular image sequences using binocular sensors when inspecting the network environment, obtain the IMU data of the drone through the inertial measurement unit carried by the drone, transmit the acquired binocular images to the visual front end, and transmit the IMU data of the drone to the optimization back end; The visual front end is used to extract ORB feature points from the binocular images after time synchronization, remove the extracted ORB feature points through optical flow, retain the ORB feature points with higher inspection modeling information, use the graph attention network to match the ORB feature points, and transmit the matching valid ORB feature points to the optimization back end; The optimization backend is used to synchronize the time and space of the matching valid ORB feature point pairs with the IMU data and complete the IMU pre-integration to obtain the pose information, and perform backend BA optimization on the ORB feature point pairs and the IMU to update the pose information.
[0012] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the robust binocular vision SLAM method for distribution network drone inspection through a processor.
[0013] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the robust binocular vision SLAM method for distribution network drone inspection through a processor.
[0014] Compared with the prior art, the beneficial effects of the present invention mainly include: 1. The present invention uses binocular sensors to collect binocular image sequences when the UAV inspects the distribution network environment, obtains the IMU data of the UAV through the inertial measurement unit carried by the UAV, and deploys a feasible SLAM algorithm using the three-level architecture of the UAV, the visual front end, and the optimized back end to achieve pure visual scene mapping of the distribution network lines. It relies on the relative distance between the object posture in the scene and the UAV to achieve effective perception and positioning of the UAV without RTK, thereby reducing the cost of distribution network inspection.
[0015] 2. The present invention performs time alignment on the collected binocular data and IMU data, alleviates the collection equipment error caused by unstable power supply of the UAV, and ensures the accuracy and efficiency of distribution network line modeling.
[0016] 3. The present invention includes a visual front-end that performs time synchronization and ORB feature point extraction on the acquired binocular images, performs optical flow elimination on the extracted ORB feature points, retains ORB feature points with higher inspection modeling information, uses a graph attention network to match ORB feature points, and transmits the effectively matched ORB feature points to the optimization back-end, which can eliminate some equipment and environmental interference introduced in the distribution network line acquisition process, and ensure the robustness of power line target mapping in such difficult tasks as autonomous drone inspections. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.
[0018] Figure 2 It is a principle framework diagram of the front-end visual odometer in an embodiment of the present invention.
[0019] Figure 3 4 is a flow chart of removing dynamic points from optical flow dynamic points in an embodiment of the present invention.
[0020] Figure 4 It is a schematic diagram of time synchronization between ORB feature point pairs and IMU data in an embodiment of the present invention.
[0021] Figure 5 It is a flowchart of backend optimization in an embodiment of the present invention. DETAILED DESCRIPTION
[0022] like Figure 1 As shown, the robust binocular vision SLAM method for distribution network drone inspection in this embodiment includes: S1, using binocular sensors to collect the UAV inspection network environment to obtain binocular image sequences, obtaining the UAV's IMU data through the UAV's inertial measurement unit, transmitting the acquired binocular images to the visual front end, and transmitting the UAV's IMU data to the optimization back end; S2, the visual front end synchronizes the acquired binocular images and extracts ORB feature points, removes the extracted ORB feature points by optical flow, retains the ORB feature points with higher inspection modeling information, uses the graph attention network to match the ORB feature points, and transmits the matching valid ORB feature points to the optimization back end; S3, the optimization backend will match the valid ORB feature point pairs with the IMU data for time and space synchronization and complete the IMU pre-integration to obtain the pose information, and perform backend BA optimization on the ORB feature point pairs and IMU to update the pose information.
[0023] In step S1 of this embodiment, the binocular sensor d435i is used to collect the network distribution environment of the drone inspection to obtain a binocular image sequence, the IMU data of the drone is collected by the inertial measurement unit, the information acquisition module controls the subscription of environmental information, the binocular image obtained is time-synchronized, and the collected data is transmitted to the SLAM visual front end. When collecting data through the drone, first obtain the take-off environment position information, set the detailed trajectory of the drone around the initial tower and the planned track points of the inspection route. The drone takes off in a predetermined manner, and collects image information and IMU information of the drone network distribution inspection through the D435i binocular sensor carried. The visual front end specifically uses the edge computer Jeston XavierNX, and the data acquisition module of the edge computer Jeston Xavier NX subscribes to the image and IMU information in real time, and processes the time synchronization of the binocular image. During the inspection process around the initial pole tower, the information acquisition module continuously inputs the binocular images of the effective time frame into the front-end visual odometer module configured by the edge computer, and uses the feature matching method based on the elimination of unmatched points to extract the key point position coordinates of the distribution pole tower and its equipment, calculate the pose matrix of the key points projected to the SLAM coordinate system, and combine the relative distance of the drone and the camera parameters to perform the initial stage of mapping the take-off environment in the SLAM coordinate system. According to the mapping results, the drone flies along the planned track points, and the binocular sensor continuously collects the surrounding environment information of the poles along the line. The information acquisition module of the edge computer updates the subscription of the binocular image and IMU information in real time.
[0024] The principle of time synchronization of the acquired binocular images in step S2 of this embodiment is to save the binocular images to the image buffer queue, and discard the left and right images with errors by comparing the timestamps of the binocular images. Specifically, the following steps are performed: for the two cache queues used to cache the acquired left and right binocular images respectively, whenever the acquired left and right binocular images are written into the two cache queues, the top timestamps are taken out from the two cache queues respectively to obtain the timestamp of the left image. Time 0 and right eye image timestamp Time 1; Timestamp the left image Time 0 and right eye image timestamp Time 1Compare, if satisfied Time 0 <Time 1- a , then discard the left image; if Time 0 >Time 1+ a , then discard the right eye image, a is the preset timestamp difference threshold. For example, in this embodiment, the preset timestamp difference threshold is 0.003s, then: Time 0 <Time 1-0.003s, discard the left eye image; Time 0 >Time 1+0.003s, discard the right eye image.
[0025] The timestamp difference of the left-right aligned binocular images cannot exceed 0.003s. The binocular images of the same frame with timestamp difference below the threshold are transmitted to the front-end visual odometer module for feature extraction and matching. The acquisition module performs the above operations until the cache queue is empty.
[0026] The visual front-end mainly includes a binocular image time synchronization module and a front-end visual odometry module. The front-end visual odometry module is used for ORB feature point extraction, optical flow method to eliminate dynamic points, and feature matching based on graph attention network. Figure 2It is a principle framework diagram of the front-end visual odometer module in an embodiment of the present invention. The entire process of ORB feature point extraction includes feature detection and feature description, and the workflow is: 1) Feature Detection: Obtain the binocular image data of the current frame, use the image scale pyramid and perform the fast feature detection algorithm (FAST) on each layer of the pyramid to extract feature points that are significantly different from the surrounding pixels. For the extracted feature points, the grayscale centroid method (Intensity Centroid) is used to calculate the deviation between the corner point and the neighborhood center of gravity, and the main direction of spatial motion corresponding to the feature point is determined according to the offset vector. 2) Feature Description: Gaussian smoothing is performed on the binocular image of the current frame, and a certain number of random point pairs are selected in sequence according to the Gaussian distribution near the extracted feature points based on the BRIEF algorithm and rotated according to the main direction of spatial motion of the feature points. The pixels and size of the rotated random point pair subwindows are compared and binary encoded to generate feature descriptors with the same dimension as the number of selected random point pairs. As Figure 2 As shown, in step S2 of this embodiment, the visual front end extracts ORB feature points from the acquired binocular image, including: S101, obtaining binocular image data of the current frame, using an image scale pyramid and using a fast feature detection algorithm FAST in each layer of the pyramid to extract feature points that are significantly different from surrounding pixels; the fast feature detection algorithm FAST is a well-known method, see the literature: Rosten E. Machine learning for high-speed corner detection [C] / / European Conference on Computer Vision. Springer-Verlag, 2006. DOI: 10.1007 / 11744023_34; S102, for the extracted feature points, the deviation between the corner point and the neighborhood centroid is calculated using the grayscale centroid method, and the main direction of spatial motion corresponding to the feature point is determined according to the offset vector; S103, Gaussian smoothing is performed on the binocular image of the current frame, and a specified number of random point pairs are selected in sequence according to Gaussian distribution near the extracted feature points based on the BRIEF algorithm and rotated according to the main direction of spatial motion of the feature points, and the pixels and sizes of the random point pair subwindows before and after the rotation are compared and binary encoded to generate a feature descriptor with the same dimension as the number of selected random point pairs as the feature descriptor of the extracted ORB feature points. The BRIEF algorithm is a well-known method, see the literature: Calonder M, Lepetit V, Strecha C, et al. BRIEF: Binary Robust Independent Elementary Features [J]. Springer, Berlin, Heidelberg, 2010. DOI: 10.1007 / 978-3-642-15561-1_5.
[0027] In step S102 of this embodiment, the function expression for calculating the deviation between the corner point and the neighborhood centroid using the grayscale centroid method is: , In the above formula, is the deviation between the corner point and the centroid of the neighborhood, is the coordinate of the center pixel of the image block, is the coordinate of the centroid, where the coordinate of the centroid is obtained by weighted average of the gray values of all pixels in the image window; The calculation function expression for determining the main direction of spatial motion corresponding to the feature point according to the offset vector is: , In the above formula, is the main direction of space motion, and are the weighted offset vector and the components in the x and y directions respectively, and we have: , , , , , In the above formula, is the offset vector magnitude, and Image blocks Offset vector in the x and y directions.
[0028] In step S103 of this embodiment, comparing the pixels and sizes of the random point pair subwindows before and after the rotation and performing binary encoding to generate a feature descriptor with a dimension consistent with the number of selected random point pairs includes: for each pair of random points before and after the rotation, if the pixel value of the random point before the rotation is less than the pixel value of the random point after the rotation, the corresponding bit is set to 1, otherwise the corresponding bit is set to 0, thereby generating a binary code composed of 1 and 0 as a feature descriptor with a dimension consistent with the number of selected random point pairs.
[0029] In this embodiment, the ORB feature point extraction process mainly performs feature detection and feature description operations based on image texture information, and the extracted feature points involve distribution network inspection objects and interference objects along the line. In the case of the distribution network line involved in this example, the inspection objects along the line are mainly power equipment such as wires, poles and distribution boxes, which often have static features and belong to static objects in the collected images. On the contrary, pedestrians, insects, birds and vegetation captured during collection along the line have dynamic features. There are dynamic disturbances between adjacent frames collected in a natural environment. Directly using them as reconstruction targets will affect the mapping effect of the SLAM system. Dynamic optical flow removal is performed on the extracted feature point queue, which can effectively retain static inspection objects, thereby avoiding interference and improving the overall robustness of the system. When processing binocular images, the optical flow dynamic point removal method uses the LK optical flow algorithm to perform motion detection on the extracted feature points, and determines and removes dynamic feature points that are not related to the detection target. Specifically, Figure 3 As shown, in step S2 of this embodiment, after the visual front end performs time synchronization and ORB feature point extraction on the acquired binocular image, it also includes detecting and removing target-irrelevant dynamic feature points from the extracted ORB feature points: S201, for the ORB feature point set formed by the ORB feature points of each frame of the binocular image, determine the feature point set Q of the current frame and the feature point set P of the next frame, calculate the motion prediction points corresponding to the ORB feature points in each frame in the feature point set Q of the current frame in the next frame based on the optical flow method, and obtain the motion feature point set corresponding to the feature point set Q of the current frame , according to the feature point set Q of the current frame and its corresponding motion feature point set Calculate the basis matrix , the fundamental matrix For a point x i and its predicted point x i+1 satisfy The matrix, and the basic matrix The calculation includes: first, the extracted ORB feature points and the corresponding motion prediction points are constructed with a size of The matrix A, according to Perform singular value decomposition on matrix A, where and are left singular matrices and right singular matrices, is a diagonal matrix; then according to the right singular matrix based on The solution size is The singular vectors of , and the singular vectors The following formula is shown The basic matrix : , In the above formula, ~ is a singular vector The elements in; among them, the ORB feature points in each frame in the feature point set Q of the current frame are calculated based on the optical flow method, and the corresponding motion feature point set in the next frame The optical flow method used is a well-known method, see the reference: Bouguet, Jean-Yves. "Pyramidal implementation of the affine lucaskanade feature tracker description of the algorithm." Intel Corporation 5.1-10(2001): 4; S202: First, for the ORB feature points in the feature point set Q of the current frame, q The epipolar line is calculated through the basic matrix, and the function expression for calculating the epipolar line corresponding to the ORB feature point is: , In the above formula, is the epipolar line corresponding to the ORB feature point, is the basic matrix, is the ORB feature point in the feature point set Q of the current frame q The coordinates of the ORB feature points in the feature point set Q of the current frame are then calculated q To the pole Euclidean distance; if the ORB feature point in the feature point set Q of the current frame q To the pole When the Euclidean distance is greater than a preset threshold m, the ORB feature point is judged to be a dynamic feature point; otherwise, the ORB feature point is judged to be a static feature point; S203, delete the dynamic feature points and their feature descriptors, and retain the static feature points and their feature descriptors. In the above manner, by further matching the static valid feature points and feature descriptions, the mismatch of the dynamic feature points with poor matching ability is avoided, and accurate matching feature point pairs are generated.
[0030] In step S2 of this embodiment, using the graph attention network to perform ORB feature point matching includes: S301, for ORB feature points and their feature descriptors, a pre-trained classification network is used to calculate the matching ability of the ORB feature points, the classification network is composed of a multi-layer perceptron MLP, a linearization layer and a sigmoid activation function layer to output a matching ability with a value range of [0,1]; the matching ability refers to the probability of one ORB feature point matching another ORB feature point, and the allocation probability of all ORB feature points is obtained and the allocation probability matrix is constructed according to the following formula: , , In the above formula, is the allocation probability matrix between ORB feature points i and j, is the similarity between the feature descriptors of ORB feature points i and j, is the similarity between the feature descriptors of ORB feature points i and k, is the metric distance between feature descriptors of ORB feature points i and j, is an adjustment parameter for controlling the decay speed of similarity; S302, eliminating ORB feature points whose matching capabilities and the maximum values of the corresponding dimensions in the allocation matrix are lower than a preset threshold; S303, for the remaining ORB feature points, use the graph attention network to match the ORB feature points. For the left and right binocular images, the feature points in which the ORB features are extracted are matched with each other. For example, a feature point in the left image is the upper left corner of the blackboard. For the upper left corner of the blackboard with different pixel coordinates in the right image, its ORB feature point is matched with the feature point in the left image. It should be noted that the use of the graph attention network for ORB feature point matching is an existing well-known method, see the literature: Sarlin PE, DeTone D, Malisiewicz T, et al. Superglue: Learning featurematching with graph neural networks[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2020: 4938-4947.
[0031] In this embodiment, a pair of binocular image feature sets to be matched are taken out from the feature point queue, and the matching ability of the feature points in the set is calculated by the multi-layer perceptron network MLP of the classification network, and the feature points with poor matching ability, that is, low matching probability, are eliminated to avoid the occurrence of false matching in the early stage of matching. The classification network is composed of a multi-layer perceptron MLP, a linearization layer and a sigmoid activation function layer to output the matching ability with a value range of [0,1], and the loss function used in training is: , , , In the above formula, is the loss function, The loss after calculating the matching ability for the ORB feature point set, To eliminate the loss of optimized allocation after low matching ability, and is the loss weight, M is the set of ORB feature point pairs that are actually matched. and are the sets of ORB feature point pairs of non-real matches of binocular images, P is the assignment probability matrix, is the matching capability corresponding to the ORB feature points, i and j They are the ORB feature points to be matched.
[0032] Step S3 is to optimize the backend to synchronize the time and space of the matching effective ORB feature point pairs with the IMU data and complete the IMU pre-integration to obtain the pose information. The backend BA (Bundle Adjustment) optimization of the ORB feature point pairs and the IMU to update the pose information is a well-known method. Figure 4 As shown, in one embodiment, the ORB matching feature point pairs and IMU data are synchronized in time and space, and the workflow is: 1) constrain the timestamp of the IMU data. Specifically, the IMU data within the timestamp range of the matching feature point pair of the previous frame and the matching feature point pair of the current frame is retained as the inertial navigation processing data of the current frame to ensure time synchronization. The specific formula is: t 0 <=t <t 1; In the formula, t 0 is the timestamp of feature point pair of the previous frame of ORB, t 1 is the timestamp of feature point pair of the current frame of ORB, t The timestamp of the IMU data needs to be kept in t 0 andt 1) The IMU data between the two cameras is used to ensure time synchronization. 2) The camera internal parameters, inertial measurement sensor internal parameters, and external parameters between the camera and inertial measurement sensor are calibrated for all 435i cameras to achieve spatial synchronization. Figure 5 As shown, in one embodiment, the back-end pose optimization method determines the marginalized frame according to the inter-frame parallax relationship, optimizes the matching feature points and IMU data based on the BA algorithm, and updates and inserts the pose information of the current frame change. The working steps are: 1) The back-end receives the current frame for parallax judgment. When the parallax is small, the previous frame is marginalized, and when the parallax is large, the first frame is marginalized. The ORB feature points are fused with the IMU to determine the system initialization status. If it has been initialized, go to step 4). 2) The image depth is calculated from the binocular image, and the initial pose information is calculated using 3d-2d PnP. 3) Ensure that the number of image frames is equal to the size of the sliding window, perform re-propagation of the IMU pre-integration, optimize the marginalized prior information, the residual information of the IMU and the image residual information, update the pose information, and update the sliding window according to the marginalized frame judged by the parallax, and the initialization is completed. 4) While processing the IMU information for pre-integration, use the IMU information to calculate the pose, further eliminate invalid points to update the pose information, and update the sliding window according to the marginalized frame judged by the parallax. Determine the detection failure situation. If the detection fails, go to step 5). 5) Clear all states, initialize all parameters, and the system re-subscribes to image and IMU information, detects and optimizes. Specifically, in this embodiment, after the front-end visual odometer completes the feature point extraction of a new frame, the extracted feature points are sent to the back-end optimization module. The back-end optimization module first synchronizes the time of the IMU and the image frame, obtains the IMU data between the previous frame and the current frame, and performs pre-integration, and calculates the corresponding spatial pose of the extracted feature points based on the alignment results. When the system is not initialized, the initialization operation must be performed first. Back-end optimization can effectively reduce the physical errors of the acquisition equipment and the inertial navigation unit, and improve the robustness of mapping during drone inspections.
[0033] In summary, the robust binocular visual SLAM method for distribution network drone inspection in this embodiment uses binocular sensors to collect the distribution network environment of drone inspection to obtain binocular image sequences, obtains the IMU data of the drone through the inertial measurement unit carried by the drone, and synchronizes the acquired binocular images to the visual front end; the visual front end synchronizes the acquired binocular images in time and extracts ORB feature points, removes the extracted ORB feature points by optical flow, retains the ORB feature points with higher inspection modeling information, uses the graph attention network to match the ORB feature points and transmits the matched effective ORB feature points to the optimization back end; the optimization back end synchronizes the matched effective ORB feature point pairs with the IMU data in time and space and completes the IMU pre-integration to obtain the pose information, and performs back-end BA optimization on the ORB feature point pairs with the IMU to update the pose information. The robust binocular visual SLAM method for distribution network drone inspection in this embodiment can perform drone perception and self-positioning in the distribution network environment lacking RTK signals, significantly improves the robustness of outdoor inspection targets, and is suitable for distribution network drone inspection tasks. This embodiment uses binocular sensors to collect the distribution network environment of the drone inspection to obtain a binocular image sequence, obtains the IMU data of the drone through the inertial measurement unit carried by the drone, and uses the three-level architecture of the drone, the visual front end, and the optimization back end to deploy a feasible SLAM algorithm to realize the pure visual scene mapping of the distribution network line, relying on the relative distance between the object posture in the scene and the drone, to achieve effective perception and positioning of the drone without RTK, and reduce the inspection cost of the distribution network. This embodiment performs time alignment on the collected binocular data and IMU data, alleviates the error of the acquisition equipment caused by the unstable power supply of the drone, and ensures the accuracy and efficiency of the distribution network line modeling. This embodiment includes the visual front end performing time synchronization and ORB feature point extraction on the acquired binocular images, performing optical flow elimination of the extracted ORB feature points, retaining the ORB feature points with higher inspection modeling information, using the graph attention network to match the ORB feature points, and transmitting the matching effective ORB feature points to the optimization back end, which can eliminate some equipment and environmental interference introduced in the distribution network line acquisition process, and ensure the robustness of the power line target mapping under such difficult tasks as autonomous drone inspection.
[0034] In addition, this embodiment also provides a robust binocular vision SLAM system for applying the robust binocular vision SLAM method for distribution network drone inspection, including: The drone is used to collect binocular image sequences using binocular sensors when inspecting the network environment, obtain the IMU data of the drone through the inertial measurement unit carried by the drone, transmit the acquired binocular images to the visual front end, and transmit the IMU data of the drone to the optimization back end; The visual front end is used to extract ORB feature points from the binocular images after time synchronization, remove the extracted ORB feature points through optical flow, retain the ORB feature points with higher inspection modeling information, use the graph attention network to match the ORB feature points, and transmit the matching valid ORB feature points to the optimization back end; The optimization backend is used to synchronize the time and space of the matching valid ORB feature point pairs with the IMU data and complete the IMU pre-integration to obtain the pose information, and perform backend BA optimization on the ORB feature point pairs and the IMU to update the pose information.
[0035] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the robust binocular vision SLAM method for distribution network drone inspection through a processor.
[0036] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the robust binocular vision SLAM method for distribution network drone inspection through a processor.
[0037] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0038] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A robust binocular vision SLAM method for distribution network drone inspection. It is characterized in that include: S1, using binocular sensors to collect the UAV inspection network environment to obtain binocular image sequences, obtaining the UAV's IMU data through the UAV's inertial measurement unit, transmitting the acquired binocular images to the visual front end, and transmitting the UAV's IMU data to the optimization back end; S2, the visual front end synchronizes the acquired binocular images and extracts ORB feature points, removes the extracted ORB feature points by optical flow, retains the ORB feature points with higher inspection modeling information, uses the graph attention network to match the ORB feature points, and transmits the matching valid ORB feature points to the optimization back end; S3, the optimization backend will match the valid ORB feature point pairs with the IMU data for time and space synchronization and complete the IMU pre-integration to obtain the pose information, and perform backend BA optimization on the ORB feature point pairs and IMU to update the pose information.
2. The robust binocular vision SLAM method for distribution network UAV inspection according to claim 1, It is characterized in that The time synchronization of the acquired binocular images in step S2 includes: for two cache queues used to cache the acquired left and right binocular images respectively, whenever the acquired left and right binocular images are written into the two cache queues, the top timestamps are taken out from the two cache queues respectively to obtain the timestamp of the left image Time 0 and right eye image timestamp Time 1 ; Timestamp the left image Time 0 and right eye image timestamp Time 1 Compare, if satisfied Time 0 <Time 1 - a , then discard the left eye image; if Time 0 >Time 1 + a , then discard the right eye image, a It is the preset timestamp difference threshold.
3. The robust binocular vision SLAM method for distribution network drone inspection according to claim 1, It is characterized in that In step S2, the visual front end extracts ORB feature points from the acquired binocular image, including: S101, obtaining binocular image data of the current frame, using an image scale pyramid and using a fast feature detection algorithm FAST in each layer of the pyramid to extract feature points that are significantly different from surrounding pixels; S102, for the extracted feature points, the deviation between the corner point and the neighborhood centroid is calculated using the grayscale centroid method, and the main direction of spatial motion corresponding to the feature point is determined according to the offset vector; S103, performing Gaussian smoothing on the binocular image of the current frame, selecting a specified number of random point pairs in the vicinity of the extracted feature points in accordance with Gaussian distribution based on the BRIEF algorithm and rotating them according to the main direction of spatial motion of the feature points, comparing the pixels and sizes of the random point pair subwindows before and after the rotation, performing binary encoding to generate a feature descriptor with the same dimension as the number of the selected random point pairs as the feature descriptor of the extracted ORB feature points.
4. The robust binocular vision SLAM method for distribution network UAV inspection according to claim 3, It is characterized in that The function expression for calculating the deviation between the corner point and the neighborhood centroid using the grayscale centroid method in step S102 is: , In the above formula, is the deviation between the corner point and the centroid of the neighborhood, is the coordinate of the center pixel of the image block, is the coordinate of the centroid, where the coordinate of the centroid is obtained by weighted average of the gray values of all pixels in the image window; The calculation function expression for determining the main direction of spatial motion corresponding to the feature point according to the offset vector is: , In the above formula, is the main direction of space motion, and are the weighted offset vector and the components in the x and y directions respectively, and we have: , , , , , In the above formula, is the offset vector magnitude, and Image blocks Offset vector in the x and y directions.
5. The robust binocular vision SLAM method for distribution network drone inspection according to claim 3, It is characterized in that In step S103, comparing the pixels and sizes of the random point pair subwindows before and after the rotation and performing binary encoding to generate a feature descriptor with a dimension consistent with the number of selected random point pairs includes: for each pair of random points before and after the rotation, if the pixel value of the random point before the rotation is less than the pixel value of the random point after the rotation, the corresponding bit is set to 1, otherwise the corresponding bit is set to 0, thereby generating a binary code composed of 1 and 0 as a feature descriptor with a dimension consistent with the number of selected random point pairs.
6. The robust binocular vision SLAM method for distribution network drone inspection according to claim 3, It is characterized in that In step S2, after the visual front end performs time synchronization and ORB feature point extraction on the acquired binocular images, it also includes detecting and removing target-independent dynamic feature points from the extracted ORB feature points: S201, for the ORB feature point set formed by the ORB feature points of each frame of the binocular image, determine the feature point set Q of the current frame and the feature point set P of the next frame, calculate the motion prediction points corresponding to the ORB feature points in each frame in the feature point set Q of the current frame in the next frame based on the optical flow method, and obtain the motion feature point set corresponding to the feature point set Q of the current frame , according to the feature point set Q of the current frame and its corresponding motion feature point set Calculate the basis matrix , the fundamental matrix For a point x i and its predicted point x i+1 satisfy The matrix, and the basic matrix The calculation includes: first, the extracted ORB feature points and the corresponding motion prediction points are constructed with a size of The matrix A, according to Perform singular value decomposition on matrix A, where and are left singular matrices and right singular matrices, is a diagonal matrix; then according to the right singular matrix based on The solution size is The singular vectors of , and the singular vectors The following formula is shown The basic matrix : , In the above formula, ~ is a singular vector Elements in S202: First, for the ORB feature points in the feature point set Q of the current frame, q The epipolar line is calculated through the basic matrix, and the function expression for calculating the epipolar line corresponding to the ORB feature point is: , In the above formula, is the epipolar line corresponding to the ORB feature point, is the basic matrix, is the ORB feature point in the feature point set Q of the current frame q The coordinates of the ORB feature points in the feature point set Q of the current frame are then calculated q To the pole Euclidean distance; if the ORB feature point in the feature point set Q of the current frame q To the pole When the Euclidean distance is greater than a preset threshold m, the ORB feature point is judged to be a dynamic feature point; otherwise, the ORB feature point is judged to be a static feature point; S203, deleting the dynamic feature points and their feature descriptors, and retaining the static feature points and their feature descriptors.
7. The robust binocular vision SLAM method for distribution network drone inspection according to claim 6, It is characterized in that In step S2, using the graph attention network to perform ORB feature point matching includes: S301, for ORB feature points and their feature descriptors, a pre-trained classification network is used to calculate the matching ability of the ORB feature points, the classification network is composed of a multi-layer perceptron MLP, a linearization layer and a sigmoid activation function layer to output a matching ability with a value range of [0,1]; the matching ability refers to the probability of one ORB feature point matching another ORB feature point, and the allocation probability of all ORB feature points is obtained and the allocation probability matrix is constructed according to the following formula: , , In the above formula, is the allocation probability matrix between ORB feature points i and j, is the similarity between the feature descriptors of ORB feature points i and j, is the similarity between the feature descriptors of ORB feature points i and k, is the metric distance between feature descriptors of ORB feature points i and j, is an adjustment parameter for controlling the decay speed of similarity; S302, eliminating ORB feature points whose matching capabilities and the maximum values of the corresponding dimensions in the allocation matrix are lower than a preset threshold; S303: For the remaining ORB feature points, use the graph attention network to perform ORB feature point matching.
8. A robust binocular vision SLAM system for applying the robust binocular vision SLAM method for distribution network drone inspection as described in any one of claims 1 to 7, It is characterized in that include: The drone is used to collect binocular image sequences using binocular sensors when inspecting the network environment, obtain the IMU data of the drone through the inertial measurement unit carried by the drone, transmit the acquired binocular images to the visual front end, and transmit the IMU data of the drone to the optimization back end; The visual front end is used to extract ORB feature points from the binocular images after time synchronization, remove the extracted ORB feature points through optical flow, retain the ORB feature points with higher inspection modeling information, use the graph attention network to match the ORB feature points, and transmit the matching valid ORB feature points to the optimization back end; The optimization backend is used to synchronize the time and space of the matching valid ORB feature point pairs with the IMU data and complete the IMU pre-integration to obtain the pose information, and perform backend BA optimization on the ORB feature point pairs and the IMU to update the pose information.
9. A computer-readable storage medium having a computer program or instruction stored therein, It is characterized in that The computer program or instruction is programmed or configured to execute the robust binocular vision SLAM method for distribution network drone inspection as described in any one of claims 1 to 7 through a processor.
10. A computer program product comprising a computer program or instructions, It is characterized in that The computer program or instruction is programmed or configured to execute the robust binocular vision SLAM method for distribution network drone inspection as described in any one of claims 1 to 7 through a processor.
Citation Information
Patent Citations
Dynamic vision SLAM (Simultaneous Localization and Mapping) method based on target detection and feature point speed constraint
CN115115672A
Map construction method and device based on lane line semantics
CN116659482A
Inspection robot self-positioning and mapping method based on plant station semantic information
CN117095054A
Cited By
Cooperative positioning and target guiding method and system for unmanned aerial vehicle cluster
CN122237610A
