Robust binocular vision slam method and system for unmanned aerial vehicle inspection of distribution network

Abstract

The application discloses a robust binocular vision SLAM method and system for power distribution network unmanned aerial vehicle inspection, and comprises the following steps: collecting binocular image sequences and IMU data by using a binocular sensor; performing ORB feature point extraction, filtering and matching on the binocular images by a visual front end, and transmitting the matched effective ORB feature points to an optimization back end; performing time synchronization and space synchronization on the matched effective ORB feature point pairs and the IMU data by the optimization back end, and obtaining pose information by completing IMU pre-integration; and performing back-end BA optimization on the ORB feature point pairs and the IMU to update the pose information. The application aims to support autonomous inspection of the unmanned aerial vehicle based on a pure vision scheme in a power distribution network environment lacking of RTK signals, eliminate part of equipment and environmental interference introduced in the power distribution network line collection process, and ensure the robustness of power line target mapping under the difficult task of unmanned aerial vehicle autonomous inspection.

Description

Technical Field

[0001] This invention pertains to unmanned aerial vehicle (UAV) inspection technology in the field of power distribution networks, specifically relating to a robust binocular vision SLAM method and system for UAV inspection of power distribution networks. Background Technology

[0002] The primary task of a power distribution network is to effectively deliver electrical energy to end users and various electrical appliances. Traditional distribution network line inspections rely on manual surveying and checking for abnormal routes. This method is labor-intensive, time-consuming, and requires highly experienced operators. With technological advancements, drone-based line inspection methods are gradually becoming a new direction in the power inspection field. Simultaneous Localization and Mapping (SLAM) technology, using sensors mounted on drones, can accurately calculate the relative pose of sensors and their surroundings, constructing a sparse point cloud map and simultaneously estimating the drone's instantaneous trajectory, even in the absence of prior environmental information. This technology provides reliable localization and environmental perception for drone-based line inspections. However, in a power distribution network environment, the decoding capability of feature matching algorithms is limited. Current SLAM technology is susceptible to dynamic disturbances between image frames. In complex power distribution network environments, interference from dynamic objects along the line on inspection targets such as poles, distribution boxes, and power lines can lead to significant robustness issues in pose estimation. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a robust binocular vision SLAM method and system for UAV inspection of power distribution networks, addressing the aforementioned problems of existing technologies. This invention aims to support autonomous inspection of UAVs in power distribution networks lacking RTK signals based on a pure vision solution, eliminating some equipment and environmental interference introduced during the data acquisition process of power distribution lines, and ensuring the robustness of power line target mapping under the difficult task of autonomous UAV inspection.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A robust binocular vision SLAM method for UAV inspection of power distribution networks includes: S1. Use a binocular sensor to collect binocular image sequences of the UAV inspecting the power distribution network environment. Obtain the UAV's IMU data through the inertial measurement unit on the UAV. Transmit the acquired binocular images to the vision front end and transmit the UAV's IMU data to the optimization back end. S2, the visual front end performs time synchronization and ORB feature point extraction on the acquired binocular images. The extracted ORB feature points are optically flow removed to remove dynamic feature points, while ORB feature points with high inspection modeling information are retained. The graph attention network is used to match ORB feature points, and the matched ORB feature points are transmitted to the optimization back end. S3, the backend optimization will match the valid ORB feature point pairs with the IMU data for time and space synchronization and complete the IMU pre-integration to obtain pose information. The ORB feature point pairs and the IMU will be optimized by backend BA to update the pose information.

[0005] Optionally, the time synchronization of the acquired stereo images in step S2 includes: for the two cache queues used to cache the acquired left and right stereo images respectively, whenever the acquired left and right stereo images are written to the two cache queues, the top timestamp of each cache queue is retrieved to obtain the timestamp of the left stereo image. Time 0 and right eye image timestamp Time 1; Timestamp the left eye image Time 0 and right eye image timestamp Time 1. Compare, if satisfied Time 0 <Time 1- a If the left eye image is discarded, then the left eye image is discarded; if the following conditions are met... Time 0 Time 1+ a Then discard the right eye image. a This is the preset timestamp difference threshold.

[0006] Optionally, in step S2, the visual front end performs ORB feature point extraction on the acquired binocular images, including: S101, acquire the stereo image data of the current frame, use the image scale pyramid and the fast feature detection algorithm FAST to extract feature points that are significantly different from the surrounding pixels in each pyramid layer; S102, for the extracted feature points, the deviation between the corner point and the neighborhood centroid is calculated using the gray-scale centroid method, and the main direction of spatial motion corresponding to the feature point is determined according to the offset vector; S103, Gaussian smoothing is performed on the stereo image of the current frame. Based on the BRIEF algorithm, a specified number of random point pairs are selected in sequence according to the Gaussian distribution near the extracted feature points and rotated according to the main direction of spatial motion of the feature points. The pixels and size of the random point pair sub-windows before and after rotation are compared and binary encoded to generate a feature descriptor with the same dimension as the number of selected random point pairs as the feature descriptor of the extracted ORB feature points.

[0007] Optionally, the functional expression for calculating the deviation between the corner point and the neighborhood centroid using the gray-scale centroid method in step S102 is as follows: , In the above formula, The deviation between the corner point and the centroid of the neighborhood. The coordinates of the center pixel of the image patch. The coordinates of the centroid are obtained by weighted averaging of the gray values ​​of all pixels within the image window. The calculation function expression for determining the principal direction of spatial motion corresponding to the feature point based on the offset vector is as follows: , In the above formula, The main direction of spatial motion, and Let x and y be the weighted offset vector and their components in the x and y directions, respectively, and we have: , , , , , In the above formula, The magnitude of the offset vector. and Image blocks Offset vectors in the x and y directions.

[0008] Optionally, step S103, which compares the pixels and size of the random point pairs before and after rotation and performs binary encoding to generate a feature descriptor with the same dimension as the number of selected random point pairs, includes: for each pair of random points before and after rotation, if the pixel value of the random point before rotation is less than the pixel value of the random point after rotation, the corresponding bit is set to 1; otherwise, the corresponding bit is set to 0, thereby generating a binary code consisting of 1s and 0s as a feature descriptor with the same dimension as the number of selected random point pairs.

[0009] Optionally, after the visual front end performs time synchronization and ORB feature point extraction on the acquired binocular images in step S2, it also includes detecting and removing target-irrelevant dynamic feature points from the extracted ORB feature points: S201, for the ORB feature point set composed of ORB feature points of each frame of stereo image, determine the feature point set Q of the current frame and the feature point set P of the next frame. Based on the optical flow method, calculate the motion prediction points of the ORB feature points in each frame of the current frame in the feature point set Q in the next frame to obtain the motion feature point set corresponding to the feature point set Q of the current frame. Based on the feature point set Q of the current frame and its corresponding motion feature point set Calculate the fundamental matrix The fundamental matrix This refers to point x i and its prediction point x i+1 satisfy The matrix, and the fundamental matrix The calculation includes: first, constructing a system with extracted ORB feature points and corresponding motion prediction points of size [size missing]. Matrix A, according to Perform singular value decomposition on matrix A, where and For left singular matrices and right singular matrices, It is a diagonal matrix; then based on the right singular matrix based on The size of the solution is singular vectors and singular vectors The following formula is formed as shown. The basic matrix : , In the above formula, ~ singular vectors Elements in; S202, firstly, for the ORB feature points in the feature point set Q of the current frame... q The epipolar lines are calculated using the fundamental matrix, and the function expression for calculating the epipolar lines corresponding to ORB feature points is as follows: , In the above formula, The epipolar lines corresponding to ORB feature points. Based on the matrix, ORB feature points in the feature point set Q of the current frame q The coordinates are then used to calculate the ORB feature points in the feature point set Q of the current frame. q to the poles The Euclidean distance; if the ORB feature points in the feature point set Q of the current frame are... q to the poles If the Euclidean distance is greater than a preset threshold m, the ORB feature point is determined to be a dynamic feature point; otherwise, the ORB feature point is determined to be a static feature point. S203, delete dynamic feature points and their feature descriptors, and retain static feature points and their feature descriptors.

[0010] Optionally, step S2, which uses a graph attention network for ORB feature point matching, includes: S301, For ORB feature points and their feature descriptors, a pre-trained classification network is used to calculate the matching ability of ORB feature points. The classification network consists of a multilayer perceptron (MLP), a linearization layer, and a sigmoid activation function layer to output the matching ability with values ​​ranging from [0,1]. The matching ability refers to the probability that one ORB feature point matches another ORB feature point. The allocation probabilities of all ORB feature points are obtained, and an allocation probability matrix is ​​constructed according to the following formula: , , In the above formula, The probability distribution matrix between ORB feature points i and j. The similarity between feature descriptors of ORB feature points i and j. The similarity between feature descriptors of ORB feature points i and k. This is the metric distance between feature descriptors of ORB feature points i and j. These are adjustment parameters used to control the rate at which similarity decays; S302, Remove ORB feature points with matching ability and whose maximum value in the corresponding dimension of the allocation matrix is ​​lower than a preset threshold; S303. For the remaining ORB feature points, a graph attention network is used to perform ORB feature point matching.

[0011] Furthermore, the present invention also provides a robust binocular vision SLAM system for applying the robust binocular vision SLAM method for UAV inspection of power distribution networks, comprising: The drone is used to collect binocular image sequences by using binocular sensors to inspect the power distribution network environment. The drone's IMU data is obtained through the inertial measurement unit on board the drone. The acquired binocular images are transmitted to the vision front end, and the drone's IMU data is transmitted to the optimization back end. The visual front end is used to extract ORB feature points from the time-synchronized binocular images. The extracted ORB feature points are optically flow-removed to remove dynamic feature points, while ORB feature points with high inspection modeling information are retained. A graph attention network is used to match ORB feature points and transmit the matched ORB feature points to the optimization back end. The backend is optimized to synchronize the matching valid ORB feature point pairs with the IMU data in time and space, and to complete the IMU pre-integration to obtain pose information. The ORB feature point pairs and the IMU are then optimized by backend BA to update the pose information.

[0012] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the robust binocular vision SLAM method for UAV inspection of power distribution networks via a processor.

[0013] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the robust binocular vision SLAM method for UAV inspection of power distribution networks via a processor.

[0014] Compared with the prior art, the beneficial effects of the present invention mainly include: 1. This invention uses a binocular sensor to collect binocular image sequences of the UAV inspection network environment, obtains the UAV's IMU data through the inertial measurement unit on the UAV, and deploys a feasible SLAM algorithm using a three-level architecture of UAV, visual front-end, and optimization back-end to realize pure visual scene mapping of the power distribution network. Relying on the relative distance between the pose of objects in the scene and the UAV, it realizes effective perception and positioning of the UAV without RTK, reducing the cost of power distribution network inspection.

[0015] 2. This invention performs time alignment on the acquired binocular data and IMU data, which alleviates the errors in the acquisition equipment caused by the unstable power supply of the UAV and ensures the accuracy and efficiency of power distribution network line modeling.

[0016] 3. This invention includes a visual front-end that performs time synchronization and ORB feature point extraction on the acquired binocular images. The extracted ORB feature points are then subjected to optical flow removal to eliminate dynamic feature points, retaining ORB feature points with high inspection modeling information. A graph attention network is used to match ORB feature points, and the matched ORB feature points are transmitted to the optimization back-end. This can eliminate some equipment and environmental interference introduced during the acquisition of power distribution lines, ensuring the robustness of power line target mapping under difficult tasks such as UAV autonomous inspection. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the front-end visual odometer in an embodiment of the present invention.

[0019] Figure 3 This is a flowchart of the dynamic point removal process for optical flow in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram illustrating the time synchronization of ORB feature point pairs and IMU data in an embodiment of the present invention.

[0021] Figure 5 This is a flowchart of backend optimization in an embodiment of the present invention. Detailed Implementation

[0022] like Figure 1 As shown, the robust binocular vision SLAM method for UAV inspection of power distribution networks in this embodiment includes: S1. Use a binocular sensor to collect binocular image sequences of the UAV inspecting the power distribution network environment. Obtain the UAV's IMU data through the inertial measurement unit on the UAV. Transmit the acquired binocular images to the vision front end and transmit the UAV's IMU data to the optimization back end. S2, the visual front end performs time synchronization and ORB feature point extraction on the acquired binocular images. The extracted ORB feature points are optically flow removed to remove dynamic feature points, while ORB feature points with high inspection modeling information are retained. The graph attention network is used to match ORB feature points, and the matched ORB feature points are transmitted to the optimization back end. S3, the backend optimization will match the valid ORB feature point pairs with the IMU data for time and space synchronization and complete the IMU pre-integration to obtain pose information. The ORB feature point pairs and the IMU will be optimized by backend BA to update the pose information.

[0023] In step S1 of this embodiment, the UAV is used to collect images of the power distribution network inspection environment via a D435i binocular sensor, resulting in a sequence of binocular images. The onboard inertial measurement unit (IMU) collects IMU data from the UAV. The information acquisition module controls environmental information subscription, performs time synchronization processing on the obtained binocular images, and transmits the collected data to the SLAM vision front-end. When collecting data via the UAV, the takeoff environmental location information is first obtained, and the detailed trajectory of the UAV around the initial pole and the planned waypoints for the inspection route are set. The UAV takes off according to the predetermined method, and the onboard D435i binocular sensor collects image information and IMU information for the power distribution network inspection. The vision front-end specifically uses a Jeston Xavier NX edge computer. The data acquisition module of the Jeston Xavier NX edge computer subscribes to the image and IMU information in real time and processes the time synchronization of the binocular images. During the inspection around the initial power pole, the information acquisition module continuously inputs the binocular images of effective time frames into the front-end visual odometry module configured on the edge computer. Using a feature matching method based on unmatched point elimination, the key point coordinates of the power distribution pole and its equipment are extracted, and the pose matrix of the key points projected onto the SLAM coordinate system is calculated. Combined with the relative distance of the UAV and camera parameters, an initial phase mapping of the takeoff environment is performed in the SLAM coordinate system. Based on the mapping results, the UAV flies along the planned waypoints, and the binocular sensors continuously collect environmental information about the surrounding power poles. The information acquisition module of the edge computer updates the binocular images and IMU information subscriptions in real time.

[0024] In this embodiment, step S2, which synchronizes the acquired stereo images in time, involves saving the stereo images to an image buffer queue. By comparing the timestamps of the stereo images, left and right stereo images with errors are discarded. Specifically, this includes: for the two buffer queues used to cache the acquired left and right stereo images respectively, whenever the acquired left and right stereo images are written to the two buffer queues, the top timestamp of each buffer queue is retrieved to obtain the timestamp of the left stereo image. Time 0 and right eye image timestamp Time 1; Timestamp the left eye image Time 0 and right eye image timestamp Time 1. Compare, if satisfied Time 0 <Time 1- a If the left eye image is discarded, then the left eye image is discarded; if the following conditions are met... Time 0 Time 1+ a Then discard the right eye image. a This is a preset timestamp difference threshold. For example, in this embodiment, the preset timestamp difference threshold is 0.003s, then: Time 0 <Time 1-0.003s, discard the left eye image; Time 0 Time 1 + 0.003s, discard the right eye image.

[0025] The timestamp difference between left and right aligned binocular images cannot exceed 0.003s. Binocular images of the same frame with timestamp differences below the threshold are transmitted to the front-end visual odometry module for feature extraction and matching. The acquisition module continues to perform the above operations until the buffer queue is empty.

[0026] The visual front end mainly includes a binocular image time synchronization module and a front-end visual odometry module. The front-end visual odometry module is used for ORB feature point extraction, optical flow method to remove dynamic points, and feature matching based on graph attention network. Figure 2This is a schematic diagram of the front-end visual odometry module in this embodiment of the invention. The entire ORB feature point extraction process includes feature detection and feature description. The workflow is as follows: 1) Feature Detection: Acquire the stereo image data of the current frame, use an image scale pyramid, and perform a Fast Feature Detection (FAST) algorithm on each pyramid layer to extract feature points that are significantly different from the surrounding pixels. For the extracted feature points, use the Intensity Centroid method to calculate the deviation between the corner points and the neighborhood centroids, and determine the main direction of spatial motion corresponding to the feature points based on the offset vector. 2) Feature Description: Perform Gaussian smoothing on the stereo image of the current frame. Based on the BRIEF algorithm, select a certain number of random point pairs near the extracted feature points according to a Gaussian distribution and rotate them according to the main direction of spatial motion of the feature points. Compare the pixels and size of the sub-windows of the rotated random point pairs and perform binary encoding to generate a feature descriptor with the same dimension as the number of selected random point pairs. Figure 2 As shown, in step S2 of this embodiment, the visual front end extracts ORB feature points from the acquired binocular images, including: S101: Obtain the stereo image data of the current frame, use an image scale pyramid, and use the Fast Feature Detection (FAST) algorithm in each pyramid layer to extract feature points that are significantly different from the surrounding pixels; the Fast Feature Detection (FAST) algorithm is a well-known existing method, see the following literature: Rossen E. Machine learning for high-speed corner detection[C] / / European Conference on Computer Vision.Springer-Verlag,2006.DOI:10.1007 / 11744023_34; S102, for the extracted feature points, the deviation between the corner point and the neighborhood centroid is calculated using the gray-scale centroid method, and the main direction of spatial motion corresponding to the feature point is determined according to the offset vector; S103: Gaussian smoothing is applied to the stereo image of the current frame. Based on the BRIEF algorithm, a specified number of random point pairs are selected sequentially according to a Gaussian distribution near the extracted feature points and rotated according to the main direction of spatial motion of the feature points. The pixel count and size of the random point pair sub-windows before and after rotation are compared and binary encoded to generate feature descriptors with the same dimension as the number of selected random point pairs. These feature descriptors are then used as feature descriptors for the extracted ORB feature points. The BRIEF algorithm is a well-known existing method. For details, please refer to: Calonder M, Lepetit V, Strecha C, et al. BRIEF: Binary Robust Independent Elementary Features[J]. Springer, Berlin, Heidelberg, 2010. DOI:10.1007 / 978-3-642-15561-1_5.

[0027] In step S102 of this embodiment, the functional expression for calculating the deviation between a corner point and its neighboring centroid using the gray-scale centroid method is as follows: , In the above formula, The deviation between the corner point and the centroid of the neighborhood. The coordinates of the center pixel of the image patch. The coordinates of the centroid are obtained by weighted averaging of the gray values ​​of all pixels within the image window. The calculation function expression for determining the principal direction of spatial motion corresponding to the feature point based on the offset vector is as follows: , In the above formula, The main direction of spatial motion, and Let x and y be the weighted offset vector and their components in the x and y directions, respectively, and we have: , , , , , In the above formula, The magnitude of the offset vector. and Image blocks Offset vectors in the x and y directions.

[0028] In step S103 of this embodiment, comparing the pixels and size of the random point pairs before and after rotation and performing binary encoding to generate a feature descriptor with the same dimension as the number of selected random point pairs includes: for each pair of random points before and after rotation, if the pixel value of the random point before rotation is less than the pixel value of the random point after rotation, the corresponding bit is set to 1; otherwise, the corresponding bit is set to 0, thereby generating a binary code composed of 1 and 0 as a feature descriptor with the same dimension as the number of selected random point pairs.

[0029] In this embodiment, the ORB feature point extraction process mainly relies on image texture information for feature detection and description. The extracted feature points involve power distribution network inspection objects and interference objects along the line. In the case of the power distribution network line involved in this example, the inspection objects along the line are mainly power equipment such as power lines, poles, and distribution boxes, which often have static features and are considered static objects in the acquired images. Conversely, pedestrians, insects, birds, and vegetation captured during the acquisition of images along the line have dynamic features. Dynamic disturbances exist between adjacent frames acquired in natural environments, and directly using them as reconstruction targets will affect the mapping effect of the SLAM system. Dynamic optical flow culling of the extracted feature point queue can effectively preserve static inspection objects, thereby avoiding interference and improving the overall robustness of the system. When processing binocular images, the optical flow dynamic point culling method uses the LK optical flow algorithm to perform motion detection on the extracted feature points, identify and cullate dynamic feature points that are irrelevant to the detection target. Specifically, as follows... Figure 3 As shown, in step S2 of this embodiment, after the visual front end performs time synchronization and ORB feature point extraction on the acquired binocular images, it also includes detecting and removing target-irrelevant dynamic feature points from the extracted ORB feature points: S201, for the ORB feature point set composed of ORB feature points of each frame of stereo image, determine the feature point set Q of the current frame and the feature point set P of the next frame. Based on the optical flow method, calculate the motion prediction points of the ORB feature points in each frame of the current frame in the feature point set Q in the next frame to obtain the motion feature point set corresponding to the feature point set Q of the current frame. Based on the feature point set Q of the current frame and its corresponding motion feature point set Calculate the fundamental matrix The fundamental matrix This refers to point x i and its prediction point x i+1 satisfy The matrix, and the fundamental matrix The calculation includes: first, constructing a system with extracted ORB feature points and corresponding motion prediction points of size [size missing]. Matrix A, according to Perform singular value decomposition on matrix A, where and For left singular matrices and right singular matrices, It is a diagonal matrix; then based on the right singular matrix based on The size of the solution is singular vectors and singular vectors The following formula is formed as shown. The basic matrix : , In the above formula, ~ singular vectors The elements in the set; where, based on optical flow, the set of feature points Q of the current frame is the set of motion feature points corresponding to the ORB feature points in each frame in the next frame. The optical flow method used is a well-known method; see the reference: Bouguet, Jean-Yves. "Pyramidal implementation of the affine lucaskanade feature tracker description of the algorithm." Intel Corporation 5.1-10(2001): 4; S202, firstly, for the ORB feature points in the feature point set Q of the current frame... q The epipolar lines are calculated using the fundamental matrix, and the function expression for calculating the epipolar lines corresponding to ORB feature points is as follows: , In the above formula, The epipolar lines corresponding to ORB feature points. Based on the matrix, ORB feature points in the feature point set Q of the current frame q The coordinates are then used to calculate the ORB feature points in the feature point set Q of the current frame. q to the poles The Euclidean distance; if the ORB feature points in the feature point set Q of the current frame are... q to the poles If the Euclidean distance is greater than a preset threshold m, the ORB feature point is determined to be a dynamic feature point; otherwise, the ORB feature point is determined to be a static feature point. S203, delete dynamic feature points and their feature descriptors, and retain static feature points and their feature descriptors. Through this method, by further matching static valid feature points and feature descriptors, mismatches of dynamic feature points with poor matching ability are avoided, and accurate matching feature point pairs are generated.

[0030] In this embodiment, step S2, ORB feature point matching using a graph attention network, includes: S301, For ORB feature points and their feature descriptors, a pre-trained classification network is used to calculate the matching ability of ORB feature points. The classification network consists of a multilayer perceptron (MLP), a linearization layer, and a sigmoid activation function layer to output the matching ability with values ​​ranging from [0,1]. The matching ability refers to the probability that one ORB feature point matches another ORB feature point. The allocation probabilities of all ORB feature points are obtained, and an allocation probability matrix is ​​constructed according to the following formula: , , In the above formula, The probability distribution matrix between ORB feature points i and j. The similarity between feature descriptors of ORB feature points i and j. The similarity between feature descriptors of ORB feature points i and k. This is the metric distance between feature descriptors of ORB feature points i and j. These are adjustment parameters used to control the rate at which similarity decays; S302, Remove ORB feature points with matching ability and whose maximum value in the corresponding dimension of the allocation matrix is ​​lower than a preset threshold; S303. For the remaining ORB feature points, a graph attention network is used for ORB feature point matching. For the left and right binocular images, the feature points from which ORB features have been extracted are matched against each other. For example, if a feature point in the left image is the top left corner of the blackboard, then the ORB feature points of the top left corners of the blackboard in the right image at different pixel coordinates are matched with the feature points in the left image. It should be noted that using a graph attention network for ORB feature point matching is a well-known method, as detailed in the following literature: Sarlin PE, DeTone D, Malisiewicz T, et al. Superglue: Learning featurematching with graph neural networks[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2020: 4938-4947.

[0031] This embodiment extracts paired stereo image feature sets to be matched from the feature point queue. The matching ability of the feature points in the set is calculated using a multilayer perceptron (MLP) classification network. Feature points with poor matching ability (low matching probability) are removed to avoid mismatches in the early stages of matching. The classification network consists of an MLP, a linearization layer, and a sigmoid activation function layer to output the matching ability with values ​​ranging from [0,1]. The loss function used during training is: , , , In the above formula, For loss function, The loss is calculated after assessing the matching ability for the ORB feature point set. To eliminate the loss from optimized allocation after removing low matching ability, and To lose weight, M This is the set of true matching ORB feature point pairs. and These are sets of non-realistically matched ORB feature point pairs from stereo images. P To assign the probability matrix, The matching ability corresponding to ORB feature points. i and j These are the ORB feature points to be matched.

[0032] Step S3 involves optimizing the backend to synchronize the matching valid ORB feature point pairs with the IMU data in both time and space, and performing IMU pre-integration to obtain pose information. Backend BA (Bundle Adjustment) optimization of the ORB feature point pairs and the IMU to update the pose information are both well-known methods. Figure 4 As shown, in one embodiment, ORB matching feature point pairs and IMU data are synchronized in time and space. The workflow is as follows: 1) Constrain the timestamps of the IMU data. Specifically, IMU data within the time range of the timestamps of the matching feature point pairs in the previous frame and the current frame are retained as the inertial navigation processing data for the current frame to ensure time synchronization. The specific formula is: t 0 <=t <t 1; In the formula, t 0 represents the timestamp of the feature point pair in the previous frame of the ORB. t 1 represents the timestamp of the feature point pairs in the current ORB frame. t The timestamps for IMU data need to be retained. t 0 andt 1) Ensure time synchronization of IMU data between the camera and inertial measurement sensor. 2) Calibrate the intrinsic parameters of the 435i camera, the intrinsic parameters of the inertial measurement sensor, and the extrinsic parameters between the camera and the inertial measurement sensor to achieve spatial synchronization. For example... Figure 5 As shown, in one embodiment, the backend pose optimization method determines marginalized frames based on inter-frame disparity relationships and optimizes matching feature points and IMU data based on the BA algorithm to update the pose information of the current frame change. The working steps are as follows: 1) The backend receives the current frame and performs disparity judgment. If the disparity is small, the previous frame is marginalized; if the disparity is large, the first frame is marginalized. The ORB feature points are fused with the IMU to determine the system initialization status. If it has been initialized, proceed to step 4). 2) The image depth is calculated from the stereo image, and the initial pose information is calculated using 3d-2d PnP. 3) Ensure that the number of image frames is equal to the size of the sliding window, perform IMU pre-integration repropagation, optimize the prior information of the marginalization, the residual information of the IMU, and the residual information of the image, update the pose information, and update the sliding window according to the marginalized frames determined by disparity judgment. Initialization is completed. 4) While processing IMU information for pre-integration, pose calculation is performed using IMU information to further remove invalid points and update the pose information. The sliding window is updated according to the marginalized frames determined by disparity judgment. Determine if detection fails; if so, proceed to step 5). 5) Clear all states, initialize all parameters, and the system re-subscribes to image and IMU information, performs detection and optimization. Specifically, in this embodiment, after the front-end visual odometry completes feature point extraction for a new frame, it sends the extracted feature points to the back-end optimization module. The back-end optimization module first synchronizes the IMU with the image frame, acquires the IMU data between the previous and current frames, performs pre-integration, and calculates the spatial pose corresponding to the extracted feature points based on the alignment results. If the system has not been initialized, initialization must be performed first. Back-end optimization can effectively reduce physical errors in the acquisition equipment and inertial navigation unit, improving the robustness of mapping during UAV inspection.

[0033] In summary, this embodiment of a robust binocular vision SLAM method for distribution network drone inspection uses binocular sensors to acquire binocular image sequences of the distribution network environment inspected by the drone. The drone's IMU data is acquired via its onboard inertial measurement unit (IMU). The acquired binocular images are time-synchronized and transmitted to the vision front end. The vision front end performs time synchronization and ORB feature point extraction on the acquired binocular images. Dynamic feature points are removed from the extracted ORB feature points using optical flow, retaining those with high inspection modeling information. A graph attention network is used for ORB feature point matching, and the valid matching ORB feature points are transmitted to the optimization back end. The optimization back end synchronizes the valid matching ORB feature point pairs with the IMU data in both time and space, performs IMU pre-integration to obtain pose information, and performs backend BA optimization on the ORB feature point pairs and the IMU to update the pose information. This robust binocular vision SLAM method for distribution network drone inspection can perform drone perception and self-localization in distribution network environments lacking RTK signals, significantly improving the robustness of outdoor inspection targets and making it suitable for distribution network drone inspection tasks. This embodiment uses a binocular sensor to acquire binocular image sequences from a drone's inspection of the power distribution network environment. The drone's IMU data is obtained through its onboard inertial measurement unit (IMU). A feasible SLAM algorithm is deployed using a three-tier architecture consisting of the drone, a vision front-end, and an optimization back-end to achieve pure visual scene mapping of the power distribution network. Relying on the relative distance between the drone and the poses of objects in the scene, it achieves effective perception and localization of the drone without RTK, reducing the cost of power distribution network inspection. This embodiment performs time alignment on the acquired binocular and IMU data to mitigate errors in the acquisition equipment caused by unstable power supply to the drone, ensuring the accuracy and efficiency of power distribution network modeling. This embodiment includes a vision front-end that performs time synchronization and ORB feature point extraction on the acquired binocular images. The extracted ORB feature points are then subjected to optical flow removal to eliminate dynamic feature points, retaining ORB feature points with high inspection modeling information. A graph attention network is used for ORB feature point matching, and the matching ORB feature points are transmitted to the optimization back-end. This process can eliminate some equipment and environmental interference introduced during the power distribution network acquisition process, ensuring the robustness of power line target mapping under challenging tasks such as autonomous drone inspection.

[0034] Furthermore, this embodiment also provides a robust binocular vision SLAM system for applying the robust binocular vision SLAM method for power distribution network UAV inspection, comprising: The drone is used to collect binocular image sequences by using binocular sensors to inspect the power distribution network environment. The drone's IMU data is obtained through the inertial measurement unit on board the drone. The acquired binocular images are transmitted to the vision front end, and the drone's IMU data is transmitted to the optimization back end. The visual front end is used to extract ORB feature points from the time-synchronized binocular images. The extracted ORB feature points are optically flow-removed to remove dynamic feature points, while ORB feature points with high inspection modeling information are retained. A graph attention network is used to match ORB feature points and transmit the matched ORB feature points to the optimization back end. The backend is optimized to synchronize the matching valid ORB feature point pairs with the IMU data in time and space, and to complete the IMU pre-integration to obtain pose information. The ORB feature point pairs and the IMU are then optimized by backend BA to update the pose information.

[0035] In addition, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the robust binocular vision SLAM method for UAV inspection of power distribution networks via a processor.

[0036] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the robust binocular vision SLAM method for UAV inspection of power distribution networks via a processor.

[0037] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0038] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A robust binocular vision SLAM method for UAV inspection of power distribution networks, characterized in that, include: S1. Use a binocular sensor to collect binocular image sequences of the UAV inspecting the power distribution network environment. Obtain the UAV's IMU data through the inertial measurement unit on the UAV. Transmit the acquired binocular images to the vision front end and transmit the UAV's IMU data to the optimization back end. S2, the visual front end performs time synchronization and ORB feature point extraction on the acquired binocular images. The extracted ORB feature points are optically flow removed to remove dynamic feature points, while ORB feature points with high inspection modeling information are retained. The graph attention network is used to match ORB feature points, and the matched ORB feature points are transmitted to the optimization back end. S3, the backend will match the valid ORB feature point pairs with the IMU data for time and space synchronization and complete the IMU pre-integration to obtain pose information. The ORB feature point pairs and the IMU will be optimized by backend BA to update the pose information. In step S2, after the visual front end performs time synchronization and ORB feature point extraction on the acquired binocular images, it also includes detecting and removing target-irrelevant dynamic feature points from the extracted ORB feature points: S201, for the ORB feature point set composed of ORB feature points of each frame of stereo image, determine the feature point set Q of the current frame and the feature point set P of the next frame. Based on the optical flow method, calculate the motion prediction points of the ORB feature points in each frame of the current frame in the feature point set Q in the next frame to obtain the motion feature point set corresponding to the feature point set Q of the current frame. Based on the feature point set Q of the current frame and its corresponding motion feature point set Calculate the fundamental matrix The fundamental matrix This refers to point x i and its prediction point x i+1 satisfy The matrix, and the fundamental matrix The calculation includes: first, constructing a system with extracted ORB feature points and corresponding motion prediction points of size [size missing]. Matrix A, according to Perform singular value decomposition on matrix A, where and For left singular matrices and right singular matrices, It is a diagonal matrix; then based on the right singular matrix based on The size of the solution is singular vectors and singular vectors The following formula is formed as shown. The basic matrix : ， In the above formula, ~ singular vectors Elements in; S202, firstly, for the ORB feature points in the feature point set Q of the current frame... q The epipolar lines are calculated using the fundamental matrix, and the function expression for calculating the epipolar lines corresponding to ORB feature points is as follows: ， In the above formula, The epipolar lines corresponding to ORB feature points. Based on the matrix, ORB feature points in the feature point set Q of the current frame q The coordinates are then used to calculate the ORB feature points in the feature point set Q of the current frame. q to the poles The Euclidean distance; if the ORB feature points in the feature point set Q of the current frame are... q to the poles If the Euclidean distance is greater than a preset threshold m, the ORB feature point is determined to be a dynamic feature point; otherwise, the ORB feature point is determined to be a static feature point. S203, delete dynamic feature points and their feature descriptors, and retain static feature points and their feature descriptors.

2. The robust binocular vision SLAM method for UAV inspection of power distribution networks according to claim 1, characterized in that, Step S2, which involves time synchronization of the acquired stereo images, includes: for the two cache queues used to cache the acquired left and right stereo images respectively, whenever the acquired left and right stereo images are written to the two cache queues, the top timestamp of each cache queue is retrieved to obtain the timestamp of the left stereo image. Time 0 and right eye image timestamp Time 1; Timestamp the left eye image Time 0 and right eye image timestamp Time 1. Compare, if satisfied Time 0 <Time 1- a If the left eye image is discarded, then the left eye image is discarded; if the following conditions are met... Time 0 Time 1+ a Then discard the right eye image. a This is the preset timestamp difference threshold.

3. The robust binocular vision SLAM method for UAV inspection of power distribution networks according to claim 1, characterized in that, In step S2, the visual front end performs ORB feature point extraction on the acquired binocular images, including: S101, acquire the stereo image data of the current frame, use the image scale pyramid and the fast feature detection algorithm FAST to extract feature points that are significantly different from the surrounding pixels in each pyramid layer; S102, for the extracted feature points, the deviation between the corner point and the neighborhood centroid is calculated using the gray-scale centroid method, and the main direction of spatial motion corresponding to the feature point is determined according to the offset vector; S103, Gaussian smoothing is performed on the stereo image of the current frame. Based on the BRIEF algorithm, a specified number of random point pairs are selected in sequence according to the Gaussian distribution near the extracted feature points and rotated according to the main direction of spatial motion of the feature points. The pixels and size of the random point pair sub-windows before and after rotation are compared and binary encoded to generate a feature descriptor with the same dimension as the number of selected random point pairs as the feature descriptor of the extracted ORB feature points.

4. The robust binocular vision SLAM method for UAV inspection of power distribution networks according to claim 3, characterized in that, In step S102, the function expression for calculating the deviation between the corner point and the neighborhood centroid using the gray-scale centroid method is as follows: ， In the above formula, The deviation between the corner point and the centroid of the neighborhood. The coordinates of the center pixel of the image patch. The coordinates of the centroid are obtained by weighted averaging of the gray values ​​of all pixels within the image window. The calculation function expression for determining the main spatial motion direction corresponding to the feature point based on the offset vector is as follows: ， In the above formula, The main direction of spatial motion, and Let x and y be the weighted offset vector and their components in the x and y directions, respectively, and we have: ， ， ， ， ， In the above formula, The magnitude of the offset vector. and Image blocks Offset vectors in the x and y directions.

5. The robust binocular vision SLAM method for UAV inspection of power distribution networks according to claim 3, characterized in that, Step S103, which compares the pixels and size of the random point pairs before and after rotation and performs binary encoding to generate a feature descriptor with the same dimension as the number of selected random point pairs, includes: for each pair of random points before and after rotation, if the pixel value of the random point before rotation is less than the pixel value of the random point after rotation, the corresponding bit is set to 1; otherwise, the corresponding bit is set to 0, thereby generating a binary code consisting of 1s and 0s as a feature descriptor with the same dimension as the number of selected random point pairs.

6. The robust binocular vision SLAM method for UAV inspection of power distribution networks according to claim 1, characterized in that, Step S2, using a graph attention network for ORB feature point matching, includes: S301, For ORB feature points and their feature descriptors, a pre-trained classification network is used to calculate the matching ability of ORB feature points. The classification network consists of a multilayer perceptron (MLP), a linearization layer, and a sigmoid activation function layer to output the matching ability with values ​​ranging from [0,1]. The matching ability refers to the probability that one ORB feature point matches another ORB feature point. The allocation probabilities of all ORB feature points are obtained, and an allocation probability matrix is ​​constructed according to the following formula: ， ， In the above formula, The probability distribution matrix between ORB feature points i and j. The similarity between feature descriptors of ORB feature points i and j. The similarity between feature descriptors of ORB feature points i and k. This is the metric distance between feature descriptors of ORB feature points i and j. These are adjustment parameters used to control the rate at which similarity decays; S302, Remove ORB feature points with matching ability and whose maximum value in the corresponding dimension of the allocation matrix is ​​lower than a preset threshold; S303. For the remaining ORB feature points, a graph attention network is used to perform ORB feature point matching.

7. A robust binocular vision SLAM system for applying the robust binocular vision SLAM method for UAV inspection of power distribution networks as described in any one of claims 1 to 6, characterized in that, include: The drone is used to collect binocular image sequences by using binocular sensors to inspect the power distribution network environment. The drone's IMU data is obtained through the inertial measurement unit on board the drone. The acquired binocular images are transmitted to the vision front end, and the drone's IMU data is transmitted to the optimization back end. The visual front end is used to extract ORB feature points from the time-synchronized binocular images. The extracted ORB feature points are optically flow-removed to remove dynamic feature points, while ORB feature points with high inspection modeling information are retained. A graph attention network is used to match ORB feature points and transmit the matched ORB feature points to the optimization back end. The backend is optimized to synchronize the matching valid ORB feature point pairs with the IMU data in time and space, and to complete the IMU pre-integration to obtain pose information. The ORB feature point pairs and the IMU are then optimized by backend BA to update the pose information.

8. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the robust binocular vision SLAM method for UAV inspection of power distribution networks as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the robust binocular vision SLAM method for UAV inspection of power distribution networks as described in any one of claims 1 to 6.

Images