A machine vision-based unmanned aerial vehicle (UAV) robotic arm control system and method

Abstract

This invention discloses a machine vision-based unmanned aerial vehicle (UAV) robotic arm control system and method, relating to the field of robotic arm control technology. The system includes: an image acquisition module, a feature point extraction and matching module, a pose analysis module, a control parameter calculation and adjustment module, and a motion control and execution module. The image acquisition module acquires image data of the target area; the feature point extraction and matching module extracts feature points of the target object from the reference image, constructs a feature set, and performs feature point extraction and matching on real-time image data, filtering valid matching points; the pose estimation module analyzes the actual pose of the UAV robotic arm based on the matching points and calculates the actual and desired poses through an optimization algorithm; the control parameter calculation and adjustment module calculates the deviation between the desired and actual poses, sets the control gain, and updates the motion control parameters; the motion control and execution module updates the motion trajectory of the UAV robotic arm in real time based on the adjusted control parameters.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control technology, specifically to a machine vision-based unmanned aerial vehicle (UAV) robotic arm control system and method. Background Technology

[0002] With the rapid development of robotics technology, robotic arms for power line maintenance have been gradually applied to power transmission line maintenance operations. This has improved work efficiency, reduced the cost of training professionals, and enhanced operational safety. However, the movement trajectory of the robotic arm needs to be adjusted according to different tasks. Therefore, rapid planning of the robotic arm's movements has become a new development requirement. Drones equipped with robotic arms, controlled by maintenance personnel to grasp objects, are currently in their initial stages both domestically and internationally. They can be used in various power operation and maintenance scenarios, and their future applications are expected to be very broad, with enormous potential for commercialization.

[0003] However, in practical applications, existing drone robotic arm control systems have the following shortcomings: Existing systems have limited ability to identify the features of power facilities in complex environments (such as changes in light and shadow interference), especially in the detection and positioning of small objects (such as insulators and bolts), where they are prone to missed or false detections. This directly leads to a decrease in the robotic arm's grasping accuracy and increases operational risks. Furthermore, existing systems typically rely on static environmental models and have weak adaptability to dynamic environments. In power line inspection operations, the environment may constantly change due to weather, equipment variations, and other factors. Existing systems often cannot quickly adjust their trajectories, resulting in reduced operational efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a machine vision-based unmanned aerial vehicle (UAV) robotic arm control system and method to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A machine vision-based method for controlling a drone robotic arm, comprising the following steps:

[0007] Step S100. Obtain reference image data for UAV power line inspection operations, extract target object feature points from the reference image data, analyze the target object feature points for power line inspection operations, and thus construct a target object feature set.

[0008] Step S200. Acquire real-time image data of UAV power line inspection operation, extract real-time image feature points from real-time image data, perform matching analysis between real-time image feature points and target object feature set, filter out matching feature points, and mark target objects in real-time historical image data according to matching feature points.

[0009] Step S300. Based on the matching feature points obtained from the matching analysis, analyze the actual pose of the robotic arm in the UAV power inspection operation, and use the intrinsic and extrinsic parameters of the UAV camera to convert the three-dimensional coordinates of the target object into pose information in the desired grasping position coordinate system, thereby obtaining the desired pose.

[0010] Step S400. Calculate the deviation between the actual pose and the desired pose of the UAV robotic arm, and adjust the motion control parameters of the UAV robotic arm based on the deviation; adjust the current motion trajectory of the UAV robotic arm accordingly based on the adjusted motion control parameters.

[0011] Furthermore, step S100 includes:

[0012] S101. Using the camera equipment of a drone, conduct power line inspections according to a preset flight path, collect multi-angle, multi-time image data of the target area, and use the collected image data as reference image data; preprocess the reference image data, and extract the preprocessed reference image data I... f Starting with (x,y), each layer of the pyramid is constructed; the image Gk(x,y) of each layer is obtained through Gaussian blur and downsampling, as follows:

[0013] For the first layer image G0(x,y)=I f Gaussian blur is applied to (x,y), and downsampling is used to obtain the next layer image: G k (x,y)=Downsample(G k-1 (2x, 2y)); For each layer k, the following Gaussian formula is used for calculation:

[0014] G k (x,y)=(1 / 2πσ k 2 )e^[-(x 2 +y 2 ) / (2σ k 2 ),

[0015] Where (x,y) represents the position coordinates of a pixel in the Gaussian image, σ k This represents the standard deviation of the k-th layer;

[0016] S102. Divide each Gaussian image into quadtrees, and define each grid region S as: S={(x,y)|0≤x<W,0≤y<H}, where W and H are the width and height of the Gaussian image, respectively; execute the FAST algorithm within each grid region, and use non-maximum suppression around each detected candidate feature point to retain the feature point with the highest response value as the target object feature point; for grids with a small number of detected feature points, lower the threshold and re-detect; calculate the image moments of all target object feature points, using the following formula:

[0017] m pq =∑ x∈[0,W1] ∑ y∈[0,H1] I(x,y)·x p ·y q ,

[0018] Where I(x,y) represents the pixel value of the current grid region at coordinates (x,y), W1 and H1 represent the width and height of the current grid region, respectively, and p and q are the orders of the moments; based on the image moments, the centroid C of all target object feature points is calculated, with the following formula:

[0019] C = (C x C y )=(m 10 / m 00 ,m 01 / m 00 ),

[0020] Where C x C represents the coordinate of the centroid in the x-direction. y This represents the coordinate of the centroid in the y-direction; m 10 represents the first moment, the centroid of the current mesh region in the x-direction; m 00 The zeroth moment of the current grid region is equal to the sum of all pixel values; m 01 This represents the first moment, and the centroid of the current mesh region in the y-direction; the orientation angle θ of the target object's feature points is calculated using the following formula:

[0021] θ = arctan(C) y / C x ) = arctan(m 01 / m 10 );

[0022] S103. Generate corresponding feature point descriptors based on the centroid position and orientation angle of each target object feature point, and summarize the feature points and feature point descriptors of the same target object to form the target object feature set Kd.

[0023] Furthermore, the FAST algorithm is executed within each grid region. Around each detected candidate feature point, a non-maximum suppression method is used to retain the feature point with the highest response value as the target object feature point. The specific analysis process is as follows:

[0024] For each detected candidate feature point I(x,y), a circular region with radius r is defined around it. Typically, r=3 is chosen, meaning 16 pixels around it are considered. For each candidate feature point I(x,y), the corresponding brightness value Ic is obtained. Within the circular region, the brightness values ​​Ii of the corresponding N neighboring pixels are obtained, where i represents the neighboring pixel number, which is a positive integer from 1 to N. If there are n neighboring pixels that satisfy: Ii>Ic+T, or Ii<Ic-T, where T is a set threshold, then the current candidate feature point I(x,y) is retained as a candidate corner point.

[0025] The edge responsivity R of each candidate corner point is calculated using the following formula:

[0026] R = det(M) - k·trace(M) 2 ,

[0027] Where M is the structure tensor of the image, det(M) represents the determinant of the structure tensor, trace(M) represents the trace of the structure tensor, and k represents an empirical constant. For each candidate corner point, non-maximum suppression is performed within a radius r around it. Specifically, within the circular region of the candidate corner point with a radius r, the response value R of that point is compared with the response values ​​of all other points in the neighborhood. Only when the response value R of that point is the largest in its neighborhood will it be retained.

[0028] Furthermore, step S200 includes:

[0029] S201. Using the camera equipment of the drone, capture real-time image data during power line inspection operations, and store these images as an image sequence It, where It = {I_1, I_2, ..., I_h}, where I_1 represents the real-time image corresponding to the first time point, I_2 represents the real-time image corresponding to the second time point, and so on, I_h represents the real-time image corresponding to the h-th time point, and h represents the time index; for each real-time image in the image sequence It, analyze the target object feature points as described in step S100 to obtain the real-time image feature points and corresponding descriptors;

[0030] S202. Obtain the target object feature set Kd. For each target object in the target object feature set Kd, perform brute-force matching between the descriptor corresponding to each target object and the descriptor of each real-time image in the image sequence It. Calculate the Hamming distance Dh between the corresponding descriptors and filter out matching pairs that are less than the threshold Dh0. Use the RANSAC algorithm to remove mismatch points. Find the inlier set Nd based on the matching point pairs and count the number of inliers in the inlier set. If the number of inliers is greater than or equal to 4, it is considered a valid match. Thus, the matching feature points are obtained, and the target object is marked in each real-time image in the image sequence It based on the matching feature points.

[0031] Furthermore, step S300 includes:

[0032] S301. For each real-time image in the image sequence It, select any matching feature point of the marked target object in the first real-time image of the image sequence It as the center to construct an initial two-dimensional image coordinate system; map each real-time image in the image sequence It sequentially according to the initial two-dimensional image coordinate system, so that each real-time image in the image sequence It is mapped to the initial two-dimensional image coordinate system; the specific mapping process is as follows:

[0033] From the first real-time image I_1, select a matching feature point of the marked target object as the center, denoted as C = (x_c, y_c). Establish an initial two-dimensional coordinate system (X', Y') with C as the center. The coordinate transformation relationship is as follows: X' = x - x_c, Y' = y - y_c, thus ensuring that the position of C in the initial two-dimensional coordinate system is (0, 0). Based on the initial two-dimensional coordinate system, perform coordinate transformation on all matching feature points in real-time image I_1 to obtain the position coordinates of all matching feature points in real-time image I_1 in the initial two-dimensional coordinate system. For all matching feature points in the second real-time image I_2... For feature points, any matching feature point existing in both real-time image I_1 and real-time image I_2 is selected as a reference point. Based on the mapping relationship between the position coordinates of the reference point in real-time image I_2 and its position coordinates in real-time image I_1, the other matching feature points in real-time image I_2 are transformed and calculated accordingly to obtain the position coordinates of all matching feature points in real-time image I_2 in the initial two-dimensional coordinate system. Similarly, the above transformation and calculation are performed on all real-time images in image sequence It to obtain the position coordinates of the matching feature points of all real-time images in image sequence It in the initial two-dimensional coordinate system.

[0034] S302. Obtain the position coordinates of all matching feature points in the initial two-dimensional coordinate system, denoted as: p_g=(x_g,y_g), where g is the index of the matching feature point in the initial two-dimensional coordinate system; convert the two-dimensional coordinates of each matching feature point into homogeneous coordinates p_g'=(x_g,y_g,1). T Define the position coordinates of the matching feature points corresponding to the target object in the three-dimensional world coordinate system as G_g=(X_g,Y_g,Z_g). T Using the intrinsic parameter matrix K and extrinsic parameter matrix [Rr|tr] of the UAV camera device, where Rr represents the rotation matrix and tr represents the translation vector, it is expressed as: p_g=K[Rr|tr]G_g; using the equations of all matching feature points, an error function is established:

[0035] E(Rr,tr)=∑g∈[1,M],||p_g'-K[Rr|tr]G_g|| 2 ,

[0036] Where M represents the total number of matching feature points; the optimized rotation and translation parameters are obtained by minimizing E(Rr,tr), thereby obtaining the actual pose T_actual of the robotic arm;

[0037] S303. Let the origin of the desired capture position coordinate system be O'. Its relationship with the target object's 3D world coordinate system is determined by the rotation matrix R_target and the translation vector t_target, specifically: G_g' = R_targetG_g + t_target, where G_g' represents the target object's coordinates in the desired capture position coordinate system. For each matching feature point G_g, using the optimized rotation matrix Rr and translation vector tr, the target object's 3D coordinates G_g are converted into homogeneous coordinates p_g' on the image plane through the camera model. P_g' is then mapped back to the 3D world coordinate system to obtain the target object's position in the world coordinate system: G_g = K -1 p_g'; Using the relationship between the defined desired grasping position coordinate system and the target object coordinate system, G_g is converted to G_g'; Through the above conversion, the three-dimensional coordinates G_g' of the target object in the desired grasping position coordinate system are obtained, thus obtaining the desired pose T_desired of the robotic arm.

[0038] Furthermore, step S400 includes:

[0039] S401. Transform the desired pose T_desired of the drone robotic arm from the desired grasping position coordinate system to the three-dimensional world coordinate system corresponding to the actual pose, thereby obtaining the desired pose T_desired_actual in the three-dimensional world coordinate system, and T_desired_actual = (T_actual).-1 T_desired, where (T_actual) -1 This represents the inverse transformation of the actual pose; calculate the deviation F between the desired pose T_desired_actual and the actual pose T_actual, and F = ||Δt|| + R_e, where Δt represents the displacement deviation, ||Δt|| represents the magnitude of the displacement deviation, and R_e represents the rotation error measure;

[0040] S402. Let the control gains be Kp and Kr, which are used for adjusting displacement and rotation respectively. Calculate the adjusted displacement control parameter Δt_control = KpΔt and the adjusted rotation control parameter ΔR_control = KrR_e. Update the motion control parameters of the UAV robotic arm according to the calculated adjustment parameters Δt_control and ΔR_control. And update the motion trajectory of the UAV robotic arm in real time based on the adjusted displacement and rotation control parameters.

[0041] A machine vision-based unmanned aerial vehicle (UAV) robotic arm control system includes: an image acquisition module, a feature point extraction and matching module, a pose analysis module, a control parameter calculation and adjustment module, and a motion control and execution module;

[0042] The image acquisition module uses the camera equipment of the drone to conduct power line inspections along a preset path, collecting image data of the target area; it processes the collected image data, generates reference image data, and records images from multiple angles and at multiple times.

[0043] The feature point extraction and matching module extracts feature points of the target object from the reference image data and constructs a target object feature set. It then extracts feature points from the real-time image data and matches them with the target object feature set to filter out valid matching feature points.

[0044] The pose analysis module analyzes the actual pose of the drone's robotic arm based on the matching feature points, converts the three-dimensional coordinates of the target object into pose information in the coordinate system of the desired grasping position, and calculates the actual pose and desired pose of the robotic arm through an optimization algorithm.

[0045] The control parameter calculation and adjustment module calculates the deviation between the desired pose and the actual pose, analyzes the displacement and rotation errors, sets the control gain, calculates the adjusted displacement and rotation control parameters, and updates the motion control parameters of the UAV robotic arm.

[0046] The motion control and execution module updates the motion trajectory of the drone's robotic arm in real time based on the adjusted motion control parameters.

[0047] Furthermore, the image acquisition module includes an image acquisition unit and an image storage unit;

[0048] The image acquisition unit is responsible for capturing image data during power line inspection operations; the image storage unit stores the captured image data in a time sequence.

[0049] The feature point extraction and matching module includes an image preprocessing unit, a feature point extraction unit, and a feature point matching unit;

[0050] The image preprocessing unit preprocesses the acquired images; the feature point extraction unit uses techniques such as the FAST algorithm to extract the feature points of the target object in the preprocessed image and calculates the descriptors of the feature points of the target object; the feature point matching unit matches the feature points in the real-time image with the feature points in the reference image, uses brute-force matching and the RANSAC algorithm to remove incorrect matches, and filters out the valid matching feature points.

[0051] Furthermore, the pose analysis module includes a coordinate system mapping unit, an actual pose calculation unit, and a desired pose calculation unit;

[0052] The coordinate system mapping unit maps the matching feature points in the real-time image to the initial two-dimensional coordinate system and performs position transformation; the actual pose calculation unit calculates the three-dimensional world coordinates of the matching feature points using the camera's intrinsic and extrinsic parameters, and establishes an error function to obtain the actual pose of the UAV robotic arm; the desired pose calculation unit calculates the desired pose of the robotic arm based on the three-dimensional coordinates of the target object and its relationship with the desired grasping position coordinate system.

[0053] Furthermore, the control parameter calculation and adjustment module includes a deviation calculation unit and a control parameter adjustment unit;

[0054] The deviation calculation unit calculates the deviation between the actual pose and the desired pose of the UAV robotic arm, including displacement and rotation errors; the control parameter adjustment unit calculates the adjusted displacement and rotation control parameters based on the set control gain.

[0055] Compared with existing technologies, the beneficial effects of this invention are as follows: By constructing a target object feature set and performing real-time image feature point matching, this invention effectively improves the recognition accuracy of small objects (such as insulators, bolts, etc.), significantly reducing the occurrence of missed and false detections. This improvement reduces operational risks and improves the accuracy of robotic arm grasping. Unlike traditional systems that rely on static environment models, this invention can analyze and match target object feature points in real time, adapting to complex and dynamically changing environments, such as weather changes or equipment displacement. This flexibility allows drones to adjust their trajectories more quickly during inspection operations, improving operational efficiency. This invention acquires image data in real time and combines it with the camera's intrinsic and extrinsic parameters to identify target objects. The three-dimensional coordinates are converted into pose information in the desired grasping position coordinate system. This dynamic calculation capability enables the robotic arm to react quickly and adjust motion control parameters, further improving grasping accuracy and operational safety. This invention employs multi-angle, multi-time image data acquisition methods and uses preprocessing techniques such as Gaussian blurring and downsampling to ensure feature point detection performance under different environmental conditions. This method improves the system's adaptability and robustness to complex backgrounds. By calculating the deviation between the desired pose and the actual pose and using control gain for real-time adjustment of displacement and rotation, this invention effectively improves the motion control accuracy of the robotic arm. This mechanism not only enhances system stability but also improves operational efficiency. Attached Figure Description

[0056] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0057] Figure 1 This is a schematic diagram of a machine vision-based unmanned aerial vehicle (UAV) robotic arm control system module according to the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Please see Figure 1 The present invention provides the following technical solution:

[0060] A machine vision-based unmanned aerial vehicle (UAV) robotic arm control system includes: an image acquisition module, a feature point extraction and matching module, a pose analysis module, a control parameter calculation and adjustment module, and a motion control and execution module;

[0061] The image acquisition module uses the camera equipment of the drone to conduct power line inspections along a preset path, collecting image data of the target area; it processes the collected image data, generates reference image data, and records images from multiple angles and at multiple times.

[0062] The feature point extraction and matching module extracts feature points of the target object from the reference image data and constructs a target object feature set. It then extracts feature points from the real-time image data and matches them with the target object feature set to filter out valid matching feature points.

[0063] The pose analysis module analyzes the actual pose of the drone's robotic arm based on the matching feature points, converts the three-dimensional coordinates of the target object into pose information in the coordinate system of the desired grasping position, and calculates the actual pose and desired pose of the robotic arm through an optimization algorithm.

[0064] The control parameter calculation and adjustment module calculates the deviation between the desired pose and the actual pose, analyzes the displacement and rotation errors, sets the control gain, calculates the adjusted displacement and rotation control parameters, and updates the motion control parameters of the UAV robotic arm.

[0065] The motion control and execution module updates the motion trajectory of the drone's robotic arm in real time based on the adjusted motion control parameters.

[0066] The image acquisition module includes an image acquisition unit and an image storage unit;

[0067] The image acquisition unit is responsible for capturing image data during power line inspection operations; the image storage unit stores the captured image data in a time sequence.

[0068] The feature point extraction and matching module includes an image preprocessing unit, a feature point extraction unit, and a feature point matching unit;

[0069] The image preprocessing unit preprocesses the acquired images; the feature point extraction unit uses techniques such as the FAST algorithm to extract the feature points of the target object in the preprocessed image and calculates the descriptors of the feature points of the target object; the feature point matching unit matches the feature points in the real-time image with the feature points in the reference image, uses brute-force matching and the RANSAC algorithm to remove incorrect matches, and filters out the valid matching feature points.

[0070] The pose analysis module includes a coordinate system mapping unit, an actual pose calculation unit, and a desired pose calculation unit.

[0071] The coordinate system mapping unit maps the matching feature points in the real-time image to the initial two-dimensional coordinate system and performs position transformation; the actual pose calculation unit calculates the three-dimensional world coordinates of the matching feature points using the camera's intrinsic and extrinsic parameters, and establishes an error function to obtain the actual pose of the UAV robotic arm; the desired pose calculation unit calculates the desired pose of the robotic arm based on the three-dimensional coordinates of the target object and its relationship with the desired grasping position coordinate system.

[0072] The control parameter calculation and adjustment module includes a deviation calculation unit and a control parameter adjustment unit;

[0073] The deviation calculation unit calculates the deviation between the actual pose and the desired pose of the UAV robotic arm, including displacement and rotation errors; the control parameter adjustment unit calculates the adjusted displacement and rotation control parameters based on the set control gain.

[0074] A machine vision-based method for controlling a drone robotic arm, comprising the following steps:

[0075] Step S100. Obtain reference image data for UAV power line inspection operations, extract target object feature points from the reference image data, analyze the target object feature points for power line inspection operations, and thus construct a target object feature set.

[0076] Step S200. Acquire real-time image data of UAV power line inspection operation, extract real-time image feature points from real-time image data, perform matching analysis between real-time image feature points and target object feature set, filter out matching feature points, and mark target objects in real-time historical image data according to matching feature points.

[0077] Step S300. Based on the matching feature points obtained from the matching analysis, analyze the actual pose of the robotic arm in the UAV power inspection operation, and use the intrinsic and extrinsic parameters of the UAV camera to convert the three-dimensional coordinates of the target object into pose information in the desired grasping position coordinate system, thereby obtaining the desired pose.

[0078] Step S400. Calculate the deviation between the actual pose and the desired pose of the UAV robotic arm, and adjust the motion control parameters of the UAV robotic arm based on the deviation; adjust the current motion trajectory of the UAV robotic arm accordingly based on the adjusted motion control parameters.

[0079] Step S100 includes:

[0080] S101. Using the camera equipment of a drone, conduct power line inspections according to a preset flight path, collect multi-angle, multi-time image data of the target area, and use the collected image data as reference image data; preprocess the reference image data, and extract the preprocessed reference image data I... fStarting with (x,y), each layer of the pyramid is constructed; the image Gk(x,y) of each layer is obtained through Gaussian blur and downsampling, as follows:

[0081] For the first layer image G0(x,y)=I f Gaussian blur is applied to (x,y), and downsampling is used to obtain the next layer image: G k (x,y)=Downsample(G k-1 (2x, 2y)); For each layer k, the following Gaussian formula is used for calculation:

[0082] G k (x,y)=(1 / 2πσ k 2 )e^[-(x 2 +y 2 ) / (2σ k 2 ),

[0083] Where (x,y) represents the position coordinates of a pixel in the Gaussian image, σ k This represents the standard deviation of the k-th layer;

[0084] S102. Divide each Gaussian image into quadtrees, and define each grid region S as: S={(x,y)|0≤x<W,0≤y<H}, where W and H are the width and height of the Gaussian image, respectively; execute the FAST algorithm within each grid region, and use non-maximum suppression around each detected candidate feature point to retain the feature point with the highest response value as the target object feature point; for grids with a small number of detected feature points, lower the threshold and re-detect; calculate the image moments of all target object feature points, using the following formula:

[0085] m pq =∑ x∈[0,W1] ∑ y∈[0,H1] I(x,y)·x p ·y q ,

[0086] Where I(x,y) represents the pixel value of the current grid region at coordinates (x,y), W1 and H1 represent the width and height of the current grid region, respectively, and p and q are the orders of the moments; based on the image moments, the centroid C of all target object feature points is calculated, with the following formula:

[0087] C = (C x C y )=(m 10 / m 00 ,m 01 / m 00 ),

[0088] Where C x C represents the coordinate of the centroid in the x-direction. y This represents the coordinate of the centroid in the y-direction; m 10 represents the first moment, the centroid of the current mesh region in the x-direction; m 00 The zeroth moment of the current grid region is equal to the sum of all pixel values; m 01 This represents the first moment, and the centroid of the current mesh region in the y-direction; the orientation angle θ of the target object's feature points is calculated using the following formula:

[0089] θ = arctan(C) y / C x ) = arctan(m 01 / m 10 );

[0090] S103. Generate corresponding feature point descriptors based on the centroid position and orientation angle of each target object feature point, and summarize the feature points and feature point descriptors of the same target object to form the target object feature set Kd.

[0091] The FAST algorithm is executed within each grid region. Around each detected candidate feature point, a non-maximum suppression method is used to retain the feature point with the highest response value as the target object feature point. The specific analysis process is as follows:

[0092] For each detected candidate feature point I(x,y), a circular region with radius r is defined around it. Typically, r=3 is chosen, meaning 16 pixels around it are considered. For each candidate feature point I(x,y), the corresponding brightness value Ic is obtained. Within the circular region, the brightness values ​​Ii of the corresponding N neighboring pixels are obtained, where i represents the neighboring pixel number, which is a positive integer from 1 to N. If there are n neighboring pixels that satisfy: Ii>Ic+T, or Ii<Ic-T, where T is a set threshold, then the current candidate feature point I(x,y) is retained as a candidate corner point.

[0093] The edge responsivity R of each candidate corner point is calculated using the following formula:

[0094] R = det(M) - k·trace(M) 2 ,

[0095] Where M is the structure tensor of the image, det(M) represents the determinant of the structure tensor, trace(M) represents the trace of the structure tensor, and k represents an empirical constant. For each candidate corner point, non-maximum suppression is performed within a radius r around it. Specifically, within the circular region of the candidate corner point with a radius r, the response value R of that point is compared with the response values ​​of all other points in the neighborhood. Only when the response value R of that point is the largest in its neighborhood will it be retained.

[0096] Step S200 includes:

[0097] S201. Using the camera equipment of the drone, capture real-time image data during power line inspection operations, and store these images as an image sequence It, where It = {I_1, I_2, ..., I_h}, where I_1 represents the real-time image corresponding to the first time point, I_2 represents the real-time image corresponding to the second time point, and so on, I_h represents the real-time image corresponding to the h-th time point, and h represents the time index; for each real-time image in the image sequence It, analyze the target object feature points as described in step S100 to obtain the real-time image feature points and corresponding descriptors;

[0098] S202. Obtain the target object feature set Kd. For each target object in the target object feature set Kd, perform brute-force matching between the descriptor corresponding to each target object and the descriptor of each real-time image in the image sequence It. Calculate the Hamming distance Dh between the corresponding descriptors and filter out matching pairs that are less than the threshold Dh0. Use the RANSAC algorithm to remove mismatch points. Find the inlier set Nd based on the matching point pairs and count the number of inliers in the inlier set. If the number of inliers is greater than or equal to 4, it is considered a valid match. Thus, the matching feature points are obtained, and the target object is marked in each real-time image in the image sequence It based on the matching feature points.

[0099] In this embodiment, the RANSAC algorithm is used to remove mismatched points, and the set of interior points Nd is found based on the matching point pairs. The specific process is as follows:

[0100] Set an inlier threshold D1 and an iteration count N1. Initialize the optimal inlier set Nd0 and the maximum number of inliers m = 0. For each iteration from 1 to N1, randomly select 4 pairs of matching points as the base point set. Use the DLT method to calculate the homography matrix H based on the selected 4 points. Initialize the inlier set Nd. For each pair of matching points, apply the homography matrix H to the target point kp_t to obtain the estimated point k'_t. Calculate the distance D between the estimated point k'_t and the target point kp_t. If D < D1, add the point to the inlier set Nd. If the current number of inliers in the inlier set Nd is greater than m, update the optimal inlier set, and: Nd0 = Nd, m = |Nd|. Finally, output the optimal inlier set Nd0.

[0101] Step S300 includes:

[0102] S301. For each real-time image in the image sequence It, select any matching feature point of the marked target object in the first real-time image of the image sequence It as the center to construct an initial two-dimensional image coordinate system; map each real-time image in the image sequence It sequentially according to the initial two-dimensional image coordinate system, so that each real-time image in the image sequence It is mapped to the initial two-dimensional image coordinate system; the specific mapping process is as follows:

[0103] From the first real-time image I_1, select a matching feature point of the marked target object as the center, denoted as C = (x_c, y_c). Establish an initial two-dimensional coordinate system (X', Y') with C as the center. The coordinate transformation relationship is as follows: X' = x - x_c, Y' = y - y_c, thus ensuring that the position of C in the initial two-dimensional coordinate system is (0, 0). Based on the initial two-dimensional coordinate system, perform coordinate transformation on all matching feature points in real-time image I_1 to obtain the position coordinates of all matching feature points in real-time image I_1 in the initial two-dimensional coordinate system. For all matching feature points in the second real-time image I_2... For feature points, any matching feature point existing in both real-time image I_1 and real-time image I_2 is selected as a reference point. Based on the mapping relationship between the position coordinates of the reference point in real-time image I_2 and its position coordinates in real-time image I_1, the other matching feature points in real-time image I_2 are transformed and calculated accordingly to obtain the position coordinates of all matching feature points in real-time image I_2 in the initial two-dimensional coordinate system. Similarly, the above transformation and calculation are performed on all real-time images in image sequence It to obtain the position coordinates of the matching feature points of all real-time images in image sequence It in the initial two-dimensional coordinate system.

[0104] S302. Obtain the position coordinates of all matching feature points in the initial two-dimensional coordinate system, denoted as: p_g=(x_g,y_g), where g is the index of the matching feature point in the initial two-dimensional coordinate system; convert the two-dimensional coordinates of each matching feature point into homogeneous coordinates p_g'=(x_g,y_g,1). T Define the position coordinates of the matching feature points corresponding to the target object in the three-dimensional world coordinate system as G_g=(X_g,Y_g,Z_g). T Using the intrinsic parameter matrix K and extrinsic parameter matrix [Rr|tr] of the UAV camera device, where Rr represents the rotation matrix and tr represents the translation vector, it is expressed as: p_g=K[Rr|tr]G_g; using the equations of all matching feature points, an error function is established:

[0105] E(Rr,tr)=∑g∈[1,M],||p_g'-K[Rr|tr]G_g|| 2 ,

[0106] Where M represents the total number of matching feature points; the optimized rotation and translation parameters are obtained by minimizing E(Rr,tr), thereby obtaining the actual pose T_actual of the robotic arm;

[0107] S303. Let the origin of the desired capture position coordinate system be O'. Its relationship with the target object's 3D world coordinate system is determined by the rotation matrix R_target and the translation vector t_target, specifically: G_g' = R_targetG_g + t_target, where G_g' represents the target object's coordinates in the desired capture position coordinate system. For each matching feature point G_g, using the optimized rotation matrix Rr and translation vector tr, the target object's 3D coordinates G_g are converted into homogeneous coordinates p_g' on the image plane through the camera model. P_g' is then mapped back to the 3D world coordinate system to obtain the target object's position in the world coordinate system: G_g = K -1 p_g'; Using the relationship between the defined desired grasping position coordinate system and the target object coordinate system, G_g is converted to G_g'; Through the above conversion, the three-dimensional coordinates G_g' of the target object in the desired grasping position coordinate system are obtained, thus obtaining the desired pose T_desired of the robotic arm.

[0108] Step S400 includes:

[0109] S401. Transform the desired pose T_desired of the drone robotic arm from the desired grasping position coordinate system to the three-dimensional world coordinate system corresponding to the actual pose, thereby obtaining the desired pose T_desired_actual in the three-dimensional world coordinate system, and T_desired_actual = (T_actual). -1 T_desired, where (T_actual) -1 This represents the inverse transformation of the actual pose; calculate the deviation F between the desired pose T_desired_actual and the actual pose T_actual, and F = ||Δt|| + R_e, where Δt represents the displacement deviation, ||Δt|| represents the magnitude of the displacement deviation, and R_e represents the rotation error measure;

[0110] S402. Let the control gains be Kp and Kr, which are used for adjusting displacement and rotation respectively. Calculate the adjusted displacement control parameter Δt_control = KpΔt and the adjusted rotation control parameter ΔR_control = KrR_e. Update the motion control parameters of the UAV robotic arm according to the calculated adjustment parameters Δt_control and ΔR_control. And update the motion trajectory of the UAV robotic arm in real time based on the adjusted displacement and rotation control parameters.

[0111] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0112] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A machine vision-based method for controlling a drone robotic arm, characterized in that: The method includes the following steps: Step S100. Obtain reference image data for UAV power line inspection operations, extract target object feature points from the reference image data, analyze the target object feature points for power line inspection operations, and thus construct a target object feature set. Step S100 includes: S101. Using the camera equipment of a drone, conduct power line inspections according to a preset flight path, collect multi-angle, multi-time image data of the target area, and use the collected image data as reference image data; preprocess the reference image data, and extract the preprocessed reference image data I... f Starting with (x,y), each layer of the pyramid is constructed; the image Gk(x,y) of each layer is obtained through Gaussian blur and downsampling, as follows: For the first layer image G0(x,y)=I f Gaussian blur is applied to (x,y), and downsampling is used to obtain the next layer image: G k (x,y)=Downsample(G k-1 (2x, 2y)); For each layer k, the following Gaussian formula is used for calculation: G k (x,y)=(1 / 2πσ k 2 )e^[-(x 2 +y 2 ) / (2σ k 2 ), Where (x,y) represents the position coordinates of a pixel in the Gaussian image, σ k This represents the standard deviation of the k-th layer; S102. Divide each Gaussian image into quadtrees, and define each grid region S as: S={(x,y)|0≤x<W,0≤y<H}, where W and H are the width and height of the Gaussian image, respectively; execute the FAST algorithm within each grid region, and around each detected candidate feature point, use non-maximum suppression to retain the feature point with the highest response value as the target object feature point; calculate the image moments of all target object feature points, using the following formula: m pq =∑ x∈[0,W1] ∑ y∈[0,H1] I(x,y)·x p ·y q , Where I(x,y) represents the pixel value of the current grid region at coordinates (x,y), W1 and H1 represent the width and height of the current grid region, respectively, and p and q are the orders of the moments; based on the image moments, the centroid C of all target object feature points is calculated, with the following formula: C=(C x ,C y )=(m 10 / m 00 ,m 01 / m 00 ), Where C x C represents the coordinate of the centroid in the x-direction. y This represents the coordinate of the centroid in the y-direction; m 10 represents the first moment, the centroid of the current mesh region in the x-direction; m 00 The zeroth moment of the current grid region is equal to the sum of all pixel values; m 01 This represents the first moment, and the centroid of the current mesh region in the y-direction; the orientation angle θ of the target object's feature points is calculated using the following formula: θ=arctan(C y / C x )=arctan(m 01 / m 10 ); S103. Generate corresponding feature point descriptors based on the centroid position and orientation angle of each target object feature point, and summarize the feature points and feature point descriptors of the same target object to form the target object feature set Kd; Step S200. Acquire real-time image data of UAV power line inspection operation, extract real-time image feature points from real-time image data, perform matching analysis between real-time image feature points and target object feature set, filter out matching feature points, and mark target objects in real-time historical image data according to matching feature points. Step S300. Based on the matching feature points obtained from the matching analysis, analyze the actual pose of the robotic arm in the UAV power inspection operation, and use the intrinsic and extrinsic parameters of the UAV camera to convert the three-dimensional coordinates of the target object into pose information in the desired grasping position coordinate system, thereby obtaining the desired pose. Step S400. Calculate the deviation between the actual pose and the desired pose of the UAV robotic arm, and adjust the motion control parameters of the UAV robotic arm based on the deviation; adjust the current motion trajectory of the UAV robotic arm accordingly based on the adjusted motion control parameters.

2. The machine vision-based unmanned aerial vehicle (UAV) robotic arm control method according to claim 1, characterized in that: The FAST algorithm is executed within each grid region. Around each detected candidate feature point, a non-maximum suppression method is used to retain the feature point with the highest response value as the target object feature point. The specific analysis process is as follows: For each detected candidate feature point I(x,y), a circular region with radius r is defined around it. For each candidate feature point I(x,y), the corresponding brightness value Ic is obtained. Within the circular region, the brightness values ​​Ii of the corresponding N neighboring pixels are obtained, where i represents the neighboring pixel number, which is a positive integer from 1 to N. If there are n neighboring pixels that satisfy: Ii > Ic + T, or Ii < Ic - T, where T is a set threshold, then the current candidate feature point I(x,y) is retained as a candidate corner point. The edge responsivity R of each candidate corner point is calculated using the following formula: R=det(M)-k·trace(M) 2 , Where M is the structure tensor of the image, det(M) represents the determinant of the structure tensor, trace(M) represents the trace of the structure tensor, and k represents an empirical constant. For each candidate corner point, non-maximum suppression is performed within a radius r around it. Specifically, within the circular region of radius r of the candidate corner point, the response value R of the candidate corner point is compared with the response values ​​of all other points in the neighborhood. Only when the response value R of the candidate corner point is the largest in its neighborhood will it be retained.

3. The machine vision-based unmanned aerial vehicle (UAV) robotic arm control method according to claim 1, characterized in that: Step S200 includes: S201. Using the camera equipment of the drone, capture real-time image data during power line inspection operations, and store these images as an image sequence It, where It = {I_1, I_2, ..., I_h}, where I_1 represents the real-time image corresponding to the first time point, I_2 represents the real-time image corresponding to the second time point, and so on, with I_h representing the real-time image corresponding to the h-th time point, and h representing the time index; for each real-time image in the image sequence It, analyze the target object feature points as described in step S100 to obtain the real-time image feature points and corresponding descriptors; S202. Obtain the target object feature set Kd. For each target object in the target object feature set Kd, perform brute-force matching between the descriptor corresponding to each target object and the descriptor of each real-time image in the image sequence It. Calculate the Hamming distance Dh between the corresponding descriptors and filter out matching pairs that are less than the threshold Dh0. Use the RANSAC algorithm to remove mismatch points. Find the inlier set Nd based on the matching point pairs and count the number of inliers in the inlier set. If the number of inliers is greater than or equal to 4, it is considered a valid match. Thus, the matching feature points are obtained, and the target object is marked in each real-time image in the image sequence It based on the matching feature points.

4. The machine vision-based unmanned aerial vehicle (UAV) robotic arm control method according to claim 3, characterized in that: Step S300 includes: S301. For each real-time image in the image sequence It, select any matching feature point of the marked target object in the first real-time image of the image sequence It as the center to construct an initial two-dimensional image coordinate system; map each real-time image in the image sequence It sequentially according to the initial two-dimensional image coordinate system, so that each real-time image in the image sequence It is mapped to the initial two-dimensional image coordinate system; the specific mapping process is as follows: From the first real-time image I_1, select a matching feature point of the marked target object as the center, denoted as C=(x_c,y_c). Establish an initial two-dimensional coordinate system (X',Y') with C as the center. The coordinate transformation relationship is as follows: X'=x-x_c, Y'=y-y_c, thus ensuring that the position of C in the initial two-dimensional coordinate system is (0,0). Based on the initial two-dimensional coordinate system, perform coordinate transformation on all matching feature points in real-time image I_1 to obtain the position coordinates of all matching feature points in real-time image I_1 in the initial two-dimensional coordinate system. For all matching feature points in the second real-time image I_2... For feature points, any matching feature point existing in both real-time image I_1 and real-time image I_2 is selected as a reference point. Based on the mapping relationship between the position coordinates of the reference point in real-time image I_2 and its position coordinates in real-time image I_1, the other matching feature points in real-time image I_2 are transformed and calculated accordingly to obtain the position coordinates of all matching feature points in real-time image I_2 in the initial two-dimensional coordinate system. Similarly, the above transformation and calculation are performed on all real-time images in image sequence It to obtain the position coordinates of the matching feature points of all real-time images in image sequence It in the initial two-dimensional coordinate system. S302. Obtain the position coordinates of all matching feature points in the initial two-dimensional coordinate system, denoted as: p_g=(x_g,y_g), where g is the index of the matching feature point in the initial two-dimensional coordinate system; convert the two-dimensional coordinates of each matching feature point into homogeneous coordinates p_g'=(x_g,y_g,1). T Define the position coordinates of the matching feature points corresponding to the target object in the three-dimensional world coordinate system as G_g=(X_g,Y_g,Z_g). T Using the intrinsic parameter matrix K and extrinsic parameter matrix [Rr|tr] of the UAV camera device, where Rr represents the rotation matrix and tr represents the translation vector, it is expressed as: p_g=K[Rr|tr]G_g; using the equations of all matching feature points, an error function is established: E(Rr,tr)=∑g∈[1,M],||p_g’-K[Rr|tr]G_g|| 2 , Where M represents the total number of matching feature points; the optimized rotation and translation parameters are obtained by minimizing E(Rr,tr), thereby obtaining the actual pose T_actual of the robotic arm; S303. Let the origin of the desired capture position coordinate system be O'. Its relationship with the target object's 3D world coordinate system is determined by the rotation matrix R_target and the translation vector t_target, specifically: G_g' = R_targetG_g + t_target, where G_g' represents the target object's coordinates in the desired capture position coordinate system. For each matching feature point G_g, using the optimized rotation matrix Rr and translation vector tr, the target object's 3D coordinates G_g are converted into homogeneous coordinates p_g' on the image plane through the camera model. Mapping p_g' back to the 3D world coordinate system yields the target object's position in the world coordinate system: G_g = K -1 p_g'; Using the relationship between the defined desired grasping position coordinate system and the target object coordinate system, G_g is converted to G_g'; Through the above conversion, the three-dimensional coordinates G_g' of the target object in the desired grasping position coordinate system are obtained, thus obtaining the desired pose T_desired of the robotic arm.

5. The machine vision-based unmanned aerial vehicle (UAV) robotic arm control method according to claim 4, characterized in that: Step S400 includes: S401. Transform the desired pose T_desired of the drone robotic arm from the desired grasping position coordinate system to the three-dimensional world coordinate system corresponding to the actual pose, thereby obtaining the desired pose T_desired_actual in the three-dimensional world coordinate system, and T_desired_actual = (T_actual). -1 T_desired, where (T_actual) -1 This represents the inverse transformation of the actual pose; calculate the deviation F between the desired pose T_desired_actual and the actual pose T_actual, and F = ||Δt|| + R_e, where Δt represents the displacement deviation, ||Δt|| represents the magnitude of the displacement deviation, and R_e represents the rotation error measure; S402. Let the control gains be Kp and Kr, which are used for adjusting displacement and rotation respectively. Calculate the adjusted displacement control parameter Δt_control=KpΔt and the adjusted rotation control parameter ΔR_control=KrR_e. Update the motion control parameters of the UAV robotic arm according to the calculated adjustment parameters Δt_control and ΔR_control. And update the motion trajectory of the UAV robotic arm in real time based on the adjusted displacement and rotation control parameters.

6. A machine vision-based unmanned aerial vehicle (UAV) robotic arm control system, applied to the machine vision-based UAV robotic arm control method according to any one of claims 1-5, characterized in that: The system includes: an image acquisition module, a feature point extraction and matching module, a pose analysis module, a control parameter calculation and adjustment module, and a motion control and execution module; The image acquisition module uses the camera equipment of the drone to perform power line inspections along a preset path, collecting image data of the target area; it processes the collected image data, generates reference image data, and records images from multiple angles and at multiple times. The feature point extraction and matching module extracts feature points of the target object from the reference image data and constructs a target object feature set. It then extracts feature points from the real-time image data and matches them with the target object feature set to filter out valid matching feature points. The pose analysis module analyzes the actual pose of the UAV robotic arm based on the matching feature points, converts the three-dimensional coordinates of the target object into pose information in the coordinate system of the desired grasping position, and calculates the actual pose and desired pose of the robotic arm through an optimization algorithm. The control parameter calculation and adjustment module calculates the deviation between the desired pose and the actual pose, analyzes the displacement and rotation errors, sets the control gain, calculates the adjusted displacement and rotation control parameters, and updates the motion control parameters of the UAV robotic arm. The motion control and execution module updates the motion trajectory of the drone's robotic arm in real time based on the adjusted motion control parameters.

7. A machine vision-based unmanned aerial vehicle (UAV) robotic arm control system according to claim 6, characterized in that: The image acquisition module includes an image acquisition unit and an image storage unit; The image acquisition unit is responsible for capturing image data during power inspection operations; the image storage unit stores the captured image data in a time sequence. The feature point extraction and matching module includes an image preprocessing unit, a feature point extraction unit, and a feature point matching unit; The image preprocessing unit preprocesses the acquired images; the feature point extraction unit uses the FAST algorithm to extract the target object feature points in the preprocessed images and calculates the descriptors of the target object feature points; the feature point matching unit matches the feature points in the real-time image with the feature points in the reference image, uses brute-force matching and the RANSAC algorithm to remove incorrect matches, and filters out valid matching feature points.

8. A machine vision-based unmanned aerial vehicle (UAV) robotic arm control system according to claim 6, characterized in that: The pose analysis module includes a coordinate system mapping unit, an actual pose calculation unit, and a desired pose calculation unit. The coordinate system mapping unit maps the matching feature points in the real-time image to the initial two-dimensional coordinate system to perform position transformation. The actual pose calculation unit uses the camera's intrinsic and extrinsic parameters to calculate the three-dimensional world coordinates of the matching feature points and establishes an error function to obtain the actual pose of the UAV robotic arm; the desired pose calculation unit calculates the desired pose of the robotic arm based on the three-dimensional coordinates of the target object and its relationship with the desired grasping position coordinate system.

9. A machine vision-based unmanned aerial vehicle (UAV) robotic arm control system according to claim 6, characterized in that: The control parameter calculation and adjustment module includes a deviation calculation unit and a control parameter adjustment unit; The deviation calculation unit calculates the deviation between the actual pose and the desired pose of the UAV robotic arm, including displacement and rotation errors; the control parameter adjustment unit calculates the adjusted displacement and rotation control parameters according to the set control gain.

Images