Unmanned aerial vehicle mechanical arm control system and method based on machine vision

Abstract

The invention discloses an unmanned aerial vehicle mechanical arm control system and method based on machine vision, and relates to the technical field of mechanical arm control. The system comprises an image acquisition module, a feature point extraction and matching module, a pose analysis module, a control parameter calculation and adjustment module and a motion control and execution module. The image data acquisition module acquires target area image data; the feature point extraction and matching module is used for extracting feature points of a target object from a reference image, constructing a feature set, extracting and matching the feature points of real-time image data, and screening effective matching points; the pose estimation module analyzes the actual pose of the unmanned aerial vehicle mechanical arm according to the matching points and calculates the actual pose and the expected pose through an optimization algorithm; the control parameter calculation and adjustment module is used for calculating the deviation between the expected pose and the actual pose, setting a control gain and updating a motion control parameter; and the motion control and execution module updates the motion track of the unmanned aerial vehicle mechanical arm in real time based on the adjusted control parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm control, and in particular to a robot arm control system and method based on machine vision. Background Art

[0002] With the rapid development of robot technology, power maintenance robotic arms have been gradually applied to power transmission line maintenance operations. While improving the efficiency of power transmission line maintenance work, it has reduced the cost of training professionals and improved the safety of operations. However, the motion trajectory of the robotic arm needs to be adjusted according to different operation tasks. Therefore, the rapid planning of the power maintenance robotic arm's movements has become a new development demand. Drones with robotic arms cooperate with maintenance personnel to control objects to fly. They are currently in the initial stage at home and abroad and can be used in a variety of power operation and maintenance scenarios. In the future, the application field will be very broad and have great transformation prospects.

[0003] However, in practical applications, the existing UAV robotic arm control system has the following shortcomings: the existing system has limited feature recognition capabilities for power facilities in complex environments (such as light changes, shadow interference, etc.), especially in the detection and positioning of small objects (such as insulators, bolts, etc.), which are prone to missed detection or false detection, which directly leads to a decrease in the grasping accuracy of the robotic arm and increases the operational risk; in addition, the existing system usually relies on static environmental models and has weak adaptability to dynamic environments. During power inspection operations, the environment may change continuously due to factors such as weather and equipment changes. The existing system often cannot quickly adjust the trajectory, resulting in reduced operational efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a UAV mechanical arm control system and method based on machine vision to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A method for controlling a UAV mechanical arm based on machine vision, the method comprising the following steps:

[0007] Step S100. Acquire reference image data of the UAV power inspection operation, extract target object feature points of the power inspection operation from the reference image data, analyze the target object feature points of the power inspection operation, and thus construct a target object feature set;

[0008] Step S200. Acquire real-time image data of the UAV power inspection operation, extract real-time image feature points from the real-time image data, match and analyze the real-time image feature points with the target object feature set, screen out matching feature points, and mark the target object in the real-time historical image data according to the matching feature points;

[0009] Step S300. Analyze the actual posture of the robot arm in the UAV power inspection operation according to the matching feature points obtained by the matching analysis, and use the internal and external parameters of the UAV camera to convert the three-dimensional coordinates of the target object into the posture information in the desired grasping position coordinate system, so as to obtain the desired posture;

[0010] Step S400. Calculate the deviation between the actual posture of the UAV robotic arm and the expected posture, and adjust the motion control parameters of the UAV robotic arm based on the deviation; according to the adjusted motion control parameters, adjust the motion trajectory of the current UAV robotic arm accordingly.

[0011] Furthermore, step S100 includes:

[0012] S101. Use the camera equipment of the drone to perform power inspection according to the preset flight path, collect image data of the target area from multiple angles and multiple times, and use the collected image data as reference image data; pre-process the reference image data, and f (x, y), construct each layer of the pyramid; the image Gk(x, y) of each layer is obtained by Gaussian blurring and downsampling. The specific process is as follows:

[0013] For the first layer image G0(x,y)=I f (x, y) is Gaussian blurred and downsampled to get the next layer of image: G k (x,y)=Downsample(G k-1 (2x,2y)); for each layer k, the following Gaussian formula is used for calculation:

[0014] G k (x,y)=(1 / 2πσ k 2 )e^[-(x 2 +y 2 ) / (2σ k 2 ),

[0015] Where (x, y) represents the position coordinates of a pixel in the Gaussian image, σ k represents the standard deviation of the kth layer;

[0016] S102. Perform quadtree division on each layer of Gaussian image, and define each grid area S as: S = {(x, y) | 0 ≤ x < W, 0 ≤ y < H}, where W and H are the width and height of the Gaussian image, respectively; execute the FAST algorithm in each grid area, and use the non-maximum suppression method around each detected candidate feature point to retain the feature point with the highest response value as the target object feature point; for grids with a small number of feature point detections, lower the threshold and perform re-detection; calculate the image moment of all target object feature points, and the specific formula is as follows:

[0017] m pq =∑ x∈[0,W1] ∑ y∈[0,H1] I(x,y)·x p ·y q ,

[0018] Where I(x,y) represents the pixel value of the current grid area at the coordinate (x,y), W1 and H1 represent the width and height of the current grid area respectively, and p and q are the orders of the moment. According to the image moment, the centroid point C of all the feature points of the target object is calculated. The specific formula is:

[0019] C=(C x ,C y )=(m 10 / m 00 ,m 01 / m 00 ),

[0020] Among them C x represents the coordinate of the center of mass in the x direction, C y Indicates the coordinate of the center of mass in the y direction; m 10 Represents the first-order moment, the center of gravity of the current grid area in the x direction; m 00 Represents the zero-order moment of the current grid area, which is equal to the sum of all pixel values; m 01 Represents the first-order moment, the center of gravity of the current grid area in the y direction; calculates the direction angle θ of the feature point of the target object. The specific calculation formula is:

[0021] θ=arctan(C y / C x )=arctan(m 01 / m 10 );

[0022] S103. Generate a corresponding feature point descriptor according to the centroid position and direction angle of each target object feature point, and aggregate the feature points and feature point descriptors of the same target object to form a target object feature set Kd.

[0023] Furthermore, the FAST algorithm is executed in each grid area, and the non-maximum suppression method is used around each detected candidate feature point to retain the feature point with the highest response value as the target object feature point; the specific analysis process is as follows:

[0024] For each detected candidate feature point I(x, y), define a circular area with a radius of r around it. Usually, r=3 is selected, that is, 16 pixels around it are considered; for each candidate feature point I(x, y), obtain the corresponding brightness value Ic, and in the circular area, obtain the brightness values ​​Ii of the corresponding N neighboring pixels, where i represents the neighboring pixel number, which is a positive integer from 1 to N; if there are n neighboring pixels that satisfy: Ii>Ic+T, or Ii<Ic-T; where T is the set threshold, the current candidate feature point I(x, y) is retained as a candidate corner point;

[0025] Calculate the edge response R of each candidate corner point. The specific calculation formula is:

[0026] R = det(M) - k·trace(M) 2 ,

[0027] Where M is the structure tensor of the image, det(M) represents the determinant of the structure tensor, trace(M) represents the trace of the structure tensor, and k represents an empirical constant. For each candidate corner point, non-maximum suppression is performed within a radius of r around it. The specific process is: within the circular area of ​​radius r of the candidate corner point, the response value R of the point is compared with the response values ​​of all other points in the neighborhood. Only when the response value R of the point is the largest in its neighborhood will it be retained.

[0028] Further, step S200 includes:

[0029] S201. Capture real-time image data in the power inspection operation through the camera equipment of the drone, and store these images as an image sequence It, and It = {I_1, I_2, ..., I_h}, where I_1 represents the real-time image corresponding to the first time point, I_2 represents the real-time image corresponding to the second time point, and so on, I_h represents the real-time image corresponding to the hth time point, and h represents the time index; for each real-time image in the image sequence It, analyze the feature points of the target object in step S100, so as to obtain the real-time image feature points and corresponding descriptors;

[0030] S202. Obtain the target object feature set Kd, perform brute force matching on the descriptors corresponding to each target object in the target object feature set Kd and the descriptors of each real-time image in the image sequence It in turn, calculate the Hamming distance Dh between the descriptors corresponding to the two, and screen out matching pairs that are less than the threshold Dh0; use the RANSAC algorithm to remove mismatched points, find the inlier set Nd based on the matching point pairs, and count the number of inliers in the inlier set. If the number of inliers is greater than or equal to 4, it is considered a valid match, thereby obtaining matching feature points, and marking the target object in each real-time image in the image sequence It based on the matching feature points.

[0031] Furthermore, step S300 includes:

[0032] S301. For each real-time image in the image sequence It, a matching feature point of a marked target object in the first real-time image in the image sequence It is selected as the center of a circle to construct an initial two-dimensional image coordinate system; each real-time image in the image sequence It is sequentially mapped according to the initial two-dimensional image coordinate system, so that each real-time image in the image sequence It is mapped to the initial two-dimensional image coordinate system; the specific mapping process is as follows:

[0033] Select a matching feature point of a marked target object from the first real-time image I_1 as the center of the circle, denoted as C = (x_c, y_c), and establish an initial two-dimensional coordinate system (X', Y') with C as the center of the circle. The coordinate transformation relationship is as follows: X' = x-x_c, Y' = y-y_c, so as to ensure that the position of C in the initial two-dimensional coordinate system is (0,0); according to the initial two-dimensional coordinate system, coordinate transformation is performed on all matching feature points in the real-time image I_1, so as to obtain the position coordinates of all matching feature points in the real-time image I_1 in the initial two-dimensional coordinate system; for all matching feature points in the second real-time image I_2, Feature point, select any matching feature point that exists in both the real-time image I_1 and the real-time image I_2 as a reference point, and perform corresponding conversion calculations on other matching feature points in the real-time image I_2 according to the mapping relationship between the position coordinates of the reference point in the real-time image I_2 and the position coordinates in the real-time image I_1, so as to obtain the position coordinates of all matching feature points in the real-time image I_2 in the initial two-dimensional coordinate system, and so on, perform the above conversion calculations on all real-time images in the image sequence It, so as to obtain the position coordinates of the matching feature points of all real-time images in the image sequence It in the initial two-dimensional coordinate system;

[0034] S302. Obtain the position coordinates of the matching feature points of all real-time images in the initial two-dimensional coordinate system, recorded as: p_g = (x_g, y_g), where g is the index of the matching feature point in the initial two-dimensional coordinate system; convert the two-dimensional coordinates of each matching feature point into homogeneous coordinates p_g' = (x_g, y_g, 1) T ; Define the position coordinates of the matching feature points corresponding to the target object in the three-dimensional world coordinate system as G_g = (X_g, Y_g, Z_g) T , using the intrinsic parameter matrix K and extrinsic parameter matrix [Rr|tr] of the drone camera device, where Rr represents the rotation matrix and tr represents the translation vector; expressed as: p_g=K[Rr|tr]G_g; using the equations of all matching feature points, establish the error function:

[0035] E(Rr,tr)=∑g∈[1,M],||p_g'-K[Rr|tr]G_g|| 2 ,

[0036] Where M represents the total number of matching feature points; the optimized rotation and translation parameters are obtained by minimizing E(Rr, tr), thereby obtaining the actual position T_actual of the robot arm;

[0037] S303. The origin of the desired grasping position coordinate system is recorded as O', and the relationship with the three-dimensional world coordinate system of the target object is determined by the rotation matrix R_target and the translation vector t_target. The specific relationship is: G_g'=R_targetG_g+t_target, G_g' represents the coordinates of the target object in the desired grasping position coordinate system; for each matching feature point G_g, the three-dimensional coordinates G_g of the target object are converted into homogeneous coordinates p_g' on the image plane through the camera model using the optimized rotation matrix Rr and translation vector tr; p_g' is mapped back to the three-dimensional world coordinate system to obtain the position of the target object in the world coordinate system: G_g=K -1 p_g'; using the defined relationship between the desired grasping position coordinate system and the target object coordinate system, G_g is converted into G_g'; through the above conversion, the three-dimensional coordinates G_g' of the target object in the desired grasping position coordinate system are obtained, thereby obtaining the desired position and posture T_desired of the robot arm.

[0038] Furthermore, step S400 includes:

[0039] S401. Convert the desired posture T_desired of the UAV manipulator from the desired grasping position coordinate system to the three-dimensional world coordinate system corresponding to the actual posture, thereby obtaining the desired posture T_desired_actual of the three-dimensional world coordinate system, and T_desired_actual = (T_actual)-1 T_desired, where (T_actual) -1 represents the inverse transformation of the actual posture; calculates the deviation value F between the desired posture T_desired_actual and the actual posture T_actual, and F = ||Δt||+R_e, where Δt represents the displacement deviation, ||Δt|| represents the modulus of the displacement deviation, and R_e represents the rotation error metric;

[0040] S402. Assume that the control gains are Kp and Kr, which are used for adjusting the displacement and rotation respectively, calculate the adjusted displacement control parameter Δt_control = KpΔt, and calculate the adjusted rotation control parameter ΔR_control = KrR_e; update the motion control parameters of the UAV robotic arm according to the calculated adjustment parameters Δt_control and ΔR_control; and update the motion trajectory of the UAV robotic arm in real time based on the adjusted displacement and rotation control parameters.

[0041] A UAV mechanical arm control system based on machine vision, the system includes: an image acquisition module, a feature point extraction and matching module, a posture analysis module, a control parameter calculation and adjustment module, and a motion control and execution module;

[0042] The image acquisition module uses the camera equipment of the drone to conduct power inspections along a preset path and collect image data of the target area; processes the collected image data, generates reference image data, and records images from multiple angles and at multiple times;

[0043] The feature point extraction and matching module extracts the feature points of the target object from the reference image data and constructs the target object feature set, extracts the feature points of the real-time image data, matches it with the target object feature set, and selects the effective matching feature points;

[0044] The posture analysis module analyzes the actual posture of the UAV robotic arm based on the matching feature points, converts the three-dimensional coordinates of the target object into the posture information in the coordinate system of the desired grasping position, and calculates the actual posture and expected posture of the robotic arm through the optimization algorithm;

[0045] The control parameter calculation and adjustment module calculates the deviation between the desired posture and the actual posture, analyzes the displacement and rotation errors, sets the control gain, calculates the adjusted displacement and rotation control parameters, and updates the motion control parameters of the UAV robotic arm;

[0046] The motion control and execution module updates the motion trajectory of the UAV robotic arm in real time based on the adjusted motion control parameters.

[0047] Further, the image acquisition module includes an image acquisition unit and an image storage unit;

[0048] The image acquisition unit is responsible for capturing image data during power inspection operations; the image storage unit stores the captured image data in time series;

[0049] The feature point extraction and matching module includes an image preprocessing unit, a feature point extraction unit and a feature point matching unit;

[0050] The image preprocessing unit preprocesses the collected images; the feature point extraction unit uses the FAST algorithm and other technologies to extract the feature points of the target object in the preprocessed image, and calculates the descriptor of the feature points of the target object; the feature point matching unit matches the feature points in the real-time image with the feature points in the reference image, uses brute force matching and RANSAC algorithm to remove incorrect matches, and screens out valid matching feature points.

[0051] Further, the posture analysis module includes a coordinate system mapping unit, an actual posture calculation unit and a desired posture calculation unit;

[0052] The coordinate system mapping unit maps the matching feature points in the real-time image to the initial two-dimensional coordinate system for position conversion; the actual posture calculation unit calculates the three-dimensional world coordinates of the matching feature points using the internal and external parameters of the camera, and establishes an error function to obtain the actual posture of the UAV robotic arm; the expected posture calculation unit calculates the expected posture of the robotic arm based on the three-dimensional coordinates of the target object and its relationship in the coordinate system of the expected grasping position.

[0053] Further, the control parameter calculation and adjustment module includes a deviation calculation unit and a control parameter adjustment unit;

[0054] The deviation calculation unit calculates the deviation between the actual posture of the UAV mechanical arm and the expected posture, including displacement and rotation errors; the control parameter adjustment unit calculates the adjusted displacement and rotation control parameters according to the set control gain.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows: by constructing a target object feature set and performing real-time image feature point matching, the present invention effectively improves the recognition accuracy of small objects (such as insulators, bolts, etc.), significantly reduces the occurrence of missed detection and false detection, and this improvement reduces the operational risk and improves the accuracy of robotic arm grasping; unlike traditional systems that rely on static environmental models, the present invention can analyze and match target object feature points in real time and adapt to complex and dynamically changing environments, such as weather changes or equipment displacement. This flexibility allows drones to adjust their trajectories more quickly during inspection operations, thereby improving operational efficiency; the present invention acquires image data in real time, combines the internal and external parameters of the camera, and locates the target object The three-dimensional coordinates are converted into the posture information in the coordinate system of the desired grasping position. This dynamic calculation capability enables the robot arm to respond quickly and adjust the motion control parameters, further improving the grasping accuracy and operation safety. The present invention adopts a multi-angle and multi-time image data acquisition method, and ensures the feature point detection effect under different environmental conditions through preprocessing means such as Gaussian blur and downsampling. This method can improve the system's adaptability and robustness to complex backgrounds. By calculating the deviation between the desired posture and the actual posture, and using the control gain to adjust the displacement and rotation in real time, the present invention effectively improves the motion control accuracy of the robot arm. This mechanism not only enhances the stability of the system, but also improves the operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0057] Figure 1 It is a schematic diagram of a UAV mechanical arm control system module based on machine vision of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figure 1 , the present invention provides a technical solution:

[0060] A UAV mechanical arm control system based on machine vision, the system includes: an image acquisition module, a feature point extraction and matching module, a posture analysis module, a control parameter calculation and adjustment module, and a motion control and execution module;

[0061] The image acquisition module uses the camera equipment of the drone to conduct power inspections along a preset path and collect image data of the target area; processes the collected image data, generates reference image data, and records images from multiple angles and at multiple times;

[0062] The feature point extraction and matching module extracts the feature points of the target object from the reference image data and constructs the target object feature set, extracts the feature points of the real-time image data, matches it with the target object feature set, and selects the effective matching feature points;

[0063] The posture analysis module analyzes the actual posture of the UAV robotic arm based on the matching feature points, converts the three-dimensional coordinates of the target object into the posture information in the coordinate system of the desired grasping position, and calculates the actual posture and expected posture of the robotic arm through the optimization algorithm;

[0064] The control parameter calculation and adjustment module calculates the deviation between the desired posture and the actual posture, analyzes the displacement and rotation errors, sets the control gain, calculates the adjusted displacement and rotation control parameters, and updates the motion control parameters of the UAV robotic arm;

[0065] The motion control and execution module updates the motion trajectory of the UAV robotic arm in real time based on the adjusted motion control parameters.

[0066] The image acquisition module includes an image acquisition unit and an image storage unit;

[0067] The image acquisition unit is responsible for capturing image data during power inspection operations; the image storage unit stores the captured image data in time series;

[0068] The feature point extraction and matching module includes an image preprocessing unit, a feature point extraction unit and a feature point matching unit;

[0069] The image preprocessing unit preprocesses the collected images; the feature point extraction unit uses the FAST algorithm and other technologies to extract the feature points of the target object in the preprocessed image, and calculates the descriptor of the feature points of the target object; the feature point matching unit matches the feature points in the real-time image with the feature points in the reference image, uses brute force matching and RANSAC algorithm to remove incorrect matches, and screens out valid matching feature points.

[0070] The posture analysis module includes a coordinate system mapping unit, an actual posture calculation unit and an expected posture calculation unit;

[0071] The coordinate system mapping unit maps the matching feature points in the real-time image to the initial two-dimensional coordinate system for position conversion; the actual posture calculation unit calculates the three-dimensional world coordinates of the matching feature points using the internal and external parameters of the camera, and establishes an error function to obtain the actual posture of the UAV robotic arm; the expected posture calculation unit calculates the expected posture of the robotic arm based on the three-dimensional coordinates of the target object and its relationship in the coordinate system of the expected grasping position.

[0072] The control parameter calculation and adjustment module includes a deviation calculation unit and a control parameter adjustment unit;

[0073] The deviation calculation unit calculates the deviation between the actual posture of the UAV mechanical arm and the expected posture, including displacement and rotation errors; the control parameter adjustment unit calculates the adjusted displacement and rotation control parameters according to the set control gain.

[0074] A method for controlling a UAV mechanical arm based on machine vision, the method comprising the following steps:

[0075] Step S100. Acquire reference image data of the UAV power inspection operation, extract target object feature points of the power inspection operation from the reference image data, analyze the target object feature points of the power inspection operation, and thus construct a target object feature set;

[0076] Step S200. Acquire real-time image data of the UAV power inspection operation, extract real-time image feature points from the real-time image data, match and analyze the real-time image feature points with the target object feature set, screen out matching feature points, and mark the target object in the real-time historical image data according to the matching feature points;

[0077] Step S300. Analyze the actual posture of the robot arm in the UAV power inspection operation according to the matching feature points obtained by the matching analysis, and use the internal and external parameters of the UAV camera to convert the three-dimensional coordinates of the target object into the posture information in the desired grasping position coordinate system, so as to obtain the desired posture;

[0078] Step S400. Calculate the deviation between the actual posture of the UAV robotic arm and the expected posture, and adjust the motion control parameters of the UAV robotic arm based on the deviation; according to the adjusted motion control parameters, adjust the motion trajectory of the current UAV robotic arm accordingly.

[0079] Step S100 includes:

[0080] S101. Use the camera equipment of the drone to perform power inspection according to the preset flight path, collect image data of the target area from multiple angles and multiple times, and use the collected image data as reference image data; pre-process the reference image data, and f(x, y), construct each layer of the pyramid; the image Gk(x, y) of each layer is obtained by Gaussian blurring and downsampling. The specific process is as follows:

[0081] For the first layer image G0(x,y)=I f (x, y) is Gaussian blurred and downsampled to get the next layer of image: G k (x,y)=Downsample(G k-1 (2x,2y)); for each layer k, the following Gaussian formula is used for calculation:

[0082] G k (x,y)=(1 / 2πσ k 2 )e^[-(x 2 +y 2 ) / (2σ k 2 ),

[0083] Where (x, y) represents the position coordinates of a pixel in the Gaussian image, σ k represents the standard deviation of the kth layer;

[0084] S102. Perform quadtree division on each layer of Gaussian image, and define each grid area S as: S = {(x, y) | 0 ≤ x < W, 0 ≤ y < H}, where W and H are the width and height of the Gaussian image, respectively; execute the FAST algorithm in each grid area, and use the non-maximum suppression method around each detected candidate feature point to retain the feature point with the highest response value as the target object feature point; for grids with a small number of feature point detections, lower the threshold and perform re-detection; calculate the image moment of all target object feature points, and the specific formula is as follows:

[0085] m pq =∑ x∈[0,W1] ∑ y∈[0,H1] I(x,y)·x p ·y q ,

[0086] Where I(x,y) represents the pixel value of the current grid area at the coordinate (x,y), W1 and H1 represent the width and height of the current grid area respectively, and p and q are the orders of the moment. According to the image moment, the centroid point C of all the feature points of the target object is calculated. The specific formula is:

[0087] C=(C x ,C y )=(m 10 / m 00 ,m 01 / m 00 ),

[0088] Among them C x represents the coordinate of the center of mass in the x direction, C y Indicates the coordinate of the center of mass in the y direction; m 10 Represents the first-order moment, the center of gravity of the current grid area in the x direction; m 00 Represents the zero-order moment of the current grid area, which is equal to the sum of all pixel values; m 01 Represents the first-order moment, the center of gravity of the current grid area in the y direction; calculates the direction angle θ of the feature point of the target object. The specific calculation formula is:

[0089] θ=arctan(C y / C x )=arctan(m 01 / m 10 );

[0090] S103. Generate a corresponding feature point descriptor according to the centroid position and direction angle of each target object feature point, and aggregate the feature points and feature point descriptors of the same target object to form a target object feature set Kd.

[0091] The FAST algorithm is executed in each grid area. The non-maximum suppression method is used around each detected candidate feature point to retain the feature point with the highest response value as the target object feature point. The specific analysis process is as follows:

[0092] For each detected candidate feature point I(x, y), define a circular area with a radius of r around it. Usually, r=3 is selected, that is, 16 pixels around it are considered; for each candidate feature point I(x, y), obtain the corresponding brightness value Ic, and in the circular area, obtain the brightness values ​​Ii of the corresponding N neighboring pixels, where i represents the neighboring pixel number, which is a positive integer from 1 to N; if there are n neighboring pixels that satisfy: Ii>Ic+T, or Ii<Ic-T; where T is the set threshold, the current candidate feature point I(x, y) is retained as a candidate corner point;

[0093] Calculate the edge response R of each candidate corner point. The specific calculation formula is:

[0094] R = det(M) - k·trace(M) 2 ,

[0095] Where M is the structure tensor of the image, det(M) represents the determinant of the structure tensor, trace(M) represents the trace of the structure tensor, and k represents an empirical constant. For each candidate corner point, non-maximum suppression is performed within a radius of r around it. The specific process is: within the circular area of ​​radius r of the candidate corner point, the response value R of the point is compared with the response values ​​of all other points in the neighborhood. Only when the response value R of the point is the largest in its neighborhood will it be retained.

[0096] Step S200 includes:

[0097] S201. Capture real-time image data in the power inspection operation through the camera equipment of the drone, and store these images as an image sequence It, and It = {I_1, I_2, ..., I_h}, where I_1 represents the real-time image corresponding to the first time point, I_2 represents the real-time image corresponding to the second time point, and so on, I_h represents the real-time image corresponding to the hth time point, and h represents the time index; for each real-time image in the image sequence It, analyze the feature points of the target object in step S100, so as to obtain the real-time image feature points and corresponding descriptors;

[0098] S202. Obtain the target object feature set Kd, perform brute force matching on the descriptors corresponding to each target object in the target object feature set Kd and the descriptors of each real-time image in the image sequence It in turn, calculate the Hamming distance Dh between the descriptors corresponding to the two, and screen out matching pairs that are less than the threshold Dh0; use the RANSAC algorithm to remove mismatched points, find the inlier set Nd based on the matching point pairs, and count the number of inliers in the inlier set. If the number of inliers is greater than or equal to 4, it is considered a valid match, thereby obtaining matching feature points, and marking the target object in each real-time image in the image sequence It based on the matching feature points.

[0099] In this embodiment, the RANSAC algorithm is used to remove mismatched points, and the inlier point set Nd is found based on the matching point pairs. The specific process is as follows:

[0100] Set the inlier threshold D1 and the number of iterations N1, initialize the optimal inlier set Nd0 and the maximum number of inliers m=0; for each iteration from 1 to N1, randomly select 4 pairs of matching points as the basic point set; use the DLT method to calculate the homography matrix H based on the selected 4 points; initialize the inlier set Nd, for each pair of matching points, transform the target point kp_t by the homography matrix H to obtain the estimated point k'_t, calculate the distance D between the estimated point k'_t and the target point kp_t, if D<D1, add the point to the inlier set Nd; if the number of current inlier set Nd is greater than m, update the optimal inlier set, and: Nd0=Nd, m=|Nd|, and finally output the optimal inlier set Nd0.

[0101] Step S300 includes:

[0102] S301. For each real-time image in the image sequence It, a matching feature point of a marked target object in the first real-time image in the image sequence It is selected as the center of a circle to construct an initial two-dimensional image coordinate system; each real-time image in the image sequence It is sequentially mapped according to the initial two-dimensional image coordinate system, so that each real-time image in the image sequence It is mapped to the initial two-dimensional image coordinate system; the specific mapping process is as follows:

[0103] Select a matching feature point of a marked target object from the first real-time image I_1 as the center of the circle, denoted as C = (x_c, y_c), and establish an initial two-dimensional coordinate system (X', Y') with C as the center of the circle. The coordinate transformation relationship is as follows: X' = x-x_c, Y' = y-y_c, so as to ensure that the position of C in the initial two-dimensional coordinate system is (0,0); according to the initial two-dimensional coordinate system, coordinate transformation is performed on all matching feature points in the real-time image I_1, so as to obtain the position coordinates of all matching feature points in the real-time image I_1 in the initial two-dimensional coordinate system; for all matching feature points in the second real-time image I_2, Feature point, select any matching feature point that exists in both the real-time image I_1 and the real-time image I_2 as a reference point, and perform corresponding conversion calculations on other matching feature points in the real-time image I_2 according to the mapping relationship between the position coordinates of the reference point in the real-time image I_2 and the position coordinates in the real-time image I_1, so as to obtain the position coordinates of all matching feature points in the real-time image I_2 in the initial two-dimensional coordinate system, and so on, perform the above conversion calculations on all real-time images in the image sequence It, so as to obtain the position coordinates of the matching feature points of all real-time images in the image sequence It in the initial two-dimensional coordinate system;

[0104] S302. Obtain the position coordinates of the matching feature points of all real-time images in the initial two-dimensional coordinate system, recorded as: p_g = (x_g, y_g), where g is the index of the matching feature point in the initial two-dimensional coordinate system; convert the two-dimensional coordinates of each matching feature point into homogeneous coordinates p_g' = (x_g, y_g, 1) T ; Define the position coordinates of the matching feature points corresponding to the target object in the three-dimensional world coordinate system as G_g = (X_g, Y_g, Z_g) T , using the intrinsic parameter matrix K and extrinsic parameter matrix [Rr|tr] of the drone camera device, where Rr represents the rotation matrix and tr represents the translation vector; expressed as: p_g=K[Rr|tr]G_g; using the equations of all matching feature points, establish the error function:

[0105] E(Rr,tr)=∑g∈[1,M],||p_g'-K[Rr|tr]G_g|| 2 ,

[0106] Where M represents the total number of matching feature points; the optimized rotation and translation parameters are obtained by minimizing E(Rr, tr), thereby obtaining the actual position T_actual of the robot arm;

[0107] S303. The origin of the desired grasping position coordinate system is recorded as O', and the relationship with the three-dimensional world coordinate system of the target object is determined by the rotation matrix R_target and the translation vector t_target. The specific relationship is: G_g'=R_targetG_g+t_target, G_g' represents the coordinates of the target object in the desired grasping position coordinate system; for each matching feature point G_g, the three-dimensional coordinates G_g of the target object are converted into homogeneous coordinates p_g' on the image plane through the camera model using the optimized rotation matrix Rr and translation vector tr; p_g' is mapped back to the three-dimensional world coordinate system to obtain the position of the target object in the world coordinate system: G_g=K -1 p_g'; using the defined relationship between the desired grasping position coordinate system and the target object coordinate system, G_g is converted into G_g'; through the above conversion, the three-dimensional coordinates G_g' of the target object in the desired grasping position coordinate system are obtained, thereby obtaining the desired position and posture T_desired of the robot arm.

[0108] Step S400 includes:

[0109] S401. Convert the desired posture T_desired of the UAV manipulator from the desired grasping position coordinate system to the three-dimensional world coordinate system corresponding to the actual posture, thereby obtaining the desired posture T_desired_actual of the three-dimensional world coordinate system, and T_desired_actual = (T_actual) -1 T_desired, where (T_actual) -1 represents the inverse transformation of the actual posture; calculates the deviation value F between the desired posture T_desired_actual and the actual posture T_actual, and F = ||Δt||+R_e, where Δt represents the displacement deviation, ||Δt|| represents the modulus of the displacement deviation, and R_e represents the rotation error metric;

[0110] S402. Assume that the control gains are Kp and Kr, which are used for adjusting the displacement and rotation respectively, calculate the adjusted displacement control parameter Δt_control = KpΔt, and calculate the adjusted rotation control parameter ΔR_control = KrR_e; update the motion control parameters of the UAV robotic arm according to the calculated adjustment parameters Δt_control and ΔR_control; and update the motion trajectory of the UAV robotic arm in real time based on the adjusted displacement and rotation control parameters.

[0111] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0112] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for controlling a UAV mechanical arm based on machine vision, characterized in that: The method comprises the following steps: Step S100. Acquire reference image data of the UAV power inspection operation, extract target object feature points of the power inspection operation from the reference image data, analyze the target object feature points of the power inspection operation, and thus construct a target object feature set; Step S200. Acquire real-time image data of the UAV power inspection operation, extract real-time image feature points from the real-time image data, match and analyze the real-time image feature points with the target object feature set, screen out matching feature points, and mark the target object in the real-time historical image data according to the matching feature points; Step S300. Analyze the actual posture of the robot arm in the UAV power inspection operation according to the matching feature points obtained by the matching analysis, and use the internal and external parameters of the UAV camera to convert the three-dimensional coordinates of the target object into the posture information in the desired grasping position coordinate system, so as to obtain the desired posture; Step S400. Calculate the deviation between the actual posture of the UAV robotic arm and the expected posture, and adjust the motion control parameters of the UAV robotic arm based on the deviation; according to the adjusted motion control parameters, adjust the motion trajectory of the current UAV robotic arm accordingly.

2. The method for controlling a UAV mechanical arm based on machine vision according to claim 1, characterized in that: The step S100 includes: S101. Use the camera equipment of the drone to perform power inspection according to the preset flight path, collect image data of the target area from multiple angles and multiple times, and use the collected image data as reference image data; pre-process the reference image data, and f (x, y), construct each layer of the pyramid; the image Gk(x, y) of each layer is obtained by Gaussian blurring and downsampling. The specific process is as follows: For the first layer image G0(x,y)=I f (x, y) is Gaussian blurred and downsampled to get the next layer of image: G k (x,y)=Downsample(G k-1 (2x,2y)); for each layer k, the following Gaussian formula is used for calculation: G k (x,y)=(1 / 2πσ k 2 )e^[-(x 2 +y 2 ) / (2σ k 2 ), Where (x, y) represents the position coordinates of a pixel in the Gaussian image, σ k represents the standard deviation of the kth layer; S102. Perform quadtree division on each layer of Gaussian image, and define each grid area S as: S = {(x, y) | 0 ≤ x < W, 0 ≤ y < H}, where W and H are the width and height of the Gaussian image, respectively; execute the FAST algorithm in each grid area, and use the non-maximum suppression method around each detected candidate feature point to retain the feature point with the highest response value as the target object feature point; calculate the image moment of all target object feature points, and the specific formula is as follows: m pq =∑ x∈[0,W1] ∑ y∈[0,H1] I(x,y)·x p ·y q , Where I(x,y) represents the pixel value of the current grid area at the coordinate (x,y), W1 and H1 represent the width and height of the current grid area respectively, and p and q are the orders of the moment. According to the image moment, the centroid point C of all the feature points of the target object is calculated. The specific formula is: C=(C x ,C y )=(m 10 / m 00 ,m 01 / m 00 ), Among them C x represents the coordinate of the center of mass in the x direction, C y Indicates the coordinate of the center of mass in the y direction; m 10 Represents the first-order moment, the center of gravity of the current grid area in the x direction; m 00 Represents the zero-order moment of the current grid area, which is equal to the sum of all pixel values; m 01 Represents the first-order moment, the center of gravity of the current grid area in the y direction; calculates the direction angle θ of the feature point of the target object. The specific calculation formula is: θ=arctan(C y / C x )=arctan(m 01 / m 10 ); S103. Generate a corresponding feature point descriptor according to the centroid position and direction angle of each target object feature point, and aggregate the feature points and feature point descriptors of the same target object to form a target object feature set Kd.

3. The method for controlling a UAV mechanical arm based on machine vision according to claim 2, characterized in that: The FAST algorithm is executed in each grid area, and the non-maximum suppression method is used around each detected candidate feature point to retain the feature point with the highest response value as the target object feature point; the specific analysis process is as follows: For each detected candidate feature point I(x, y), define a circular area with a radius of r around it. For each candidate feature point I(x, y), obtain the corresponding brightness value Ic. In the circular area, obtain the brightness values ​​Ii of the corresponding N neighboring pixels, where i represents the neighboring pixel number, which is a positive integer from 1 to N; if there are n neighboring pixels that satisfy: Ii>Ic+T, or Ii<Ic-T; where T is the set threshold, the current candidate feature point I(x, y) is retained as a candidate corner point; Calculate the edge response R of each candidate corner point. The specific calculation formula is: R=det(M)-k·trace(M) 2 , Where M is the structure tensor of the image, det(M) represents the determinant of the structure tensor, trace(M) represents the trace of the structure tensor, and k represents an empirical constant. For each candidate corner point, non-maximum suppression is performed within a radius of r around it. The specific process is: within the circular area of ​​radius r of the candidate corner point, the response value R of the point is compared with the response values ​​of all other points in the neighborhood. Only when the response value R of the point is the largest in its neighborhood will it be retained.

4. The method for controlling a UAV mechanical arm based on machine vision according to claim 2, characterized in that: The step S200 includes: S201. Capture real-time image data in the power inspection operation through the camera equipment of the drone, and store these images as an image sequence It, and It = {I_1, I_2, ..., I_h}, where I_1 represents the real-time image corresponding to the first time point, I_2 represents the real-time image corresponding to the second time point, and so on, I_h represents the real-time image corresponding to the hth time point, and h represents the time index; for each real-time image in the image sequence It, analyze the feature points of the target object in step S100, so as to obtain the real-time image feature points and corresponding descriptors; S202. Obtain the target object feature set Kd, perform brute force matching on the descriptors corresponding to each target object in the target object feature set Kd and the descriptors of each real-time image in the image sequence It in turn, calculate the Hamming distance Dh between the descriptors corresponding to the two, and screen out matching pairs that are less than the threshold Dh0; use the RANSAC algorithm to remove mismatched points, find the inlier set Nd based on the matching point pairs, and count the number of inliers in the inlier set. If the number of inliers is greater than or equal to 4, it is considered a valid match, thereby obtaining matching feature points, and marking the target object in each real-time image in the image sequence It based on the matching feature points.

5. The method for controlling a UAV mechanical arm based on machine vision according to claim 4, characterized in that: The step S300 includes: S301. For each real-time image in the image sequence It, a matching feature point of a marked target object in the first real-time image in the image sequence It is selected as the center of a circle to construct an initial two-dimensional image coordinate system; each real-time image in the image sequence It is sequentially mapped according to the initial two-dimensional image coordinate system, so that each real-time image in the image sequence It is mapped to the initial two-dimensional image coordinate system; the specific mapping process is as follows: Select a matching feature point of a marked target object from the first real-time image I_1 as the center of the circle, denoted as C = (x_c, y_c), and establish an initial two-dimensional coordinate system (X', Y') with C as the center of the circle. The coordinate transformation relationship is as follows: X' = x-x_c, Y' = y-y_c, so as to ensure that the position of C in the initial two-dimensional coordinate system is (0,0); according to the initial two-dimensional coordinate system, coordinate transformation is performed on all matching feature points in the real-time image I_1, so as to obtain the position coordinates of all matching feature points in the real-time image I_1 in the initial two-dimensional coordinate system; for all matching feature points in the second real-time image I_2, Feature point, select any matching feature point that exists in both the real-time image I_1 and the real-time image I_2 as a reference point, and perform corresponding conversion calculations on other matching feature points in the real-time image I_2 according to the mapping relationship between the position coordinates of the reference point in the real-time image I_2 and the position coordinates in the real-time image I_1, so as to obtain the position coordinates of all matching feature points in the real-time image I_2 in the initial two-dimensional coordinate system, and so on, perform the above conversion calculations on all real-time images in the image sequence It, so as to obtain the position coordinates of the matching feature points of all real-time images in the image sequence It in the initial two-dimensional coordinate system; S302. Obtain the position coordinates of the matching feature points of all real-time images in the initial two-dimensional coordinate system, recorded as: p_g = (x_g, y_g), where g is the index of the matching feature point in the initial two-dimensional coordinate system; convert the two-dimensional coordinates of each matching feature point into homogeneous coordinates p_g' = (x_g, y_g, 1) T ; Define the position coordinates of the matching feature points corresponding to the target object in the three-dimensional world coordinate system as G_g = (X_g, Y_g, Z_g) T , using the intrinsic parameter matrix K and extrinsic parameter matrix [Rr|tr] of the drone camera device, where Rr represents the rotation matrix and tr represents the translation vector; expressed as: p_g=K[Rr|tr]G_g; using the equations of all matching feature points, establish the error function: E(Rr,tr)=∑g∈[1,M],||p_g’-K[Rr|tr]G_g|| 2 , Where M represents the total number of matching feature points; the optimized rotation and translation parameters are obtained by minimizing E(Rr, tr), thereby obtaining the actual position T_actual of the robot arm; S303. The origin of the desired grasping position coordinate system is recorded as O', and the relationship with the three-dimensional world coordinate system of the target object is determined by the rotation matrix R_target and the translation vector t_target. The specific relationship is: G_g'=R_targetG_g+t_target, G_g' represents the coordinates of the target object in the desired grasping position coordinate system; for each matching feature point G_g, the three-dimensional coordinates G_g of the target object are converted into homogeneous coordinates p_g' on the image plane through the camera model using the optimized rotation matrix Rr and translation vector tr; p_g' is mapped back to the three-dimensional world coordinate system to obtain the position of the target object in the world coordinate system: G_g=K -1 p_g'; using the defined relationship between the desired grasping position coordinate system and the target object coordinate system, G_g is converted into G_g'; through the above conversion, the three-dimensional coordinates G_g' of the target object in the desired grasping position coordinate system are obtained, thereby obtaining the desired position and posture T_desired of the robot arm.

6. The method for controlling a UAV mechanical arm based on machine vision according to claim 5, characterized in that: The step S400 includes: S401. Convert the desired posture T_desired of the UAV manipulator from the desired grasping position coordinate system to the three-dimensional world coordinate system corresponding to the actual posture, thereby obtaining the desired posture T_desired_actual of the three-dimensional world coordinate system, and T_desired_actual = (T_actual) -1 T_desired, where (T_actual) -1 represents the inverse transformation of the actual posture; calculates the deviation value F between the desired posture T_desired_actual and the actual posture T_actual, and F = ||Δt||+R_e, where Δt represents the displacement deviation, ||Δt|| represents the modulus of the displacement deviation, and R_e represents the rotation error metric; S402. Assume that the control gains are Kp and Kr, which are used for adjusting the displacement and rotation respectively, calculate the adjusted displacement control parameter Δt_control = KpΔt, and calculate the adjusted rotation control parameter ΔR_control = KrR_e; update the motion control parameters of the UAV robotic arm according to the calculated adjustment parameters Δt_control and ΔR_control; and update the motion trajectory of the UAV robotic arm in real time based on the adjusted displacement and rotation control parameters.

7. A UAV mechanical arm control system based on machine vision, applied to a UAV mechanical arm control method based on machine vision according to any one of claims 1 to 6, characterized in that: The system includes: an image acquisition module, a feature point extraction and matching module, a posture analysis module, a control parameter calculation and adjustment module, and a motion control and execution module; The image acquisition module uses the camera equipment of the drone to perform power inspection along a preset path to collect image data of the target area; processes the collected image data, generates reference image data, and performs image recording at multiple angles and multiple times; The feature point extraction and matching module extracts the feature points of the target object from the reference image data and constructs a target object feature set, extracts feature points from the real-time image data, matches the feature points with the target object feature set, and selects effective matching feature points; The posture analysis module analyzes the actual posture of the UAV mechanical arm according to the matching feature points, converts the three-dimensional coordinates of the target object into posture information in the coordinate system of the desired grasping position, and calculates the actual posture and expected posture of the mechanical arm through an optimization algorithm; The control parameter calculation and adjustment module calculates the deviation between the desired posture and the actual posture, analyzes the displacement and rotation errors, sets the control gain, calculates the adjusted displacement and rotation control parameters, and updates the motion control parameters of the UAV mechanical arm; The motion control and execution module updates the motion trajectory of the UAV mechanical arm in real time based on the adjusted motion control parameters.

8. The UAV mechanical arm control system based on machine vision according to claim 7, characterized in that: The image acquisition module includes an image acquisition unit and an image storage unit; The image acquisition unit is responsible for capturing image data during the power inspection operation; the image storage unit stores the captured image data in time series; The feature point extraction and matching module includes an image preprocessing unit, a feature point extraction unit and a feature point matching unit; The image preprocessing unit preprocesses the collected image; the feature point extraction unit extracts the feature points of the target object in the preprocessed image using technologies such as the FAST algorithm, and calculates the descriptor of the feature points of the target object; the feature point matching unit matches the feature points in the real-time image with the feature points in the reference image, uses brute force matching and the RANSAC algorithm to remove incorrect matches, and screens out valid matching feature points.

9. The UAV mechanical arm control system based on machine vision according to claim 7, characterized in that: The posture analysis module includes a coordinate system mapping unit, an actual posture calculation unit and an expected posture calculation unit; The coordinate system mapping unit maps the matching feature points in the real-time image to the initial two-dimensional coordinate system to perform position conversion; The actual posture calculation unit calculates the three-dimensional world coordinates of the matching feature points using the internal and external parameters of the camera, and establishes an error function to obtain the actual posture of the UAV robotic arm; the expected posture calculation unit calculates the expected posture of the robotic arm based on the three-dimensional coordinates of the target object and its relationship in the expected grasping position coordinate system.

10. The UAV mechanical arm control system based on machine vision according to claim 7, characterized in that: The control parameter calculation and adjustment module includes a deviation calculation unit and a control parameter adjustment unit; The deviation calculation unit calculates the deviation between the actual posture and the expected posture of the UAV mechanical arm, including displacement and rotation errors; the control parameter adjustment unit calculates the adjusted displacement and rotation control parameters according to the set control gain.

Images