A high-altitude foreign object handling flying robotic arm and its usage method

Abstract

A high-altitude foreign object handling flying robotic arm and its usage method are disclosed. The high-altitude foreign object handling flying robotic arm includes a flying drone with a connecting base at its bottom. A three-degree-of-freedom robotic arm is connected to the bottom of the connecting base. The three-degree-of-freedom robotic arm has a robotic gripper at its bottom and a vision module on the robotic gripper. The vision module can identify high-altitude foreign objects and transmit monitoring images to a host computer. By combining the robotic arm with the drone, the drone can have the ability to operate autonomously, transforming passive operations into active operations, thereby completing more complex tasks.

Description

Technical Field

[0001] This invention belongs to the field of mechanical structure technology, specifically relating to a high-altitude foreign object handling flying robotic arm and its usage method. Background Technology

[0002] Foreign object (FOO) handling at heights is a common problem in many industries, including power, construction, and transportation. It is particularly problematic when FEOs are attached to high-voltage cables or other overhead structures, as their suspension can severely impact the safe operation of these facilities. Currently, the main FEOs include kites, sky lanterns, woven bags, dust nets, and greenhouse plastic sheeting. These objects can be blown onto high-voltage cables or other overhead structures by strong winds or other natural factors. These FEOs can cause short circuits in power lines, signal interference, and even fires, posing a threat to public safety. Traditional methods rely on manual climbing or the use of long poles for removal, which is inefficient, time-consuming, and poses a safety risk to workers. Alternatively, using a flamethrower to burn the FEOs can damage high-voltage cables and the environment.

[0003] With the rapid development of drone technology, drones are playing an important role in fields such as aerial photography, on-site surveys, environmental modeling, and rescue missions. However, the aforementioned tasks do not involve contact interaction and perception between drones and the environment. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a high-altitude foreign object handling flying robotic arm and its usage method. The concept is to combine the robotic arm with a drone, which enables the drone to operate autonomously, transforming passive operations into active operations, thereby completing more complex tasks.

[0005] A high-altitude foreign object handling flying robotic arm includes a flying drone 1, with a connecting base 2 at the bottom of the flying drone 1; a three-degree-of-freedom robotic arm 3 is connected to the bottom of the connecting base 2, and a robotic gripper 5 is provided at the bottom of the three-degree-of-freedom robotic arm 3. The robotic gripper 5 is equipped with a vision module 4, which can identify high-altitude foreign objects and transmit monitoring images to a host computer.

[0006] More specifically, the connecting base 2 includes a butterfly-shaped connecting plate 7, which is connected to the bottom of the flying drone 1 by screws 6; the bottom of the butterfly-shaped connecting plate 7 is connected to the shoulder joint base 9 by a vertical internally threaded cylindrical pin 8, and the bottom of the shoulder joint base 9 is connected to the three-degree-of-freedom robotic arm 3.

[0007] More specifically, the three-degree-of-freedom robotic arm 3 includes a shoulder joint 10, with the top of the shoulder joint 10 connected to a shoulder joint base 9 and the bottom of the shoulder joint 10 connected to a first link 14; the first link 14 is connected to a second link 12 via an elbow joint 11; the second link 12 is connected to a third link 15 via a wrist joint 13, and the third link 15 is a robotic gripper 5.

[0008] More specifically, the rotation angles of the shoulder joint 10, elbow joint 11, and wrist joint 13 are all 0° to 180°.

[0009] More specifically, the flying drone 1 is the DJI M350RTK drone.

[0010] More specifically, the steps of using the high-altitude foreign object handling robotic arm of the present invention are as follows:

[0011] A. Manual mode;

[0012] A1. The operator remotely flies the drone 1 to the location of the high-altitude foreign object to be cleared, ensuring that the high-altitude foreign object is within the travel range of the three-degree-of-freedom robotic arm 3;

[0013] A2. The operator can individually control the forward and reverse rotation of the shoulder joint 10, elbow joint 11 and wrist joint 13 of the three-degree-of-freedom robotic arm 3 through the joint control buttons in the external robotic arm control unit; the operator can observe the position of the robotic gripper 5 in real time through the image transmitted back by the vision module 4, and thus adjust the robotic gripper 5 to reach the appropriate position.

[0014] A3. The operator controls the opening and closing of the robotic gripper 5 through the external robotic arm control unit to grab foreign objects from high altitudes;

[0015] B. Automatic mode;

[0016] B1. The operator presses the automatic control grasping button on the external robotic arm control unit. The host computer establishes the DH coordinate system of the three-degree-of-freedom robotic arm using the center of the drone's bottom as the base coordinate system, constructs the transformation matrix between each link, and thus obtains the spatial pose of the three-degree-of-freedom robotic arm. The DH coordinate system of the three-degree-of-freedom robotic arm is as follows:

[0017]

[0018] Where α i For from z i To z i+1 Around x i The rotation angle; a i Indicates along x i axis, from z i Move to z i+1 distance; d i For from x i-1 To x i Along z i Distance in direction; θ i For from x i-1 To x i Around z i The rotation angle;

[0019] The joint transformation matrix of the three-degree-of-freedom robotic arm is: The pose transformation matrix of the i-th joint relative to the world coordinate system and the position coordinates of the i-th joint can be derived from the transformation matrix. In a three-degree-of-freedom robotic arm, if the angles of the three joints are known as θ1, θ2, and θ3, the pose P(X, Y, Z, R) of the end effector relative to the base coordinate system can be obtained. x ,R y ,R z Then (X,Y,Z,R) can be established. x ,R y ,R z The transformation relationship of ) = H(θ1,θ2,θ3);

[0020] Where H(θ1,θ2,θ3) is the relationship function between the joint angles of the robotic arm and the end-effector pose of the robotic arm;

[0021] B2. Construct a self-made high-altitude foreign object dataset, preprocess the label information of the images in the dataset, and divide the dataset into a validation set and a test set;

[0022] Specifically, images of various types of high-altitude foreign objects were captured in different scenarios. A publicly available dataset of recyclable object images was collected online. The collected images were categorized and organized according to types such as kites, sky lanterns, woven bags, dust nets, and greenhouse plastic sheeting. Annotation tools were used to accurately label the recyclable objects in each image, ensuring the accuracy and consistency of the labels. The dataset was divided into training, validation, and test sets to balance the needs of model training and performance evaluation. The YOLOv7 object detection model was used to train the model on the training set, and its performance was continuously improved through validation and test sets.

[0023] Vision module 4 uses a trained object detection model to identify foreign objects at high altitudes and determine their types. Then, using the contours of the foreign objects as feature points, it correlates the velocity change matrix of the robotic gripper with the rate of change of the contour feature points to construct the image Jacobian matrix. Its differential relationship is as follows:

[0024] in This indicates the speed of the end effector of the robotic arm. Indicates the rate of change of image feature points. The image Jacobian matrix is ​​expressed as follows:

[0025]

[0026] The Jacobian matrix is ​​m rows and n columns. When calculating the end effector velocity using the image Jacobian, two cases need to be considered: when the rank of the Jacobian matrix is ​​n... It is a singular matrix. If the inverse matrix exists, then we can find:

[0027]

[0028] When the rank of the Jacobian matrix is ​​not n The inverse matrix does not exist; in this case, assume... Full rank, calculate the least squares solution:

[0029]

[0030] in yes The pseudo-inverse of , where b is any vector of the appropriate dimension;

[0031] B3. The host computer constructs the objective function. Where e is the image error function, which is related to the joint angle θ of the robotic arm and time t; the global minimum of the objective function is obtained by using the Gauss-Newton iteration method, so as to continuously feed back and control the joint movement of the robotic arm until the target object is grasped.

[0032] 1. The beneficial effects of this invention are as follows: Clearing foreign objects from high altitudes eliminates the need for manual climbing to the object, significantly reducing the risks associated with working at heights and preventing falls. Furthermore, for foreign objects on high-voltage cables, there is no need to interrupt the power supply, nor is it necessary to burn the object, thus minimizing damage to the high-voltage cables and the environment.

[0033] 2. The automatic control mode can quickly identify targets and execute removal actions, significantly improving the speed and efficiency of foreign object removal and reducing operation time. The manual control mode allows operators to make fine adjustments according to the actual situation, improving the ability to handle complex or irregular foreign objects. Operators can choose one of the two modes individually or combine the two modes, making it highly adaptable to complex environments. Attached Figure Description

[0034] Figure 1 This is a structural diagram of a high-altitude foreign object handling flying robotic arm according to the present invention.

[0035] Figure 2 This is a structural diagram of the connecting base of the present invention.

[0036] Figure 3 This is a structural diagram of the three-degree-of-freedom robotic arm 3 of the present invention.

[0037] Figure 4 This is a schematic diagram of the drone remote control interface structure of the present invention. Detailed Implementation

[0038] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0039] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0040] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0041] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0042] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0043] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0044] The present invention will now be described in detail with reference to the accompanying drawings and exemplary embodiments.

[0045] A high-altitude foreign object handling flying robotic arm includes a flying drone 1, with a connecting base 2 at the bottom of the flying drone 1; a three-degree-of-freedom robotic arm 3 is connected to the bottom of the connecting base 2, and a robotic gripper 5 is provided at the bottom of the three-degree-of-freedom robotic arm 3. The robotic gripper 5 is equipped with a vision module 4, which can identify high-altitude foreign objects and transmit monitoring images to a host computer.

[0046] More specifically, the connecting base 2 includes a butterfly-shaped connecting plate 7, which is connected to the bottom of the flying drone 1 by screws 6; the bottom of the butterfly-shaped connecting plate 7 is connected to the shoulder joint base 9 by a vertical internally threaded cylindrical pin 8, and the bottom of the shoulder joint base 9 is connected to the three-degree-of-freedom robotic arm 3.

[0047] More specifically, the three-degree-of-freedom robotic arm 3 includes a shoulder joint 10, with the top of the shoulder joint 10 connected to a shoulder joint base 9 and the bottom of the shoulder joint 10 connected to a first link 14; the first link 14 is connected to a second link 12 via an elbow joint 11; the second link 12 is connected to a third link 15 via a wrist joint 13, and the third link 15 is a robotic gripper 5.

[0048] More specifically, the rotation angles of the shoulder joint 10, elbow joint 11, and wrist joint 13 are all 0° to 180°.

[0049] More specifically, the flying drone 1 is the DJI M350RTK drone.

[0050] More specifically, the steps of using the high-altitude foreign object handling robotic arm of the present invention are as follows:

[0051] A. Manual mode;

[0052] A1. The operator remotely flies the drone 1 to the location of the high-altitude foreign object to be cleared, ensuring that the high-altitude foreign object is within the travel range of the three-degree-of-freedom robotic arm 3;

[0053] A2. The operator can individually control the forward and reverse rotation of the shoulder joint 10, elbow joint 11 and wrist joint 13 of the three-degree-of-freedom robotic arm 3 through the joint control buttons in the external robotic arm control unit; the operator can observe the position of the robotic gripper 5 in real time through the image transmitted back by the vision module 4, and thus adjust the robotic gripper 5 to reach the appropriate position.

[0054] A3. The operator controls the opening and closing of the robotic gripper 5 through the external robotic arm control unit to grab foreign objects from high altitudes;

[0055] B. Automatic mode;

[0056] B1. The operator presses the automatic control grasping button on the external robotic arm control unit. The host computer establishes the DH coordinate system of the three-degree-of-freedom robotic arm using the center of the drone's bottom as the base coordinate system, constructs the transformation matrix between each link, and thus obtains the spatial pose of the three-degree-of-freedom robotic arm. The DH coordinate system of the three-degree-of-freedom robotic arm is as follows:

[0057]

[0058] Where α i For from z i To z i+1 Around x i The rotation angle; a i Indicates along x i axis, from z i Move to z i+1 distance; d i For from x i-1 To x i Along z i Distance in direction; θ i For from x i-1 To x i Around z i The rotation angle;

[0059] The joint transformation matrix of the three-degree-of-freedom robotic arm is: The pose transformation matrix of the i-th joint relative to the world coordinate system and the position coordinates of the i-th joint can be derived from the transformation matrix. In a three-degree-of-freedom robotic arm, if the angles of the three joints are known as θ1, θ2, and θ3, the pose P(X, Y, Z, R) of the end effector relative to the base coordinate system can be obtained. x ,R y ,R z Then (X,Y,Z,R) can be established. x ,R y ,R z The transformation relationship of ) = H(θ1,θ2,θ3);

[0060] Where H(θ1,θ2,θ3) is the relationship function between the joint angles of the robotic arm and the end-effector pose of the robotic arm;

[0061] B2. Construct a self-made high-altitude foreign object dataset, preprocess the label information of the images in the dataset, and divide the dataset into a validation set and a test set;

[0062] Specifically, images of various types of high-altitude foreign objects were captured in different scenarios. A publicly available dataset of recyclable object images was collected online. The collected images were categorized and organized according to types such as kites, sky lanterns, woven bags, dust nets, and greenhouse plastic sheeting. Annotation tools were used to accurately label the recyclable objects in each image, ensuring the accuracy and consistency of the labels. The dataset was divided into training, validation, and test sets to balance the needs of model training and performance evaluation. The YOLOv7 object detection model was used to train the model on the training set, and its performance was continuously improved through validation and test sets.

[0063] Vision module 4 uses a trained object detection model to identify foreign objects at high altitudes and determine their types. Then, using the contours of the foreign objects as feature points, it correlates the velocity change matrix of the robotic gripper with the rate of change of the contour feature points to construct the image Jacobian matrix. Its differential relationship is as follows:

[0064] in This indicates the speed of the end effector of the robotic arm. Indicates the rate of change of image feature points. The image Jacobian matrix is ​​expressed as follows:

[0065]

[0066] The Jacobian matrix is ​​m rows and n columns. When calculating the end effector velocity using the image Jacobian, two cases need to be considered: when the rank of the Jacobian matrix is ​​n... It is a singular matrix. If the inverse matrix exists, then we can find:

[0067]

[0068] When the rank of the Jacobian matrix is ​​not n The inverse matrix does not exist; in this case, assume... Full rank, calculate the least squares solution:

[0069]

[0070] in yes The pseudo-inverse of , where b is any vector of the appropriate dimension;

[0071] B3. The host computer constructs the objective function. Where e is the image error function, which is related to the joint angle θ of the robotic arm and time t; the global minimum of the objective function is obtained by using the Gauss-Newton iteration method, so as to continuously feed back and control the joint movement of the robotic arm until the target object is grasped.

[0072] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A high-altitude foreign-object debris handling flight robotic arm, characterized by: The application relates to a high-altitude foreign matter processing flight mechanical arm, which comprises a flight unmanned aerial vehicle (1), wherein the bottom of the flight unmanned aerial vehicle (1) is provided with a connecting base (2); the bottom of the connecting base (2) is connected with a three-degree-of-freedom mechanical arm (3); the bottom of the three-degree-of-freedom mechanical arm (3) is provided with a mechanical gripper (5); and a vision module (4) is arranged on the mechanical gripper (5), wherein the vision module (4) can identify high-altitude foreign matters and transmit monitoring pictures to an upper computer. The connecting base (2) comprises a butterfly-shaped connecting plate (7), the butterfly-shaped connecting plate (7) is connected with the bottom of the flight unmanned aerial vehicle (1) through screws (6), the bottom of the butterfly-shaped connecting plate (7) is connected with a shoulder joint base (9) through a vertical internal thread cylindrical pin (8), and the bottom of the shoulder joint base (9) is connected with the three-degree-of-freedom mechanical arm (3). The three-degree-of-freedom mechanical arm (3) comprises a shoulder joint (10), the top of the shoulder joint (10) is connected with the shoulder joint base (9), the bottom of the shoulder joint (10) is connected with a first connecting rod (14), the first connecting rod (14) is connected with a second connecting rod (12) through an elbow joint (11), the second connecting rod (12) is connected with a third connecting rod (15) through a wrist joint (13), and the third connecting rod (15) is provided with the mechanical gripper (5). The use method of the high-altitude foreign matter processing flight mechanical arm comprises the following steps: A. Manual mode; A1. An operator remotely controls the flight unmanned aerial vehicle (1) to fly to a high-altitude foreign matter to be cleaned, so that the high-altitude foreign matter is within the stroke range of the three-degree-of-freedom mechanical arm (3); A2. The operator controls the forward and reverse rotation directions of the shoulder joint (10), the elbow joint (11) and the wrist joint (13) of the three-degree-of-freedom mechanical arm (3) through the joint control buttons in an external mechanical arm control unit; the operator adjusts the position of the mechanical gripper (5) to the appropriate position by observing the position of the mechanical gripper (5) through the pictures returned by the vision module (4); A3. The operator controls the opening and closing of the mechanical gripper (5) through the external mechanical arm control unit to grasp the high-altitude foreign matter; B. Automatic mode; B1. The operator presses the automatic control grasping button of the external mechanical arm control unit, the upper computer takes the center of the bottom of the unmanned aerial vehicle as a base coordinate, establishes a three-degree-of-freedom mechanical arm D-H coordinate, constructs a transformation matrix between the connecting rods, and obtains the spatial pose of the three-degree-of-freedom mechanical arm, and the three-degree-of-freedom mechanical arm D-H coordinate is wherein is the distance from to is the angle of rotation about ; is the distance from to ; is the distance from to in the direction of ; is the angle of rotation about ; is the distance from to ; The joint transformation matrix of the three-degree-of-freedom robotic arm is: The transformation matrix can be used to derive the first... The pose transformation matrix of the first joint relative to the world coordinate system and the first joint's pose transformation matrix relative to the world coordinate system. The position coordinates of the three joints; in a three-degree-of-freedom robotic arm, if the angles of the three joints are known... To obtain the pose of the end effector relative to the base coordinate system Then it can be established The transformation relationship; wherein ; B2. A self-made high-altitude foreign matter data set is constructed, the label information of the images in the data set is preprocessed, and the data set is divided into a verification set and a test set; Specifically, images of various high-altitude foreign matters in different scenes are shot, publicly available reusable product image data sets on the Internet are collected, and the collected images are classified and arranged according to kites, Kongming lanterns, woven bags, dustproof nets and greenhouse plastic sheets; a marking tool is used to accurately mark the reusable products in each image, so that the accuracy and consistency of the labels are ensured; the data set is divided into a training set, a verification set and a test set to balance the needs of model training and performance evaluation; a YOLOv7 target detection model is adopted to train the training set, and the effect of the target detection model is continuously improved through the verification set and the test set; The target recognition unit uses the trained target detection model to recognize the high-altitude foreign matter to obtain a type of the high-altitude foreign matter; and then takes the contour of the high-altitude foreign matter as a feature point, correlates a speed change matrix of the mechanical gripper with a change rate of the contour feature point, and constructs an image Jacobian matrix, a differential relationship of which is as follows: wherein denotes the velocity of the end effector of the robot arm, denotes the rate of change of the image feature point, is the image Jacobian matrix, which is expressed as: The Jacobian matrix is row column, using the image Jacobian to calculate the end effector velocity, two cases need to be considered: when the Jacobian matrix is , singular matrix, inverse matrix exists, then the following can be solved: When the Jacobian matrix is not full rank , , the inverse matrix does not exist, in which case it is assumed that is full rank, and the least squares solution is computed: wherein is the pseudo-inverse of is an arbitrary vector of appropriate dimension; B3. The higher authorities build the objective function wherein is the image error function, and the mechanical arm joint angle and time are related; the global minimum value of the objective function is obtained by using the Gauss-Newton iterative method, so as to constantly feedback control the joint movement of the mechanical arm until the target object is grabbed.

2. The high-altitude foreign-object disposal flying robotic arm of claim 1, wherein: The rotation angles of the shoulder joint (10), the elbow joint (11) and the wrist joint (13) are all 0 DEG ~ 180 DEG.

3. The high-altitude FOD handling flight robotic arm of claim 1, wherein: The flying drone 1 is a DJI drone M350 RTK.

Images