Vision-based long flexible mechanical arm tail end positioning method and system

Abstract

The invention discloses a vision-based long flexible mechanical arm tail end positioning method and system, and the method comprises the steps: obtaining the image pixel coordinates of a to-be-distributed operation point, and mapping the image pixel coordinates to a mechanical arm coordinate system; and the tail end track of the mechanical arm is planned, and angle changes of all joints are solved based on inverse kinematics. Errors caused by gravity and flexible deformation of the long arm support are considered, a camera is installed at the tail end, feature point images of a target area are collected in real time, expected image features are calculated by combining internal and external parameters of the camera, the expected image features are compared with actual features, and image feature errors are obtained. And the compensation speed is solved through a visual servo control method, the joint speed is further solved through a mechanical arm Jacobian matrix, and accurate positioning of the tail end is achieved. And when the error converges to a preset range, the mechanical arm automatically stops. According to the method, the positioning precision of the material distribution opening in the tail end of the 13-meter material distribution machine can be improved to be within 5 cm, manual dependence and operation intensity are effectively reduced, automation and precision of material distribution operation are improved, and the method has wide application prospects.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of engineering machinery and automation control, and particularly relates to a long flexible mechanical arm end positioning method and system based on vision. BACKGROUND

[0002] A long flexible mechanical arm is a multi-joint mechanical arm system with long structure size, light weight and obvious flexibility, and common examples include airplane cleaning mechanical arms, concrete pouring mechanical arms, space operation mechanical arms and the like. These mechanical arms are widely used in construction, municipal engineering, emergency rescue and the like operation scenarios. Due to the long arm and obvious flexibility, the skill requirement of an operator is high when controlling the mechanical arm to perform fine operations.

[0003] In addition, in most operation scenarios, the operation object is usually far away from the mechanical arm base and the operator, so that the end position control precision is insufficient, and deviation is easily generated between the mechanical arm end and the target operation point. In order to ensure that the end can reach the target position, the existing operation mode often needs multiple personnel to collaboratively observe, command or guide the mechanical arm movement, which is not only time-consuming but also has high operation risk.

[0004] Although vision technology can provide auxiliary information for end positioning, a traditional fixed or fuselage-mounted vision sensor often faces problems such as limited field of view and target being easily blocked by the fuselage under the conditions of large field of view and long distance, and it is difficult to continuously and stably obtain the relative position information between the end and the target, thereby limiting the application of vision positioning technology in the long flexible mechanical arm scenario. The existing vision positioning method generally depends on a fixed camera view angle or a preset expected pixel coordinate, but these methods have multiple deficiencies in actual construction: the target is easy to deviate from the camera field of view, leading to the inability to continue servo control, and the expected pixel coordinate is difficult to accurately obtain in advance, thereby affecting the positioning precision.

[0005] In view of this problem, the present application introduces a UAV and a gimbal camera mounted at the end of the mechanical arm, combines vision recognition and image-based visual servo technology, realizes accurate positioning of the end of the long flexible mechanical arm, greatly reduces the number of personnel and operation difficulty, and improves positioning precision and operation efficiency. SUMMARY

[0006] The present application aims at the deficiencies of the prior art, and proposes a long flexible mechanical arm end positioning method and system based on vision. Before operation, a camera carried by a UAV collects an overhead image of a construction area, a target operation area and a target operation point are automatically detected by using image recognition technology, and three-dimensional coordinates of the target point are calculated by ranging technology and space mapping.

[0007] The purpose of the present application is achieved by the following technical solution: a long flexible mechanical arm end positioning method based on vision, comprising the following steps:

[0008] Step 1: Obtain an image containing the target to be worked by a UAV, and convert the pixel coordinates of the corner points of the target to be worked in the image into position coordinates in the mechanical arm coordinate system based on the positional relationship between the UAV and the mechanical arm;

[0009] Step 2: Obtain the motion trajectory of the mechanical arm according to the positions of the corner points of the target to be worked, and solve the joint angle change value of the mechanical arm;

[0010] Step 3: Install a gimbal camera at the end of the mechanical arm to detect and identify the real-time pixel coordinates of the corner points of the target to be worked;

[0011] Step 4: Based on the fixed assembly relationship between the gimbal camera and the end of the mechanical arm, convert the desired position of the end of the mechanical arm to the camera coordinate system, and complete the imaging projection combined with the internal and external parameter matrices of the camera to obtain the desired pixel coordinates of the desired position in the image plane;

[0012] Step 5: Solve the image feature error according to the pixel coordinates of the desired image feature and the real-time pixel coordinates;

[0013] Step 6: Obtain the depth value according to the binocular parallax principle and the camera internal parameter matrix, and obtain the image Jacobian matrix combined with the depth value;

[0014] Step 7: Based on the visual servo method of the image, obtain the desired camera speed required to reach the desired image feature combined with the image Jacobian matrix; based on the positional relationship between the gimbal camera and the end of the mechanical arm, calculate the mechanical arm end speed required to reach the desired position, and solve the joint angle desired speed according to the mechanical arm Jacobian matrix;

[0015] Step 8: Control the motion of the end of the mechanical arm based on the joint angle desired speed, and stop the motion of the mechanical arm when the real-time image feature error is reduced to an acceptable range.

[0016] Further, in step 1, before the cloth work starts, use the UAV and the camera carried thereon to shoot an image containing the target to be worked in the construction scene, and after obtaining the pixel information of the target to be worked, obtain the coordinates of the target point relative to the UAV coordinate system according to the focal length of the camera and the flight height of the UAV.

[0017] Further, in step 1, obtain the pixel coordinates of the corner points in the image of the target to be worked according to the contour detection method.

[0018] Further, in step 2, the motion trajectory of the mechanical arm is obtained by using a quintic polynomial interpolation method.

[0019] Further, in step 4, the specific process of obtaining the desired pixel coordinates combined with the relative relationship between the camera internal and external parameter matrices and the end of the mechanical arm is as follows:

[0020] (1) According to the desired position of the end of the mechanical arm, the angle of the last section of the arm frame of the mechanical arm, and the installation position of the gimbal camera, the conversion matrix of the end of the mechanical arm and the gimbal camera is calculated;

[0021] (2) According to the position of the to-be-operated point obtained in step 1, the coordinates of the to-be-operated target corner point in the gimbal coordinate system are calculated;

[0022] (3) According to the internal parameter matrix of the binocular camera, the expected image feature of the operation target corner point is solved in combination with the image Jacobian matrix.

[0023] Further, in step 4, the field of view is adjusted in real time by the gimbal to ensure that the target is continuously in the effective imaging range, thereby continuously providing stable and reliable expected image features.

[0024] Further, in step 5, the image feature error is the difference between the real-time pixel coordinates and the expected pixel coordinates.

[0025] Further, based on the positional relationship between the gimbal camera and the end of the mechanical arm, the end-of-arm speed required to reach the desired position is calculated.

[0026] Further, in step 8, when the image feature error is less than a threshold value, the movement of the mechanical arm is stopped, and the threshold value is a preset tolerance.

[0027] On the other hand, the application also provides a long flexible mechanical arm end positioning system based on vision, which comprises:

[0028] An operation target coordinate acquisition module is used to obtain an image containing a to-be-operated target by a UAV, and pixel coordinates of a to-be-operated target corner point in the image are converted into position coordinates in a mechanical arm coordinate system based on the positional relationship between the UAV and the mechanical arm;

[0029] A motion trajectory acquisition module is used to obtain the motion trajectory of the mechanical arm according to the obtained position of the to-be-operated target corner point, and the change value of the joint angle of the mechanical arm is solved;

[0030] A pixel coordinate identification module is used to detect and identify real-time pixel coordinates of a to-be-operated target corner point according to a gimbal camera mounted at the end of the mechanical arm;

[0031] An expected pixel coordinate solving module is used to convert the desired position of the end of the mechanical arm to the camera coordinate system based on the fixed assembly relationship between the gimbal camera and the end of the mechanical arm, and complete imaging projection in combination with the internal and external parameters of the camera to obtain the expected pixel coordinates of the desired position in the image plane;

[0032] A feature error solving module is used to solve the image feature error according to the pixel coordinates of the expected image feature and the real-time pixel coordinates.

[0033] An image Jacobian matrix module is configured to obtain a depth value according to a binocular disparity principle and a camera intrinsic parameter matrix, and obtain an image Jacobian matrix in combination with the depth value;

[0034] A joint angle desired velocity solving module is configured to obtain a camera desired velocity required to reach a desired image feature according to an image feature error and the image Jacobian matrix based on an image-based visual servoing method, calculate a mechanical arm end desired velocity required to reach a desired position based on a position relationship between the gimbal camera and the mechanical arm end, and obtain a joint angle desired velocity based on a mechanical arm Jacobian matrix solving.

[0035] A mechanical arm end control module is configured to control the mechanical arm end movement based on the joint angle desired velocity, and stop the mechanical arm movement when the real-time image feature error is reduced to an acceptable range.

[0036] Compared with the prior art, the unmanned aerial vehicle remote sensing, image recognition, motion planning, visual servoing and control of the flexible mechanical arm are deeply integrated, the automation positioning precision of the long flexible mechanical arm during the execution of the construction task is significantly improved, the operation strength and personnel risk are reduced, and the method has a wide engineering application prospect.

[0037] The method has the following beneficial effects:

[0038] 1. The method realizes the automation high-precision positioning of the mechanical arm end by using the unmanned aerial vehicle, the gimbal camera and the like, in combination with the vision and the trajectory planning technology, does not require manual close-range command or traction, greatly reduces the operation strength and the construction risk, and is suitable for complex construction environments.

[0039] 2. The unmanned aerial vehicle top-view imaging, the identification of the operation area and the target point, the spatial mapping and the multi-coordinate system conversion are combined, reliable target three-dimensional positions can be obtained in the complex environment, and high-quality data are provided for the accurate positioning.

[0040] 3. The visual servoing control is constructed based on the error between the actual and desired pixel coordinates, the end positioning error caused by the flexible deformation of the arm support and the load disturbance is compensated in real time, the mechanical arm end position always approaches the desired position during the movement process, and the positioning precision of the long flexible mechanical arm is improved.

[0041] 4. The experiments performed on the concrete spreader with a length of about 13 meters show that the positioning method can reduce the end positioning error to within 5 cm, and the significant effect of the trajectory planning, the desired image feature solving and the visual servoing technology in the positioning operation is verified.

[0042] 5. The method has good compatibility and expansibility, and can be applied to multiple types of long flexible arm support engineering equipment such as the concrete spreader. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0044] Figure 1 The schematic diagram of the positioning method of the present application.

[0045] Figure 2 The system structure schematic diagram of the end positioning method of the long flexible mechanical arm based on vision.

[0046] Figure 3 The flow chart of the unmanned aerial vehicle recognizing the column, the mechanical arm end trajectory planning and motion control.

[0047] Figure 4 The schematic diagram of the desired image solving and vision servo closed-loop control principle.

[0048] Figure 5 The mechanical arm motion process diagram under actual working conditions.

[0049] Figure 6 The end position change schematic diagram under actual operation. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described below in combination with the drawings and implementation examples. It should be understood that the specific implementation examples described here are only used to explain the present application, and are not used to limit the present application.

[0051] The system of the present application combines the spatial position relationship of the unmanned aerial vehicle, the mechanical arm base and each joint, and calculates the spatial position of the target point under the mechanical arm coordinate system. The quintic polynomial interpolation trajectory planning algorithm is adopted to generate a smooth motion trajectory for the mechanical arm from the current state to the target point, and the inverse kinematics is solved to realize the automatic control of the long flexible mechanical arm end.

[0052] Due to the large structural flexibility of the long flexible mechanical arm, the structure will be deformed due to the self-weight and dynamic load in actual operation, so that the end position deviates from the set target. The gimbal vision sensor is installed at the end of the mechanical arm in the present application, the working point image is collected in real time, and based on the vision servo control algorithm, the mechanical arm posture is dynamically adjusted according to the image feature error, so as to realize high-precision closed-loop control. When the image error converges to the set threshold, the motion is automatically stopped, so as to complete the high-precision end positioning.

[0053] As Figure 1 and Figure 2As shown, this invention proposes a vision-based end-effector positioning method for long, flexible robotic arms, targeting large-sized robotic arms with multi-degree-of-freedom boom structures, and is particularly suitable for remote, automated, and intelligent concrete placing operations in engineering machinery such as large flexible boom concrete placing machines. The main implementation process is as follows:

[0054] 1. Work site identification and spatial positioning

[0055] Before the robotic arm's end-effector positioning operation begins, the operator uses a handle to control the drone to take off and ascend to a certain height h. The drone then uses an onboard high-resolution camera to acquire a top-down view of the work area. The system uses visual algorithms such as contour detection and region segmentation to identify the area and points to be worked on, obtaining the pixel coordinates of the columns to be poured in the image. .

[0056] Based on the known UAV flight altitude h and camera focal length f, the pixel coordinates of the task point are converted into physical coordinates (x, y, z) in the UAV coordinate system using the principle of similar triangle imaging. The calculation formula is as follows:

[0057]

[0058] By combining the spatial relative positions of the UAV takeoff point and the long flexible robotic arm base, rigid body transformation is used to map the target point position to the robotic arm's own coordinate system, providing input for subsequent motion planning.

[0059] 2. Robotic arm motion trajectory planning

[0060] like Figure 3 As shown, after obtaining the spatial coordinates of the target work point, the system generates the end effector trajectory based on the current state of the robotic arm and using a fifth-order polynomial interpolation method. The trajectory is expressed as:

[0061]

[0062] in This represents the spatial path of the end effector as time t changes. The trajectory starts at the current actual position of the robotic arm's end effector and ends above the mapped target placement point (e.g., 1m above the target point). By constraining the trajectory's start and end velocities and acceleration, the smoothness and continuity of the boom's movement are achieved, avoiding large swaying and impacts.

[0063] 3. Inverse kinematics solution and joint command generation

[0064] Based on the position of the trajectory point in the robot arm coordinate system, the system uses inverse kinematics to solve the joint angles at each moment in real time, based on the arm structure parameters and constraints. :

[0065]

[0066] IK is the inverse kinematics operator. The resulting joint angle sequence is sent as control commands to the robotic arm controller, driving the robotic arm to move along the planned trajectory.

[0067] 4. Perception and Compensation of Deformation Error in Flexible Arm

[0068] Because the robotic arm has a long boom and a flexible structure, it will undergo elastic deformation due to its own weight and dynamic load during deployment and movement, resulting in an error between the actual position of the robotic arm's end and the expected trajectory.

[0069] Therefore, such as Figure 4 As shown, this invention has a gimbal camera installed at the material placement port of the robotic arm. During operation, the end-effector camera collects the pixel coordinates of the four corners of the column to be poured in real time. Combining the assembly parameters of the robotic arm end-effector and the camera, and the camera's intrinsic and extrinsic parameter matrix K, the desired spatial position of the target point is determined using the space-pixel transformation relationship. Projecting onto the camera image plane yields the desired pixel coordinates. :

[0070]

[0071] in, This is a transformation matrix that converts the absolute spatial coordinates of the target point into coordinates in the camera coordinate system.

[0072] 5. Real-time target detection and closed-loop servo correction

[0073] The system is based on target detection and key point extraction algorithms to identify the pixel coordinates s of feature points of the target structure (pillar) in the current image in real time.

[0074] With desired pixel coordinates Using this as a baseline, calculate the current image feature error e:

[0075]

[0076] A visual servoing (IBVS)-based control method is employed to construct the image Jacobian matrix. This establishes a mapping relationship between error and the end effector speed of the robotic arm.

[0077]

[0078] in, This represents the proportional gain coefficient in visual servo control. It determines the speed and intensity of the robotic arm's error correction.

[0079] Based on the aforementioned compensation speed, the main control unit uses the Jacobian matrix to calculate the end velocity. Compensation velocity mapped to joint space The robotic arm's joints are adjusted in real time to gradually converge the end-effector of the fabric to the target point in pixel space. ( When setting a tolerance, the system determines that the end effector is precisely aligned with the target point and automatically stops moving.

[0080] 6. Verification through practical experiments

[0081] To further verify the effectiveness of the proposed method on actual equipment, an end-effector positioning experiment was conducted on a 13m long three-joint fabric placing machine. Based on the position of the uprights identified by the drone, a corresponding end-effector trajectory was planned. The specific positioning operation is as follows: Figure 5 As shown, during the 0-30s interval, the end effector completes its movement along a pre-planned trajectory. Due to the long boom and hollow structure, the end effector position is affected by deformation. For example... Figure 6 As shown, at the end of the movement, there is still a positioning error of about 60cm at the end of the robotic arm. At this time, the error between the end position and the desired point is large, which affects the operation accuracy.

[0082] At this point, the desired image features are calculated and combined with an image-based visual servoing method to achieve precise end-effector localization. For example... Figure 5 and Figure 6 As shown, during the 40s-60s period, as the robotic arm moves, the end-effector positioning error is reduced from 60cm to about 5cm, which greatly improves the end-effector positioning accuracy.

[0083] On the other hand, corresponding to the aforementioned embodiment of a vision-based long flexible robotic arm end-effector positioning method, the present invention also provides a vision-based long flexible robotic arm end-effector positioning system, the system comprising:

[0084] The target coordinate acquisition module is used to obtain an image containing the target to be worked through the UAV, and convert the pixel coordinates of the corner points of the target to be worked in the image into position coordinates in the robotic arm coordinate system based on the positional relationship between the UAV and the robotic arm.

[0085] The motion trajectory acquisition module is used to obtain the motion trajectory of the robotic arm based on the position of the corner point of the target to be worked, and to solve for the change value of the joint angle of the robotic arm.

[0086] The pixel coordinate recognition module is used to detect and identify the real-time pixel coordinates of the corner points of the target to be operated based on the gimbal camera installed at the end of the robotic arm.

[0087] The desired pixel coordinate solution module, based on the fixed assembly relationship between the gimbal camera and the end effector of the robotic arm, transforms the desired position of the end effector of the robotic arm into the camera coordinate system, and combines the camera's intrinsic and extrinsic parameters to complete the imaging projection, thereby obtaining the desired pixel coordinates of the desired position in the image plane.

[0088] The feature error solving module is used to solve for the image feature error based on the pixel coordinates of the desired image features and the real-time pixel coordinates.

[0089] The image Jacobian matrix module is used to obtain depth values ​​based on the binocular parallax principle and the camera intrinsic parameter matrix, and then to obtain the image Jacobian matrix by combining the depth values.

[0090] The joint angle expected velocity calculation module is used for image-based visual servoing methods. It obtains the camera expected velocity required to achieve the desired image features based on the image feature error and the image Jacobian matrix. Based on the positional relationship between the gimbal camera and the robotic arm end effector, it calculates the robotic arm end effector velocity required to reach the desired position and obtains the joint angle expected velocity based on the robotic arm Jacobian matrix.

[0091] The robotic arm end effector control module controls the movement of the robotic arm end effector based on the expected velocity of the joint angle. When the real-time image feature error is reduced to an acceptable range, the robotic arm stops moving.

[0092] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A vision-based method for end-effector positioning of a long, flexible robotic arm, characterized in that: Includes the following steps: Step 1: Obtain an image containing the target to be worked using a drone, and convert the pixel coordinates of the corner points of the target in the image into position coordinates in the robotic arm coordinate system based on the positional relationship between the drone and the robotic arm; Step 2: Based on the obtained position of the corner point of the target to be worked, obtain the motion trajectory of the robotic arm and solve for the change value of the joint angle of the robotic arm; Step 3: Install a gimbal camera at the end of the robotic arm to detect and identify the real-time pixel coordinates of the corner points of the target to be worked on; Step 4: Based on the fixed assembly relationship between the gimbal camera and the end effector of the robotic arm, the desired position of the end effector of the robotic arm is transformed into the camera coordinate system, and the imaging projection is completed by combining the intrinsic and extrinsic parameters of the camera to obtain the desired pixel coordinates of the desired position in the image plane. Step 5: Based on the pixel coordinates of the desired image features and the real-time pixel coordinates, calculate the image feature error; Step 6: Obtain the depth value based on the binocular parallax principle and the camera intrinsic parameter matrix, and then calculate the image Jacobian matrix by combining the depth value; Step 7: Image-based visual servoing method: Based on the image feature error and the image Jacobian matrix, the desired camera speed required to achieve the desired image features is obtained; Based on the positional relationship between the gimbal camera and the robotic arm end effector, the robotic arm end effector speed required to reach the desired position is calculated, and the desired joint angle speed is obtained by solving the robotic arm Jacobian matrix. Step 8: Control the end effector movement of the robotic arm based on the expected velocity of the joint angle. When the real-time image feature error is reduced to an acceptable range, the robotic arm stops moving.

2. The vision-based end-effector positioning method for a long flexible robotic arm according to claim 1, characterized in that, In step 1, before the fabric laying operation begins, the drone and its onboard camera are used to capture images of the target in the construction scene. After obtaining the pixel information of the target, the coordinates of the target point relative to the drone coordinate system are obtained based on the camera's focal length and the drone's flight altitude.

3. The vision-based end-effector positioning method for a long, flexible robotic arm according to claim 1, characterized in that, In step 1, the pixel coordinates of the corner points in the target image to be processed are obtained according to the contour detection method.

4. The vision-based end-effector positioning method for a long flexible robotic arm according to claim 1, characterized in that, In step 2, the motion trajectory of the robotic arm is obtained using a fifth-order polynomial interpolation method.

5. The vision-based end-effector positioning method for a long, flexible robotic arm according to claim 1, characterized in that, In step 4, the specific process of obtaining the desired pixel coordinates by combining the camera's intrinsic and extrinsic parameter matrices with the relative relationship between the robotic arm's end effector is as follows: (1) Calculate the transformation matrix between the end of the robotic arm and the gimbal camera based on the desired position of the end of the robotic arm, the angle of the last section of the robotic arm and the installation position of the gimbal camera; (2) Based on the location of the target point obtained in step 1, calculate the coordinates of the corner point of the target point in the gimbal coordinate system; (3) Based on the intrinsic parameter matrix of the binocular camera, the expected image features of the target corner point are obtained by combining the image Jacobian matrix.

6. The vision-based end-effector positioning method for a long, flexible robotic arm according to claim 1, characterized in that, In step 4, the field of view is adjusted in real time by the gimbal to ensure that the target remains within the effective imaging range, thereby continuously providing stable and reliable desired image features.

7. The vision-based end-effector positioning method for a long flexible robotic arm according to claim 1, characterized in that, In step 5, the image feature error is the difference between the real-time pixel coordinates and the expected pixel coordinates.

8. The vision-based end-effector positioning method for a long flexible robotic arm according to claim 3, characterized in that, Based on the positional relationship between the gimbal camera and the robotic arm's end effector, the required end effector speed to reach the desired position is calculated.

9. The vision-based end-effector positioning method for a long flexible robotic arm according to claim 1, characterized in that, Step 8: When the image feature error is less than the threshold, stop the movement of the robotic arm. The threshold is a preset tolerance.

10. A vision-based end-effector positioning system for a long, flexible robotic arm that implements the method of any one of claims 1-9, characterized in that, The system includes: The target coordinate acquisition module is used to obtain an image containing the target to be worked through the UAV, and convert the pixel coordinates of the corner points of the target to be worked in the image into position coordinates in the robotic arm coordinate system based on the positional relationship between the UAV and the robotic arm. The motion trajectory acquisition module is used to obtain the motion trajectory of the robotic arm based on the position of the corner point of the target to be worked, and to solve for the change value of the joint angle of the robotic arm. The pixel coordinate recognition module is used to detect and identify the real-time pixel coordinates of the corner points of the target to be operated based on the gimbal camera installed at the end of the robotic arm. The desired pixel coordinate solution module, based on the fixed assembly relationship between the gimbal camera and the end effector of the robotic arm, transforms the desired position of the end effector of the robotic arm into the camera coordinate system, and combines the camera's intrinsic and extrinsic parameters to complete the imaging projection, thereby obtaining the desired pixel coordinates of the desired position in the image plane. The feature error solving module is used to solve for the image feature error based on the pixel coordinates of the desired image features and the real-time pixel coordinates. The image Jacobian matrix module is used to obtain depth values ​​based on the binocular parallax principle and the camera intrinsic parameter matrix, and then to obtain the image Jacobian matrix by combining the depth values. The joint angle expected velocity calculation module is used for image-based visual servoing methods. It obtains the camera expected velocity required to achieve the desired image features based on the image feature error and the image Jacobian matrix. Based on the positional relationship between the gimbal camera and the robotic arm end effector, it calculates the robotic arm end effector velocity required to reach the desired position and obtains the joint angle expected velocity based on the robotic arm Jacobian matrix. The robotic arm end effector control module controls the movement of the robotic arm end effector based on the expected velocity of the joint angle. When the real-time image feature error is reduced to an acceptable range, the robotic arm stops moving.

Images