Method and device for grabbing dynamic strip-shaped object based on visual servo guidance control, and test platform

Abstract

The invention discloses a method and device for controlling grabbing of dynamic strips based on visual servo guidance and a test platform. According to the method for grabbing the dynamic strip-shaped object based on visual servo guidance control, the mechanical arm is guided and controlled to grab the dynamic strip-shaped object by adopting an image-based visual servo method, and real-time tracking of the dynamic pose of the strip-shaped object and immediate response of the action of the mechanical arm are realized through continuous sensing, processing and tight coupling of execution links; therefore, a more accurate, stable and efficient grabbing effect can be achieved in a highly dynamic and uncertain working environment; a test platform capable of simulating dynamic swing of a real strip-shaped object is also built, and a prediction control method and a visual servo control method are combined, so that effective verification and optimization of the high-precision and real-time grabbing process of the dynamic strip-shaped object are realized; the platform can be used for systematically researching the grabbing capability under various disturbance conditions in a laboratory environment, and a solid technical foundation is laid for automatic and intelligent operation in practical application.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a method and device for grasping a dynamic strip-shaped object based on visual servo guidance control, and a testing platform. Background Art

[0002] With the rapid development of the power industry and the continuous improvement of safety awareness, the operation and maintenance of power lines and fault handling have increasingly higher requirements for the professionalism and safety protection of operators. Live-line operation robots for distribution networks are gradually becoming the focus of industry attention and research hotspots: using robotic arms and other devices to replace manual labor to perform precision operations such as stripping, cutting and wiring in high-voltage and high-risk live environments can reduce human risks while improving work efficiency.

[0003] However, in practical applications, robots working with live wires still face many severe challenges. Among them, "cable grasping" is particularly critical as a prerequisite for the robot to perform subsequent fine work. Unlike static targets, cables exhibit significant dynamic characteristics during high-altitude laying. For example, when the robot is working, the target cable may swing periodically or irregularly due to the combined effects of wind, gravity and electromagnetic forces, and its amplitude and frequency are difficult to accurately predict. The dynamic nature of this flexible cable causes the spatial position of the grasping point to change continuously in a short period of time, which places extremely high demands on perception and control.

[0004] At present, most mainstream live-working robots adopt the "look-then-move" operation mode: first use vision or lidar and other sensors to collect cable position and posture information, and then perform offline or semi-offline trajectory planning. When the robot arm starts to move, since the cable has changed its posture and there is a delay in the perception data, this separate mode often makes it difficult to respond to the transient dynamics of the cable in time. Therefore, the traditional "look-then-move" strategy will be difficult to ensure the accuracy and stability of grasping due to perception lag and lack of online control.

[0005] In addition, the decoupling of the perception link and the control link also leads to insufficient overall intelligence: when environmental disturbances are frequent and nonlinear changes are obvious (such as sudden gusts of wind or slight shaking of the bucket arm), the robotic arm cannot quickly iterate a new optimal grasping trajectory within a time scale of tens of milliseconds, resulting in an increase in the grasping failure rate.

[0006] To address these issues, some studies have attempted to use high-precision 3D lidar, industrial-grade high-speed cameras, and GPU workstations to improve perception and processing speeds, but such solutions are expensive and difficult to deploy on a large scale in actual power operation and maintenance scenarios. Although low-cost sensing solutions (such as monocular vision and multi-sensor fusion) are economical and flexible, they are still incapable of high-speed dynamic target tracking and positioning.

[0007] At the same time, existing research platforms are mainly used to verify grasping performance under static or quasi-static conditions, and lack a test platform that can truly reproduce cable swing and nonlinear dynamic characteristics. This makes it difficult for R&D personnel to promptly discover and solve the failure mode of the algorithm in real complex scenarios in a laboratory environment, thereby extending the iteration cycle of the technology and increasing the difficulty of debugging in actual deployment.

[0008] In summary, current live working robots have the following problems in dynamic cable grabbing:

[0009] 1. Real-time dynamic perception is difficult: Existing perception systems have significant delays and lack of accuracy in tracking dynamic cables, making it difficult to obtain timely and high-precision posture feedback.

[0010] 2. Insufficient online control and planning capabilities: Traditional trajectory planning is difficult to meet real-time update requirements and cannot perform online adaptive control based on rapidly changing cable states, resulting in low grasping success rate and stability.

[0011] 3. The economic efficiency and feasibility of perception equipment are limited: high-precision perception hardware is expensive and requires high computing resources, which is not conducive to large-scale engineering applications.

[0012] 4. Lack of experimental verification platform for dynamic cable scenarios: The lack of experimental equipment that can truly simulate the nonlinear and random swing characteristics of cables makes it impossible to fully verify and optimize the effectiveness and robustness of related technologies during the research and development stage.

[0013] 5. Insufficient intelligence: The current system lacks the ability to quickly and autonomously adjust and optimize under unstructured dynamic conditions, making it difficult to achieve real-time response and adaptive control of complex vibration characteristics. Summary of the invention

[0014] The technical problem to be solved by the present invention is to provide a method and device for grasping dynamic strips based on visual servo guidance control, and a test platform, which can realize high-precision and real-time grasping operations on dynamic strips.

[0015] In order to solve the above technical problems, the first aspect of the present invention discloses a method for grasping a dynamic bar based on visual servo guidance control, which is applied to a robotic arm with a grasping function. The method comprises the following steps:

[0016] S1. Real-time acquisition of images of the target strip; the target strip is a dynamic strip that oscillates randomly;

[0017] S2. Identify the position of the target strip according to the image of the target strip and obtain the morphological characteristics of the target strip;

[0018] S3. Convert the two-dimensional image features of the target strip into three-dimensional spatial pose data, analyze and model the motion state of the target strip using the three-dimensional spatial pose data of the continuous time series, and predict the pose information of the target strip at the next moment;

[0019] S4. Based on the predicted position information of the target bar, a model predictive control method is used to design a global path planning of the robot arm and drive the robot arm according to the plan; in the model predictive control method, a future control input sequence is optimized at each time step;

[0020] S5. According to the real-time collected image of the target bar, the position deviation information of the target bar is calculated in real time, and the motion trajectory of the robot arm is dynamically adjusted by the image-based visual servo control law;

[0021] S6. When the end of the robot arm moves to the target position, a grabbing action is performed on the target bar.

[0022] As an optional implementation, in the first aspect of the present invention, the step S2. identifying the position of the target strip according to the image of the target strip and acquiring the morphological features of the target strip includes:

[0023] S21. Analyze the image of the target strip using the rotating target detection network to obtain the ROI region of the target strip;

[0024] S22. Preprocess the ROI area and extract the contour edge;

[0025] S23. The equation of the center line of the target strip in the image is obtained by the difference fitting method, y=ax+b, as the morphological feature of the target strip; wherein the fitting error is:

[0026]

[0027] Where E is the fitting error, y i and x i is the coordinate of the i-th point on the center line, a and b are the slope and intercept of the linear fit, and n is the number of sampling points.

[0028] As another optional implementation, in the first aspect of the present invention, after step S2 and before step S3, the method further includes:

[0029] Select N key points on the center line of the target strip in the image by evenly dividing the line segments;

[0030] And, the step S3. converts the two-dimensional image features of the target strip into three-dimensional spatial posture data, uses the three-dimensional spatial posture data of the continuous time series to analyze and model the motion state of the target strip, and predicts the posture information of the target strip at the next moment, including:

[0031] S31. The coordinates of each key point are converted into three-dimensional world coordinates by the following formula:

[0032]

[0033] Wherein, (u, v) is the coordinate of the key point in the image, (X, Y, Z) is the corresponding three-dimensional world coordinate, K is the camera intrinsic parameter matrix, R and t are the rotation and translation matrices of the camera; the camera is a device for collecting the image of the target bar;

[0034] S32. The motion state of the target bar is modeled and predicted by using an extended Kalman filter, and the formula includes:

[0035] System state equation:

[0036] x cable,k+1 =f(x cable,k ,u k )+w k ,

[0037] Observation equation:

[0038] z k =h(x cable,k )+v k ;

[0039] Among them, x cable,k represents the state vector of the target bar, u k is the control input (the joint angle of the robot arm or the speed of the end effector), w k and v k are process and observation noise, respectively, f(·) and h(·) are the state transfer function and observation function;

[0040] S33. Through the two steps of state prediction and observation update, combined with the historical posture data of the previous kt moments, the posture of the target cable at time k+t is predicted:

[0041]

[0042] in, is the state prediction at time k+1 based on the information at time k, K k is the Kalman gain, which is used to balance the weight of the prediction model and the observed data; P k|k-1 is the prediction error covariance, H kis the observation matrix, R k is the observation noise covariance.

[0043] As another optional embodiment, in the first aspect of the present invention, the step S4. designs a global path planning of the robot arm using a model predictive control method according to the predicted position and posture information of the target bar and drives the robot arm according to the plan; in the model predictive control method, optimizing the future control input sequence at each time step includes:

[0044] S41. Establish a discrete time state space model to describe the dynamic relationship between the robot arm and the target bar:

[0045] x arm,k+1 =Ax arm,k +Bu k +w k ;

[0046] Among them, x arm,k is the state vector at time step k, including the position and velocity of the end of the robot, u k is the control input (joint angle or end effector velocity) at time step k, A and B are system matrices; w k is the process noise, set to zero-mean Gaussian noise;

[0047] S42. Perform global path planning based on the cost function and constraints;

[0048] The cost function is:

[0049]

[0050] Among them, x cable,k is the predicted position of the target bar at time step k, Δx margin represents the control margin, Q, R and P are weight matrices used to balance path deviation and control energy consumption, and N is the prediction horizon;

[0051] The constraints include: motion constraints of the robot arm and control input constraints; the motion constraints of the robot arm include joint angle constraints and joint speed constraints, and the control input constraints are that the control signal must be within the physical limitations of the robot arm;

[0052] S43. At each time step, the optimal control input sequence U is solved by the following formula, and the control margin Δx is dynamically adjusted margin , to accommodate the nonlinear motion of the target strip:

[0053]

[0054] Where U is the control input sequence within the forecast horizon, and represent the feasible set of control and state respectively; v, a are the speed and acceleration of the target bar, t d is the system response delay time, φ(t d ) is the angle adjustment caused by response delay.

[0055] As another optional implementation, in the first aspect of the present invention, after step S3 and before step S5, the method further includes:

[0056] Based on the N key points in the morphological features of the target strip, and using the contour edge of the target strip extracted from the image in step S2, the pixel width d of the strip corresponding to the key points in the image is fitted. i ;

[0057] And the step S5. calculates the position deviation information of the target bar in real time according to the image of the target bar collected in real time, and dynamically adjusts the motion trajectory of the robot arm through the image-based visual servo control law, including:

[0058] S51. According to the morphological characteristics of the target strip, define the image feature vector s:

[0059] s=[u 1 ,v 1 ,d 1 ,u 2 ,v 2 ,d 2 ,…,u N ,v N ,d N ] T ;

[0060] Among them, (u i ,v i ) is the pixel coordinate of the i-th key point, d i is the pixel diameter of the bar corresponding to the i-th key point in the image;

[0061] S52. For each key point feature (u i ,v i ,d i ), design the interaction matrix L s :

[0062]

[0063] Among them, P i is the distance of the key point, (u i ,v i ) is the pixel coordinate of the i-th keypoint in the image plane, d iis the pixel width of the bar corresponding to the i-th key point in the image;

[0064] Interaction matrix L s Describes the rate of change of image features and the camera motion speed v c The relationship between:

[0065]

[0066] S53. Calculate the characteristic error e:

[0067] e=ss * ;

[0068] Among them, s * is the expected feature vector;

[0069] S54. According to the interaction matrix L s And characteristic error e, the designed control law is:

[0070]

[0071] Where λ>0 is the servo gain, is the pseudo-inverse of the interaction matrix, e is the characteristic error; v c =[v x ,v y ,v z ,ω x ,ω y ,ω z ] T represents the movement speed of the camera in three-dimensional space; the camera is fixed on the end of the robotic arm;

[0072] The goal of the control law is to make the characteristic error e smaller than a target threshold.

[0073] S55. v solved based on the control law c Adjust the speed of the end of the robot arm in real time.

[0074] As another optional implementation, in the first aspect of the present invention, the step S22. preprocessing the ROI area and extracting the contour edge includes:

[0075] The ROI area is grayed and Gaussian blurred to eliminate image noise, and the edge detection algorithm is used to extract the edge contour of the target strip.

[0076] As another optional implementation, in the first aspect of the present invention, the rotating object detection network calibrates the ROI area by outputting a rotating detection box including a rotation angle:

[0077] OBB=(x,y,w,h,θ);

[0078] Where (x, y) is the center coordinate of the rotation detection frame, w and h are the width and height of the rotation detection frame, and θ is the rotation angle.

[0079] The second aspect of the present invention discloses a test platform for testing a dynamic strip grabbing method, comprising a support frame, a strip swinging module, a strip fixing member, a strip and a strip clamping device;

[0080] The strip swing module is fixed to the first end of the support frame, and the strip fixing piece is fixed to the second end of the support frame; the first end of the strip is transmission-connected to the strip swing module, and the second end of the strip is connected to the strip fixing piece, so that the strip is tensioned and suspended on the support frame and the strip is driven by the strip swing module to swing;

[0081] The strip clamping device is located below the strip, and includes a strip clamping hand, a visual sensor, and a serially connected robotic arm; the strip clamping hand is disposed at the end of the robotic arm; the visual sensor is used to capture an image of the strip, and the robotic arm executes the method described in the first aspect of the present invention to clamp the strip.

[0082] As an optional implementation, in the second aspect of the present invention, the bar swing module supports periodic vibration, random vibration and combined vibration modes.

[0083] The third aspect of the present invention discloses a device for grasping dynamic strips based on visual servo guidance control, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the method disclosed in the first aspect of the present invention.

[0084] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method disclosed in the first aspect of the present invention.

[0085] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0086] Compared with the prior art, the embodiment of the present invention adopts the "Look-and-Move" working route, that is, it adopts the image-based visual servo method (Visual Servo Control) to guide and control the robot arm to grasp the dynamic strips; through the continuous and tight coupling of perception, processing and execution links, it realizes the real-time tracking of the dynamic posture of the strips and the immediate response of the robot arm's movements, so as to achieve a more accurate, stable and efficient grasping effect in a highly dynamic and uncertain working environment. By building a strip swing platform that can simulate a real dynamic environment, and combining predictive control and visual servo control methods, it realizes the effective verification and optimization of the high-precision, real-time grasping process of dynamic strips; the platform can conduct a systematic study of the grasping ability under various disturbance conditions in a laboratory environment, laying a solid technical foundation for automation and intelligent operations in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0088] Figure 1 It is a flow chart of a method for grasping a dynamic strip object based on visual servo guidance control disclosed in an embodiment of the present invention;

[0089] Figure 2 It is a schematic diagram of the principle of a method for grasping a dynamic strip based on visual servo guidance control disclosed in an embodiment of the present invention;

[0090] Figure 3 It is a structural schematic diagram of a test platform for testing a dynamic strip-shaped object grabbing method disclosed in an embodiment of the present invention;

[0091] Figure 4 It is another structural schematic diagram of a test platform for testing a dynamic strip-shaped object grabbing method disclosed in an embodiment of the present invention;

[0092] Figure 5 It is a structural schematic diagram of a device for grasping a dynamic bar-shaped object based on visual servo guidance control disclosed in an embodiment of the present invention;

[0093] Figure 6 It is a timing diagram of a robot grabbing a dynamic bar-shaped object disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0094] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0095] Embodiment 1

[0096] See also Figures 1-2 The embodiment of the present invention discloses a method for grasping a dynamic bar-shaped object based on visual servo guidance control, which is applied to a mechanical arm with a grasping function. The bar-shaped object may be a power cable or a branch of a crop. The method comprises the following steps:

[0097] Step S1. Collecting an image of a target bar in real time. The target bar is a dynamic bar that swings randomly.

[0098] In this embodiment, an RGB camera installed at the end of the robotic arm is used to collect image sequences of the dynamic strip. The camera needs to have a sufficiently high frame rate and resolution (e.g., more than 60 frames per second, resolution VGA or higher) to capture the transient changes of the strip. The collected image data serves as the basis for subsequent detection, modeling, and control.

[0099] Step S2: Identify the position of the target strip according to the image of the target strip, and obtain the morphological features of the target strip.

[0100] In an optional embodiment, the step S2. identifying the position of the target strip according to the image of the target strip and acquiring the morphological features of the target strip includes:

[0101] S21. Analyze the image of the target strip using the rotating target detection network to obtain the ROI region of the target strip;

[0102] S22. Preprocess the ROI area and extract the contour edge;

[0103] S23. The equation of the center line of the target strip in the image is obtained by the difference fitting method, y=ax+b, as the morphological feature of the target strip; wherein the fitting error is:

[0104]

[0105] Where E is the fitting error, y i and x i is the coordinate of the i-th point on the center line, a and b are the slope and intercept of the linear fit, and n is the number of sampling points.

[0106] Optionally, the target strip can be quickly identified and the ROI area can be extracted based on a target detection algorithm based on machine vision or deep learning, and the morphological features of the strip (such as center line, tilt angle, specific position marking points, etc.) can be obtained.

[0107] Step S3. Convert the two-dimensional image features of the target strip into three-dimensional spatial pose data, use the three-dimensional spatial pose data of the continuous time series to analyze and model the motion state of the target strip, and predict the pose information of the target strip at the next moment.

[0108] In yet another optional embodiment, after step S2 and before step S3, the method further includes:

[0109] N key points are selected on the center line of the target strip in the image by evenly dividing the line segments.

[0110] Assume that the starting and ending points of the center line of the target strip in the image are (x start ,y start ) and (x end ,y end ), then N key points can be selected by evenly dividing the line segments, and the coordinates of each key point can be expressed as:

[0111]

[0112] x i =x start +i·Δx,

[0113] y i =ax i +b(i=0,1,2,…,N-1)

[0114] Where Δx is the coordinate interval, (x i ,y i ) is the coordinate of the i-th key point on the center line.

[0115] In another optional embodiment, the step S3. converts the two-dimensional image features of the target strip into three-dimensional spatial pose data, uses the three-dimensional spatial pose data of the continuous time series, analyzes and models the motion state of the target strip, and predicts the pose information of the target strip at the next moment, including:

[0116] S31. The coordinates of each key point are converted into three-dimensional world coordinates by the following formula:

[0117]

[0118] Wherein, (u, v) is the coordinate of the key point in the image, (X, Y, Z) is the corresponding three-dimensional world coordinate, K is the camera intrinsic parameter matrix, R and t are the rotation and translation matrices of the camera; the camera is a device for collecting the image of the target bar;

[0119] The PNP algorithm uses the camera calibration information and the actual diameter of the bar to calculate the mapping relationship between the key points in the two-dimensional image and the coordinates of the points in the three-dimensional space.

[0120] S32. The motion state of the target bar is modeled and predicted by using an extended Kalman filter, and the formula includes:

[0121] System state equation:

[0122] x cable,k+1 =f(x cable,k ,u k )+w k ,

[0123] Observation equation:

[0124] z k =h(x cable,k )+v k ;

[0125] Among them, x cable,k represents the state vector of the target bar, u k is the control input, such as the joint angles of the robot arm or the velocity of the end effector, w k and v k are process and observation noise, respectively, f(·) and h(·) are the state transfer function and observation function;

[0126] S33. Through the two steps of state prediction and observation update, combined with the historical posture data of the previous kt moments, the posture of the target cable at time k+t is predicted:

[0127]

[0128] in, is the state prediction at time k+1 based on the information at time k, K k is the Kalman gain, which is used to balance the weight of the prediction model and the observed data; P k|k-1 is the prediction error covariance, H k is the observation matrix, R k is the observation noise covariance.

[0129] Based on the in-camera calibration information, the two-dimensional image features are converted into three-dimensional spatial pose data. The pose data of the continuous time series is used to analyze and model the motion state of the bar, and predict the possible position and pose of the bar at the next moment. Through predictive control, the motion path of the robot arm can be arranged in advance at the global planning level.

[0130] Step S4. Based on the predicted position information of the target bar, a model predictive control method is used to design a global path plan of the robot arm and drive the robot arm according to the plan. In the model predictive control method, a future control input sequence is optimized at each time step.

[0131] In another optional embodiment, the step S4. designs a global path planning of the robot arm using a model predictive control method according to the predicted position and posture information of the target bar and drives the robot arm according to the plan; in the model predictive control method, optimizing the future control input sequence at each time step includes:

[0132] S41. Establish a discrete time state space model to describe the dynamic relationship between the robot arm and the target bar:

[0133] x arm,k+1 =Ax arm,k +Bu k +w k ;

[0134] Among them, x arm,k is the state vector at time step k, including the position and velocity of the end of the robot, u k is the control input at time step k, such as joint angles or end effector velocities, A and B are system matrices, and w k is the process noise, set to zero-mean Gaussian noise;

[0135] S42. Perform global path planning based on the cost function and constraints;

[0136] The cost function is:

[0137]

[0138] Among them, x cable,k is the predicted position of the target bar at time step k, Δx margin represents the control margin, Q, R and P are weight matrices used to balance path deviation and control energy consumption, and N is the prediction horizon;

[0139] The constraints include: motion constraints of the robot arm and control input constraints; the motion constraints of the robot arm include joint angle constraints and joint speed constraints, and the control input constraints are that the control signal must be within the physical limitations of the robot arm, that is:

[0140]

[0141] S43. At each time step, the optimal control input sequence U is solved by the following formula, and the control margin Δx is dynamically adjusted margin , to accommodate the nonlinear motion of the target strip:

[0142]

[0143] Where U is the control input sequence within the forecast horizon, and represent the feasible set of control and state respectively; v, a are the speed and acceleration of the target bar, t d is the system response delay time, φ(t d ) is the angle adjustment caused by response delay.

[0144] The control margin is calculated based on the velocity, acceleration and system response delay of the strip, while an explicit penalty term is added to the cost function. Ensure the feasibility and robustness of the planned path.

[0145] In this embodiment, the model predictive control (MPC) method is used to perform global path planning of the robot arm based on the predicted bar pose information. The MPC method aims to minimize a cost function, which is used to evaluate the quality of the path. Through MPC, the system can optimize the future control input sequence at each time step, so that the robot arm can efficiently and safely track the dynamic motion of the target bar. In order to further improve the robustness of the grasping process, a control margin is introduced in the path planning to ensure that the end of the robot arm can be adjusted in advance to the vicinity of the position where the target bar may appear in the future.

[0146] Step S5. According to the real-time collected image of the target bar, the position deviation information of the target bar is calculated in real time, and the motion trajectory of the robot arm is dynamically adjusted through the image-based visual servo control law.

[0147] In yet another optional embodiment, after step S3 and before step S5, the method further includes:

[0148] Based on the N key points in the morphological features of the target strip, and using the contour edge of the target strip extracted from the image in step S2, the pixel width d of the strip corresponding to the key points in the image is fitted. i .

[0149] In this embodiment, the pixel diameter d iThe calculation is specifically calculated from the pixel coordinates in the strip contour: by extracting two strip contour edges (such as the upper edge and the lower edge), the Euclidean distance between the two points (the pixel point of the upper edge and the pixel point of the lower edge) is calculated to obtain the pixel diameter of the key point.

[0150] In another optional embodiment, the step S5. calculates the position deviation information of the target bar in real time according to the image of the target bar collected in real time, and dynamically adjusts the motion trajectory of the robot arm through the image-based visual servo control law, including:

[0151] S51. According to the morphological characteristics of the target strip, define the image feature vector s:

[0152] s=[u 1 ,v 1 ,d 1 ,u 2 ,v 2 ,d 2 ,…,u N ,v N ,d N ] T ;

[0153] Among them, (u i ,v i ) is the pixel coordinate of the i-th key point, d i is the pixel diameter of the bar corresponding to the i-th key point in the image.

[0154] By introducing the pixel diameter d of the strip as an additional visual feature, the geometric characteristics of the strip can be described more accurately. The introduction of this feature provides an important reference for the design of the interaction matrix of visual servoing, thereby enhancing the sensitivity of visual servoing to target features and further improving the accuracy and stability of the robot arm motion control.

[0155] S52. For each key point feature (u i ,v i ,d i ), design the interaction matrix L s :

[0156]

[0157] Among them, P i is the distance of the key point, (u i ,v i ) is the pixel coordinate of the i-th keypoint in the image plane, d i is the pixel width of the bar corresponding to the i-th key point in the image;

[0158] Interaction matrix Ls Describes the rate of change of image features and the camera motion speed v c The relationship between:

[0159]

[0160] The first two rows in the figure are related to the image position change of the key points, while the third row introduces the pixel diameter of the bar to enhance the sensitivity of the visual servo system to the target features, thereby better guiding the robot arm to adjust its posture and accurately track the target bar. Overall interaction matrix is the stack of all keypoint interaction matrices.

[0161] The distance of a key point refers to the real-time distance of a point on the target bar in the three-dimensional space corresponding to the key point in the image relative to the camera in the camera coordinate system.

[0162] S53. Calculate the characteristic error e:

[0163] e=ss * ;

[0164] Among them, s * is the expected feature vector;

[0165] S54. According to the interaction matrix L s And characteristic error e, the designed control law is:

[0166]

[0167] Where λ>0 is the servo gain, is the pseudo-inverse of the interaction matrix, e is the characteristic error; v c =[v x ,v y ,v z ,ω x ,ω y ,ω z ] T represents the movement speed of the camera in three-dimensional space; the camera is fixed on the end of the robotic arm;

[0168] The goal of the control law is to make the characteristic error e smaller than a target threshold.

[0169] When the feature error is less than a certain threshold (e.g., e<20), it can be considered that the robot arm posture converges to the target position.

[0170] Due to the introduction of the pixel diameter of the strip, the characteristic error e not only includes the error of the center point in the image plane, but also includes the error of the diameter of the target strip, thus providing more comprehensive feedback information for the calculation of the control law.

[0171] S55. v solved based on the control law c Adjust the speed of the end of the robot arm in real time.

[0172] v c =[v x ,v y ,v z ,ω x ,ω y ,ω z ] T It represents the movement speed of the camera in three-dimensional space. Through the calibrated hand-eye matrix, the movement amount of the end effector of the robot arm can be calculated.

[0173] In this embodiment, when the robot arm approaches the grasping point of the target bar, it enters the visual servo stage. Using real-time image feedback, the position deviation information of the target bar is calculated, and the motion trajectory of the robot arm is dynamically adjusted through the image-based visual servoing (IBVS) control law. In this stage, the system can ensure that the end of the robot arm is always aligned with the actual grasping point of the bar, further improving the success rate and robustness of the grasping.

[0174] Step S6: When the end of the robot arm moves to the target position, a grabbing action for the target bar is performed.

[0175] In this embodiment, when the end of the robot arm is in a reasonable grasping position and the deviation is fully corrected, the final clamping or hooking action is performed to complete the stable grasping of the dynamic strip. For power cables, this step provides a basic guarantee for subsequent fine operations such as stripping, cutting, and wiring. For the branches of crops, after the robot arm grasps the branches of crops, it can be cut to obtain the fruit.

[0176] Steps S1 to S5 are a cyclic process. When the feature error calculated in S5 is less than a certain threshold (for example, the error is less than 20), it can be regarded as the robot arm has moved into place, that is, it has reached the desired position, and the grasping action of S6 can be started.

[0177] Since the bar is in a dynamic motion state, if a rigid connection is established after the robot arm grabs the bar, it may cause damage to the joints of the robot arm and shorten its service life. Therefore, in order to avoid this problem, after the end of the robot arm is in a reasonable grabbing position, the present invention also needs to fully correct the posture deviation information, and then grab the bar to achieve a flexible connection. Through the force feedback system (such as torque sensor, current feedback, etc.) on the robot arm, the posture of the robot arm is adjusted in real time to achieve a follow-up effect between the robot arm and the bar, and to build a flexible connection. In this way, not only can the joints of the robot arm be protected, but also the posture data of the bar can be obtained more accurately.

[0178] In yet another optional embodiment, the step S22. preprocessing the ROI region and extracting contour edges includes:

[0179] The ROI area is grayed and Gaussian blurred to eliminate image noise, and the edge detection algorithm is used to extract the edge contour of the target strip.

[0180] In yet another optional embodiment, the rotating object detection network calibrates the ROI area by outputting a rotating detection box including a rotation angle:

[0181] OBB=(x,y,w,h,θ);

[0182] Where (x, y) is the center coordinate of the rotation detection frame, w and h are the width and height of the rotation detection frame, and θ is the rotation angle.

[0183] Compared with the traditional horizontal detection frame, the rotating detection frame has the following advantages:

[0184] 1. Accuracy: Wrap the target strip more tightly, effectively reducing the interference of background on the detected object;

[0185] 2. Discrimination: There is almost no overlap between detection frames, which helps to clearly distinguish multiple targets;

[0186] 3. Directionality: The object’s motion direction information is roughly obtained from the rotating detection frame, thereby determining the object’s motion trajectory.

[0187] Embodiment 2

[0188] See also Figures 3-4 The embodiment of the present invention discloses a test platform for testing a dynamic strip grabbing method, comprising a support frame 1, a strip swinging module 2, a strip fixing member 3, a strip 4 and a strip clamping device 5; the strip 4 can be a power cable or a branch of a crop;

[0189] The strip swing module 2 is fixed to the first end of the support frame 1, and the strip fixing member 3 is fixed to the second end of the support frame 1; the first end of the strip 4 is transmission-connected to the strip swing module 2, and the second end of the strip 4 is connected to the strip fixing member 3, so that the strip 4 is tensioned and suspended on the support frame 1 and the strip 4 is driven by the strip swing module 2 to swing;

[0190] The strip clamping device 5 is located below the strip 4, and includes a strip clamping hand 51, a visual sensor 52, and a serially connected robotic arm 53; the strip clamping hand 51 is disposed at the end of the robotic arm 53, and the robotic arm 53 moves the strip clamping hand 51 based on the visual sensor 52 and controls the strip clamping hand 51 to clamp the strip 4.

[0191] In this embodiment, the frame support structure is assembled by bolting or welding multiple metal profiles to form a stable overall bracket. The frame can adjust the height and inclination according to the experimental requirements to adapt to different experimental environments and strip laying methods. The bottom of the frame adopts a triangular support structure to ensure the overall mechanical strength and stability.

[0192] In an optional embodiment, the strip oscillating module 2 supports periodic vibration (such as sine wave), random vibration (such as vibration simulating wind force, nonlinear disturbance) and combined vibration mode (periodic vibration+random vibration).

[0193] Optionally, the bar swing module is controlled by a controller, which has a closed-loop control function to achieve high-precision adjustment of vibration parameters (such as amplitude and frequency), with the amplitude error controlled within ±1mm and the frequency error less than ±0.1Hz. It can simulate various experimental scenarios such as tiny jitter, strong interference vibration and complex random disturbance to verify the robustness and adaptability of the bar grabbing algorithm under different conditions.

[0194] In another optional embodiment, it further includes an insulator 7, a bowl head hanging plate 8 and a tension clamp 9;

[0195] The fixed end of the insulator 7 is transmission-connected to the strip swing module 2 ; the first end of the strip 4 is mounted on the tension clamp 9 , and the tension clamp 9 is connected to the insulating end of the insulator 7 through the bowl head hanging plate 8 .

[0196] In yet another optional embodiment, the bar swing module 2 includes a guide rail 22 with a slider 21 and a servo motor 23 driving the slider 21 to move; the fixed end of the insulator 7 is transmission-connected to the slider 21 .

[0197] Optionally, the vibration frequency range of the strip swing module is 0.1Hz to 100Hz, and the amplitude range is 0.5mm to 100mm, that is, the reciprocating swing distance of the slider is 0.5mm to 100mm, and the swing frequency of the slider is 0.1Hz to 100Hz. It can realize flexible adjustment of parameters such as vibration amplitude and frequency to simulate the common dynamic disturbance scenarios in actual distribution network live operation. Through precise vibration control, this module provides efficient and repeatable experimental conditions for algorithm optimization and verification of dynamic strip grabbing system.

[0198] In another optional embodiment, the bar swing module 2 further includes a photoelectric sensor 24, which is disposed on the guide rail 22 and is used to obtain the position of the slider 21 in real time. The photoelectric sensor is combined with the controller to achieve real-time limit protection and ensure the safety of the experimental process.

[0199] In another optional embodiment, a servo motor driver (not shown in the drawings) is further included, a ball screw module is provided in the guide rail 22, the ball screw module is driven by the servo motor 23, and the ball screw module is transmission-connected to the slider 21; the servo motor driver is used to control the servo motor 23.

[0200] In yet another optional embodiment, a strip-shaped fixing adapter 10 is further included; one end of the strip-shaped fixing adapter 10 is fixed to the slider 21 , and the other end thereof is hinged to the fixed end of the insulator 7 .

[0201] In another optional embodiment, two fixing strips 6 are further included; the strip fixing member 3 is also fixed to the first end of the support frame 1, and each of the fixing strips 6 is fixed to the strip fixing member 3 at both ends of the support frame 1 and is located on both sides of the strip 4.

[0202] In another optional embodiment, the strip fixing member 3 is in the shape of a long strip, and fixing portions are provided at intervals on the strip fixing member 3 , and each fixing portion is hinged to one end of one of the fixing strips 6 .

[0203] In yet another optional embodiment, the strip clamping device 5 further includes a robot arm base 54 and a controller; the robot arm 53 is mounted on the robot arm base 54, and the robot arm 53 and the strip clamping hand 51 are driven and controlled by the controller.

[0204] In the embodiment of the present invention, by replacing the strip fixing member 3, the insulator 7, the bowl head hanging plate 8, the tension clamp 9 and the strip 4, the platform can simulate the actual operating conditions of various power lines in a laboratory environment.

[0205] The embodiment of the present invention builds a strip swinging platform that can simulate a real dynamic environment, and combines the predictive control and visual servo control methods in the first embodiment to achieve effective verification and optimization of the high-precision, real-time grasping process of dynamic strips. The platform and the method in the first embodiment can conduct a systematic study on the grasping ability under various disturbance conditions in a laboratory environment, and at the same time provide an intelligent, low-latency solution for power operation and maintenance operations in complex dynamic scenes, thereby laying a solid technical foundation for automation and intelligent operations in practical applications.

[0206] Embodiment 3

[0207] See also Figure 5 , Figure 5 It is a structural schematic diagram of a device for grabbing dynamic strips based on visual servo guidance control disclosed in an embodiment of the present invention, including a memory 201, a processor 202 and a computer program stored in the memory, and the processor executes the computer program to implement the steps in the method disclosed in Example 1.

[0208] Embodiment 4

[0209] An embodiment of the present invention discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method disclosed in the first embodiment are implemented.

[0210] Embodiment 5

[0211] The embodiment of the present invention discloses a robot capable of grasping a dynamic bar, comprising the camera and the mechanical arm in the first embodiment, and further comprising a controller, wherein the camera is communicatively connected to the controller. The camera executes step S1 in the method disclosed in the first embodiment, and the controller executes steps S2 to S5 in the method disclosed in the first embodiment.

[0212] See also Figure 6 , Figure 6 Including the main time-consuming distribution of target detection, pose estimation, path planning and servo guidance control. By optimizing the detection algorithm, path planning algorithm and servo control algorithm, the embodiment of the present invention controls the system control cycle of the robot arm within 20ms, which can meet the real-time response requirements in a high dynamic environment.

[0213] Embodiment 6

[0214] The embodiment of the present invention discloses another robot capable of grasping dynamic bars, comprising the camera and the mechanical arm in the first embodiment, and also comprising a host computer and a controller, wherein the camera is communicatively connected with the controller and the host computer.

[0215] The camera executes step S1 of the method disclosed in the first embodiment, the host computer executes steps S2 to S4 of the method disclosed in the first embodiment, and the controller executes step S5 of the method disclosed in the first embodiment. After step S2, the morphological features of the strip extracted by the host computer will be transmitted to the controller in real time through the communication protocol for visual servo guidance control, thereby realizing the coordinated control of the host computer and the controller. In step S5, the host computer obtains the visual feature information of the target strip in real time and transmits it to the controller through the real-time transmission protocol.

[0216] The contents disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for grasping a dynamic bar based on visual servo guidance control, applied to a robotic arm with grasping function, characterized in that: The method comprises the following steps: S1. Real-time acquisition of images of the target strip; the target strip is a dynamic strip that oscillates randomly; S2. Identify the position of the target strip according to the image of the target strip and obtain the morphological characteristics of the target strip; S3. Convert the two-dimensional image features of the target strip into three-dimensional spatial pose data, analyze and model the motion state of the target strip using the three-dimensional spatial pose data of the continuous time series, and predict the pose information of the target strip at the next moment; S4. Based on the predicted position information of the target bar, a model predictive control method is used to design a global path planning of the robot arm and drive the robot arm according to the plan; in the model predictive control method, a future control input sequence is optimized at each time step; S5. According to the real-time collected image of the target bar, the position deviation information of the target bar is calculated in real time, and the motion trajectory of the robot arm is dynamically adjusted by the image-based visual servo control law; S6. When the end of the robot arm moves to the target position, a grabbing action is performed on the target bar.

2. The method according to claim 1, characterized in that The step S2. identifying the position of the target strip according to the image of the target strip and acquiring the morphological features of the target strip, comprises: S21. Analyze the image of the target strip using the rotating target detection network to obtain the ROI area of ​​the target strip; S22. Preprocess the ROI area and extract the contour edge; S23. The equation of the center line of the target strip in the image is obtained by the difference fitting method, y=ax+b, as the morphological feature of the target strip; wherein the fitting error is: Where E is the fitting error, y i and x i is the coordinate of the i-th point on the center line, a and b are the slope and intercept of the linear fit, and n is the number of sampling points.

3. The method according to claim 2, characterized in that After step S2 and before step S3, the method further includes: Select N key points on the center line of the target strip in the image by evenly dividing the line segments; And, the step S3. converts the two-dimensional image features of the target strip into three-dimensional spatial posture data, uses the three-dimensional spatial posture data of the continuous time series to analyze and model the motion state of the target strip, and predicts the posture information of the target strip at the next moment, including: S31. The coordinates of each key point are converted into three-dimensional world coordinates by the following formula: Wherein, (u, v) is the coordinate of the key point in the image, (X, Y, Z) is the corresponding three-dimensional world coordinate, K is the camera intrinsic parameter matrix, R and t are the rotation and translation matrices of the camera; the camera is a device for collecting the image of the target bar; S32. The motion state of the target bar is modeled and predicted by using an extended Kalman filter, and the formula includes: System state equation: x cable,k+1 =f(x cable,k ,u k )+w k , Observation equation: z k =h(x cable,k )+v k ; Among them, x cable,k represents the state vector of the target bar, u k is the control input, w k and v k are process and observation noise, respectively, f(·) and h(·) are the state transfer function and observation function; S33. Through the two steps of state prediction and observation update, combined with the historical posture data of the previous kt moments, the posture of the target cable at time k+t is predicted: in, is the state prediction at time k+1 based on the information at time k, K k is the Kalman gain, which is used to balance the weight of the prediction model and the observed data; P k|k-1 is the prediction error covariance, H k is the observation matrix, R k is the observation noise covariance.

4. The method according to claim 1, characterized in that: The step S4. according to the predicted position information of the target bar, a global path planning of the robot arm is designed by using a model predictive control method and the robot arm is driven according to the plan; In this model predictive control approach, the future control input sequence is optimized at each time step, including: S41. Establish a discrete time state space model to describe the dynamic relationship between the robot arm and the target bar: x arm,k+1 =Ax arm,k +Bu k +w k ; Among them, x arm,k is the state vector at time step k, including the position and velocity of the end of the robot, u k is the control input at time step k, A and B are the system matrices, and w k is the process noise; S42. Perform global path planning based on the cost function and constraints; The cost function is: Among them, x cable,k is the predicted position of the target bar at time step k, Δx margin represents the control margin, Q, R and P are weight matrices used to balance path deviation and control energy consumption, and N is the prediction horizon; The constraints include: motion constraints of the robot arm and control input constraints; the motion constraints of the robot arm include joint angle constraints and joint speed constraints, and the control input constraints are that the control signal must be within the physical limitations of the robot arm; S43. At each time step, the optimal control input sequence U is solved by the following formula, and the control margin Δx is dynamically adjusted margin , to accommodate the nonlinear motion of the target strip: Where U is the control input sequence within the forecast horizon, and represent the feasible set of control and state respectively; v, a are the speed and acceleration of the target bar, t d is the system response delay time, φ(t d ) is the angle adjustment caused by response delay.

5. The method according to claim 3, characterized in that: After step S3 and before step S5, the method further includes: Based on the N key points in the morphological features of the target strip, and using the contour edge of the target strip extracted from the image in step S2, the pixel width d of the strip corresponding to the key points in the image is fitted. i ; And the step S5. calculates the position deviation information of the target bar in real time according to the image of the target bar collected in real time, and dynamically adjusts the motion trajectory of the robot arm through the image-based visual servo control law, including: S51. According to the morphological characteristics of the target strip, define the image feature vector s: <h2 style=";text-align:left;direction:ltr">s = [u1,v1,d1,u2,v2,d2,…,u<h2 style=";text-align:left;direction:ltr"> N <h2 style=";text-align:left;direction:ltr"> ,v<h2 style=";text-align:left;direction:ltr"> N <h2 style=";text-align:left;direction:ltr"> ,d<h2 style=";text-align:left;direction:ltr"> N <h2 style=";text-align:left;direction:ltr"> ]<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> ; Among them, (u i ,v i ) is the pixel coordinate of the i-th key point, d i is the pixel diameter of the bar corresponding to the i-th key point in the image; S52. For each key point feature (u i ,v i ,d i ), design the interaction matrix L s : Among them, P i is the distance of the key point, (u i ,v i ) is the pixel coordinate of the i-th keypoint in the image plane, d i is the pixel width of the bar corresponding to the i-th key point in the image; Interaction matrix L s Describes the rate of change of image features and the camera motion speed v c The relationship between: S53. Calculate the characteristic error e: Among them, s * is the expected feature vector; S54. According to the interaction matrix L s And characteristic error e, the designed control law is: Where λ>0 is the servo gain, is the pseudo-inverse of the interaction matrix, e is the characteristic error; v c =[v x ,v y ,v z ,ω x ,ω y ,ω z ] T represents the movement speed of the camera in three-dimensional space; the camera is fixed on the end of the robotic arm; The goal of the control law is to make the characteristic error e less than the target threshold; S55. v solved based on the control law c Adjust the speed of the end of the robot arm in real time.

6. The method according to claim 2, characterized in that The step S22. preprocessing the ROI area and extracting the contour edge includes: The ROI area is grayed and Gaussian blurred to eliminate image noise, and the edge detection algorithm is used to extract the edge contour of the target strip.

7. The method according to claim 2, characterized in that The rotating object detection network calibrates the ROI area by outputting a rotating detection box containing the rotation angle: OBB=(x,y,w,h,θ); Where (x, y) is the center coordinate of the rotation detection frame, w and h are the width and height of the rotation detection frame, and θ is the rotation angle.

8. A test platform for testing a dynamic bar grabbing method, characterized in that: It includes a support frame, a strip swing module, a strip fixing piece, a strip and a strip clamping device; The strip swing module is fixed to the first end of the support frame, and the strip fixing piece is fixed to the second end of the support frame; the first end of the strip is transmission-connected to the strip swing module, and the second end of the strip is connected to the strip fixing piece, so that the strip is tensioned and suspended on the support frame and the strip is driven by the strip swing module to swing; The strip clamping device is located below the strip, and includes a strip clamping hand, a visual sensor, and a serial-type robotic arm; the strip clamping hand is arranged at the end of the robotic arm; the visual sensor is used to capture the image of the strip, and the robotic arm executes the method described in any one of claims 1 to 7 to clamp the strip.

9. The test platform according to claim 8, characterized in that: The bar oscillation module supports periodic vibration, random vibration and combined vibration modes.

10. A device for grasping a dynamic bar-shaped object based on visual servo guidance control, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

Images