A robot vision servo object delivery method and system based on an Attention-LSTM neural network

Abstract

The application belongs to the technical field of robot control, and discloses a robot visual servo object delivery method and system based on an Attention-LSTM neural network. The method comprises the following steps: acquiring images in a hand movement process and identifying a hand in the images; determining the position of the hand in a robot coordinate system at different time points; inputting the hand positions at multiple time points before a to-be-predicted time point into an attention mechanism to generate corresponding context vectors at the time points; taking the hand position and the context vector at each time point as an input of an LSTM neural network to obtain an expected position of the hand at the to-be-predicted time point; controlling a robot end to move to the expected position of the hand at the to-be-predicted time point, and delivering an object carried by the robot end to the hand, so as to realize the object delivery process of the robot. Through the application, the problem of lag response of the robot to movement changes when tracking a moving object is solved.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, and more specifically, relates to a robot visual servoing object delivery method and system based on Attention-LSTM neural network. Background Technology

[0002] With the development of robotics technology, robots will be able to handle more complex tasks in dynamic environments. However, traditional robot control methods are mainly applied to known models or static environments. Improving the robot's understanding of the environment is an important direction to support the high-quality application of robots.

[0003] To improve the adaptability of robots in various complex environments, there is an increasing need to integrate vision technology into robot control. Position-based visual servoing systems use visual sensors such as cameras to receive desired position signals. By comparing the robot's current position and posture with given position and posture, they calculate the robot's control torque, thereby correcting the robot's position and posture. However, due to factors such as sensor response time and data processing, robots often experience a certain lag when tracking moving objects and struggle to respond promptly to changes in motion.

[0004] Therefore, in order to improve the transparency between humans and machines, a method is needed to predict the motion of moving objects and obtain their intentions in order to improve the timeliness of response to changes in motion. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a robot visual servoing object delivery method and system based on Attention-LSTM neural network, which solves the problem of robot lag response to motion changes when tracking moving objects.

[0006] To achieve the above objectives, according to one aspect of the present invention, a robot visual servoing object delivery method based on an Attention-LSTM neural network is provided, characterized in that the method includes the following steps:

[0007] Identify the human hand in the image to be processed at different times and determine the position of the human hand in the robot coordinate system at different times;

[0008] The sequence pair sequence model is input with the hand positions at multiple times before the time to be predicted (t) and outputs the expected hand position at the time to be predicted. The sequence pair sequence model includes an encoder, an attention mechanism, and a decoder. The encoder is input with the hand positions at multiple times before the time to be predicted, and its output is used as the input of the attention mechanism. The decoder is input with the output of the attention mechanism and the output of the decoder at time t-1, and its output is the expected hand position at the time to be predicted (t).

[0009] At the predicted time, the robot's end effector moves to the desired position of the human hand at that predicted time, and the object carried by the robot's end effector is handed over to the human hand, thereby realizing the robot's object handover process.

[0010] More preferably, PD control is used to control the robot end effector to move to the desired position of the human hand at the predicted moment.

[0011] More preferably, when using the PD control, the control force of the robot end effector is calculated using the following formula:

[0012]

[0013] e p =x d -x c

[0014]

[0015] in, This refers to the error term in the controller, which includes position error. and attitude error Two parts, x d x represents the expected position, i.e., the predicted manpower position. c Let r be the current position of the robot's end effector. ik (i∈{d,c},k∈{1,2},)3 represents the rotation matrix R i The k-th column, antisymmetric matrix

[0016] More preferably, the calculation formulas for Kp and Kd in the PD control are the same, as follows:

[0017]

[0018] in, This is the proportionality coefficient. k is the differential coefficient. imin and k imax All are pre-designed fixed values, a and b are constants, calculated by ensuring the continuity of the critical point, and d is the absolute distance from the human hand to the robot end effector.

[0019] More preferably, the calculation formulas for Kp and Kd in the PD control are as follows:

[0020]

[0021] Where, k imax and k imin These are the maximum and minimum values ​​of the proportionality coefficient or differential coefficient, respectively, d. max and d min These represent the maximum and minimum absolute distances from the human hand to the robot's end effector, respectively.

[0022] More preferably, both the decoder and encoder employ an LSTM neural network, and the output of the attention mechanism is a context vector.

[0023] More preferably, the context vector is calculated using the following formula:

[0024]

[0025] e tj =v T tanh(W a [h t-1 h j ]+b a )

[0026] Where, α tj For attention weights, h j (j = 1, 2, ..., t-1) represents the historical hidden state, e tj h represents the historical hidden state and the current hidden state. t-1 The correlation score, v T W is the weight vector. a Let b be a linear transformation matrix. a As a bias term, c t This is the generated context vector.

[0027] More preferably, the hand in the image is identified using the YOLOv5 model, which takes an image as input and outputs the pixel coordinates of the hand in the image.

[0028] More preferably, after obtaining the pixel coordinates of the person in the image, the pixel coordinates of the hand are converted into the coordinates of the hand in the robot's base coordinate system using a transformation matrix, wherein the transformation matrix is ​​as follows:

[0029]

[0030] Among them, R X T is a 3×3 rotation matrix. XIt is a 3×1 translation matrix, O X It is a 1×3 zero matrix.

[0031] According to another aspect of the present invention, a robot visual servoing object delivery system based on an Attention-LSTM neural network is provided. This system includes a hand recognition model, a coordinate calculation model, a target trajectory prediction model, and a force controller, wherein:

[0032] The hand recognition model is used to identify human hands in images and obtain the position of the hand in the pixel coordinate system;

[0033] The coordinate calculation module is used to convert the position of the human hand in the pixel coordinate system into the position of the human hand in the robot coordinate system;

[0034] The target trajectory prediction model is used to predict the expected position of the hand at the time of prediction.

[0035] Force controllers are used to control the robot end effector to move toward the desired position of the human hand at the next moment.

[0036] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0037] 1. This invention uses the hand pose at multiple moments during the object delivery process and combines it with a long short-term memory network based on the attention mechanism to predict the hand pose at the moment to be predicted. Compared with the existing technology of using a robot to track the hand, this invention can perceive the position of the hand at the moment to be predicted in advance, compensate for the lag in the robot tracking process, and improve the accuracy and timeliness of responding to changes in hand movement.

[0038] 2. This invention employs adaptive PD control, setting the error term to a fixed value, while the controller coefficient varies according to the distance between the human hand and the robot's end effector. This adaptive PD controller achieves smooth robot tracking of the target with a smaller controller coefficient during the initial task handover, while using a larger controller coefficient in the later stages of the task to improve the robot's accuracy in tracking the desired position.

[0039] 3. This invention achieves variable step size prediction of operator hand trajectory in human-machine collaboration by using a sequence-to-sequence approach in LSTM. Compared with traditional LSTM networks, the seq2seq model benefits from its encoder-decoder architecture, enabling flexibility in the length of input and output sequences. Attached Figure Description

[0040] Figure 1 This is a flowchart of a robot vision servoing object delivery method based on an Attention-LSTM neural network, constructed according to a preferred embodiment of the present invention.

[0041] Figure 2 This is a camera calibration scene diagram with the eye outside the hand according to a preferred embodiment of the present invention.

[0042] Figure 3 This is the recognition performance of the YOLOv5 algorithm according to a preferred embodiment of the present invention on the test set.

[0043] Figure 4 This is a position-based visual servo control according to a preferred embodiment of the present invention.

[0044] Figure 5 This is a trajectory cleaning result diagram according to a preferred embodiment of the present invention, wherein the red trajectory is the retained trajectory after cleaning, and the blue trajectory is the trajectory that was cleaned due to excessive curvature.

[0045] Figure 6 The image is a loss value image trained on a neural network for 1000 rounds according to a preferred embodiment of the present invention.

[0046] Figure 7 The variable step size Attention-LSTM and fixed step size Attention-LSTM models according to the preferred embodiment of the present invention are the range of changes in the predicted coordinates and robot coordinates along the x-axis when given an x-direction perturbation. Among them, (a) is the range of changes in the hand position, predicted position and robot position of the fixed step size Attention-LSTM model under x-direction perturbation; (b) is the range of changes in the hand position, predicted position and robot position of the variable step size Attention-LSTM model under x-direction perturbation.

[0047] Figure 8 This is a visual servo object delivery experiment process according to a preferred embodiment of the present invention, wherein (a) to (e) are snapshots of the actual delivery experiment within 0-5s.

[0048] Figure 9 The diagram shows the robot trajectory, human hand trajectory, and predicted human hand trajectory in a visual servoing object delivery experiment according to a preferred embodiment of the present invention, wherein (a), (b), and (c) are the human hand trajectory, predicted human hand trajectory, and robot trajectory at times t = 1s, 2s, and 3s, respectively. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0050] like Figure 1As shown, a robot vision servoing object delivery method based on Attention-LSTM neural network specifically includes the following steps:

[0051] (a) Camera calibration is performed using depth information from the RGB-D camera to obtain the transformation matrix from the camera coordinate system to the robot's base coordinate system. A custom YOLOv5 model is trained to identify targets, and the real-time position of the identified targets is obtained by combining the RGB-D camera data.

[0052] First, the intrinsic parameters of the RGB-D camera need to be calibrated to obtain more accurate visual information. The TsaiLenz hand-eye calibration algorithm is used to calibrate the hand on the eye. This represents the rotation matrix from coordinate system a to coordinate system b, where base is the robot's base coordinate system, cam is the camera coordinate system, and endn and caln are the robot's end effector coordinate system and calibration code coordinate system in the nth calibration pose, respectively. For example... Figure 2 As shown, in hand-eye calibration where the eye is outside the hand, the calibration code is placed close to the robot's end effector. Furthermore, we can obtain:

[0053]

[0054] Write it in the form AX = XB:

[0055]

[0056] In AX = XB, A, X, and B are all homogeneous transformation matrices, which can be decomposed into:

[0057]

[0058] Where R is a 3×3 rotation matrix, T is a 3×1 translation matrix, and O is a 1×3 zero matrix, AX = XB can be decomposed as follows:

[0059]

[0060] For a rotation matrix R, it can be viewed as a rotation about axis P. r =[n1,n2,n3] T Rotate by θ angle and redefine using the modified Rodriguez transform. Therefore, we can conclude that:

[0061]

[0062] in, Substituting the above equation into (R) A -I)·T X =R X ·T B -TA Calculate T X Thus, we can obtain matrix X, which is the transformation matrix from the robot's base coordinate system to the camera coordinate system, thereby completing the camera calibration with the eye outside the hand.

[0063] In addition to collecting images of the objects to be recognized, photographs of the objects in experimental environments are also needed to train a more accurate model. A total of 1200 images related to human hands were collected, including 1000 publicly available human hand image datasets and 200 photographs of human hands taken in experimental environments. The images were labeled using a calibration program, and YOLOv5 was selected as the pre-training weights. The model was trained for 300 epochs to obtain a custom YOLOv5 model. The recognition performance is as follows: Figure 3 As shown.

[0064] (b) Construct a Long Short-Term Memory (LSTM) network and collect trajectory datasets for training the neural network. Use the trained LSTM network model for target trajectory prediction, with the predicted target position as the desired position of the controller.

[0065] The seq2seq model's encoder-decoder is constructed using an LSTM network. This encoder-decoder architecture can better extract feature information from historical trajectories. It is trained using a three-layer LSTM structure, with 512 nodes in each layer. LSTM introduces forget gates, input gates, and output gates to address the problem of traditional RNN networks in handling long-range dependencies.

[0066] The forget gate determines which information is removed from the memory cell:

[0067] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0068] The input gate determines the extent to which input information is written into the memory cell:

[0069] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0070]

[0071] The results of the forget gate and the input gate are combined to update the state of the memory cell:

[0072]

[0073] The output gate determines the output information at the current time step and obtains the hidden state output:

[0074] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0075] h t =o t ·tanh(C t )

[0076] The encoder LSTM is used to process historical trajectories. Generate hidden state sequence The attention mechanism dynamically weights the input's historical trajectory, enhancing the model's focus on key features. First, an attention score is calculated for each historical hidden state. Calculation and current hidden state correlation score e tj :

[0077]

[0078] Then normalize it to attention weight α tj :

[0079]

[0080] Finally, the context vector c is generated. t :

[0081]

[0082] intermediate vector This is the hidden state of the encoder at the last time step.

[0083] The decoder is based on the context vector c t Hidden state from the previous moment Generate the current output:

[0084]

[0085] Finally, the predicted coordinates at time t are generated through linear transformation.

[0086]

[0087] Where W1 and W2 are the weight matrices in the linear transformation, and b1 and b2 are the bias vectors in the linear transformation, both of which are automatically learned through the model training process.

[0088] The encoder compresses the input sequence into an intermediate state, and the decoder generates the output step by step based on the intermediate state, thus supporting free variation in the length of the input and output sequences.

[0089] The motion trajectory of a human hand was captured using an RGB-D camera combined with the YOLOv5 algorithm. Three-dimensional trajectory data, totaling 3900 time steps over 130 seconds, was collected at a frequency of 30Hz and used as the training dataset for the neural network. Gaussian filtering was applied to preprocess the trajectory dataset to reduce jitter in the trajectory identified by the YOLOv5 algorithm; a Gaussian filter coefficient of sigma = 2 was used to obtain smoother trajectory data.

[0090] The network divides the time window, setting the window size to 10, representing trajectory data spanning 1 second. The time window is then scrolled forward in increments of one time step. Each time window's data is assigned to a target trajectory point at the desired prediction time step. Specifically, data t∈[T-m+1,T] is used as network input, and data t∈[T+1,T+n] is used as the network's expected output. The target position is predicted for the next n time steps using trajectory data from the past m time steps.

[0091] The time window trajectories are cleaned, and the curvature ρ of all points within each time window is calculated:

[0092]

[0093] If the curvature of 5% of the trajectory points within the time window is greater than the threshold ρ max If the value is 20, it will not be input into the network to reduce the impact of low-quality data on model accuracy. Taking a collected sinusoidal trajectory as an example, the trajectory cleaning effect is as follows: Figure 5 As shown.

[0094] Calculate the average distance error between the predicted trajectory and the actual trajectory, adjust the parameters, and perform multiple training iterations. While ensuring the training loss converges, minimize the error as much as possible, and save the model with the best training performance for online prediction of the target trajectory. The loss during neural network training is as follows: Figure 6 As shown.

[0095] (c) Design a robot vision servo PD controller to control the robot end effector to move to the desired position.

[0096] like Figure 4 As shown, the robot's visual servoing method uses position-based visual servoing, and the robot's driving force comes from the error. Where the position error e p =x d -x c The robot's desired posture is when the gripper is vertically downward and its movement direction is parallel to the y-axis of the robot's base coordinate system. Therefore, the posture error is... Where r ik(i∈{d,c},k∈{1,2,3}) represents the rotation matrix R. i The k-th column; antisymmetric matrix x d The location of the human hand is obtained through YOLO algorithm identification and localization. c The position of the robot's end effector is obtained by reading the robot's current state through ROS.

[0097] Calculate the end control force based on the error term. Among them, speed error The error e is obtained by differentiating it with respect to time. The angular velocity of attitude change is not considered here.

[0098] In human-robot collaborative tasks, it is necessary to ensure smooth robot movement to guarantee human safety. Furthermore, it is crucial to minimize the robot's tracking error to ensure successful object delivery. Therefore, the controller coefficient k... p k d In the design, when the distance *d* between the human hand and the robot is large, the controller coefficient is set to its minimum value; when *d* is small, the controller coefficient is set to its maximum value; when *d* is within this range, an inverse proportional function is used to achieve a smooth change in the controller coefficient, i.e.:

[0099]

[0100] Take d min =0.1m,d max =0.5m,

[0101]

[0102] Therefore, we can conclude that:

[0103]

[0104] The end-effector control force F is calculated from this, and then converted into the torque τ = J at each joint of the robot. T ·F, where J is the robot Jacobian matrix.

[0105] The present invention will be further described below with reference to specific embodiments.

[0106] The trained Attention-LSTM neural network model is read, and the position of the human hand is identified and located using an RGB-D camera combined with the YOLOv5 algorithm, capturing the hand's motion trajectory. When the collected data first fills a time window (10 time steps per second), the data within that window is input into the model for the first prediction. After each subsequent time step, the earliest data read in the time window is removed, and the data for the current time step is read in, then the new time window data is input into the model for prediction. The prediction step size is proportional to the distance between the robot's end effector and the human hand. As the task is submitted, the prediction step size continuously decreases. During the physical submission phase, a certain perturbation is used to test the robustness of the variable step size prediction model. The human hand is shaken in the x-direction to simulate the operator's fine-tuning during the physical submission phase. Fixed-step Attention-LSTM and variable-step Attention-LSTM are used to predict the hand's motion trajectory during the shaking. The range of motion in the x-direction between the predicted hand trajectory and the robot's motion trajectory under the two methods is compared. Figure 7 As shown in the figure, analysis of data within the upper and lower quartile ranges reveals that in the fixed-step-size prediction method, the robot's motion range increases by 3.446 cm compared to the human hand's motion range; while in the variable-step-size prediction method, the robot's motion range only increases by 0.072 cm compared to the human hand's motion range. This demonstrates that the proposed variable-step-size prediction method has strong anti-interference capability during the physical delivery stage.

[0107] A Franka Hand gripper is installed at the end effector of the Franka robotic arm. The small ball to be delivered is gripped on the gripper. The end effector position is set as the center point of the gripper, and the robotic arm is driven to move towards the predicted position of the human hand at a frequency of 200Hz. Figure 8 As shown in the diagram. When the human hand moves to the delivery area and the gripper stops above the human hand, the gripper is released, completing the object delivery task. The trajectory diagram during the task is shown in the diagram. Figure 9 As shown, the average error between the predicted trajectory and the actual trajectory during the entire mission was 0.0522m.

[0108] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robot visual servoing object delivery method based on Attention-LSTM neural network, characterized in that, The method includes the following steps: Identify the human hand in the image to be processed at different times and determine the position of the human hand in the robot coordinate system at different times; The sequence pair sequence model is input with the hand positions at multiple times before the time to be predicted (t) and outputs the expected hand position at the time to be predicted. The sequence pair sequence model includes an encoder, an attention mechanism, and a decoder. The encoder is input with the hand positions at multiple times before the time to be predicted, and its output is used as the input of the attention mechanism. The decoder is input with the output of the attention mechanism and the output of the decoder at time t-1, and its output is the expected hand position at the time to be predicted (t). At the predicted time, the robot's end effector is controlled to move to the desired position of the human hand at that predicted time, and the object carried by the robot's end effector is handed over to the human hand, thereby realizing the robot's object handover process. PD control is used to control the robot's end effector to move to the desired position of the human hand at the predicted time. When using the PD control, the control force of the robot end effector is calculated using the following formula: in, This refers to the error term in the controller, which includes position error. and attitude error Two parts, The expected position is the predicted manpower position. This represents the current position of the robot's end effector. Representing the rotation matrix The Columns, antisymmetric matrix ; Both the decoder and encoder employ LSTM neural networks, and the output of the attention mechanism is a context vector. The formula for calculating the context vector is as follows: in, For attention weights, To be hidden in history For historical hidden state and current hidden state The correlation score For the weight vector, It is a linear transformation matrix. For bias terms, This is the generated context vector.

2. The robot visual servoing object delivery method based on Attention-LSTM neural network as described in claim 1, characterized in that, The calculation formulas for Kp and Kd in the PD control are the same, as follows: in, This is the proportionality coefficient. The differential coefficients are... and All are pre-designed fixed values. , All are constants, calculated to ensure continuity at the critical point. The absolute distance from the human hand to the robot's end effector.

3. The robot visual servoing object delivery method based on Attention-LSTM neural network as described in claim 2, characterized in that, The calculation formulas for Kp and Kd in the PD control are as follows: in, and These are the maximum and minimum values ​​of the proportionality coefficient or differential coefficient, respectively. and These represent the maximum and minimum absolute distances from the human hand to the robot's end effector, respectively.

4. The robot visual servoing object delivery method based on Attention-LSTM neural network as described in claim 1, characterized in that, The hand in the image is identified using the YOLOv5 model, which takes an image as input and outputs the pixel coordinates of the hand in the image.

5. The robot visual servoing object delivery method based on Attention-LSTM neural network as described in claim 4, characterized in that, After obtaining the pixel coordinates of the person in the image, the pixel coordinates of the hand are converted into the coordinates of the hand in the robot's base coordinate system using a transformation matrix. The transformation matrix is ​​as follows: in, It is a 3×3 rotation matrix. It is a 3×1 translation matrix. It is a 1×3 zero matrix.

6. A robot visual servoing object delivery system based on Attention-LSTM neural network, characterized in that, The system employs a robot visual servoing object delivery method based on an Attention-LSTM neural network as described in any one of claims 1-5. The system includes a hand recognition model, a coordinate calculation model, a target trajectory prediction model, and a force controller, wherein: The hand recognition model is used to identify human hands in images and obtain the position of the hand in the pixel coordinate system; The coordinate calculation module is used to convert the position of the human hand in the pixel coordinate system into the position of the human hand in the robot coordinate system; The target trajectory prediction model is used to predict the expected position of the hand at the time of prediction. Force controllers are used to control the robot end effector to move toward the desired position of the human hand at the next moment.

Images