Robot visual servo object submission method and system based on Attention-LSTM neural network

Abstract

The invention belongs to the technical field related to robot control, and discloses a robot visual servo object submission method and system based on an Attention-LSTM neural network. The method comprises the following steps: acquiring an image in a human hand movement process and identifying a human hand in the image; determining the positions of the hands in the coordinate system of the robot at different moments; inputting the human hand positions at multiple moments before the to-be-predicted moment into an attention mechanism to generate context vectors corresponding to the moments, and taking the human hand position and the context vector at each moment as input of an LSTM neural network to obtain an expected position of the human hand at the to-be-predicted moment; and at the to-be-predicted moment, the robot tail end is controlled to move to the expected position of the human hand at the to-be-predicted moment, and the object carried by the robot tail end is controlled to be submitted to the human hand, so that the object submission process of the robot is realized. According to the invention, the problem of hysteresis response to motion change when the robot tracks the moving object is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to robot control, and more specifically, relates to a robot visual servo object delivery method and system based on an Attention-LSTM neural network. Background Art

[0002] With the development of robotics technology, robots will cope with more complex tasks in dynamic environments. However, traditional robot control methods are mainly used in known models or static environments. Improving robots' understanding of the environment is an important direction to support high-quality application of robots.

[0003] To improve robots' adaptability in complex environments, there's a growing need to integrate vision technology into robot control. Position-based visual servo systems use visual sensors such as cameras to receive desired position signals. By comparing the robot's current position and posture with a given position and posture, they calculate the robot's control torque and correct the robot's position and posture. However, due to factors such as sensor response time and data processing, robots often experience a certain lag when tracking moving objects, making it difficult to respond promptly to changes in motion.

[0004] Therefore, in order to improve the transparency between humans and machines, a method is needed to predict the movement of moving objects and obtain their intentions to improve the timeliness of responses to motion changes. Summary of the Invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a robot visual servo object delivery method and system based on an Attention-LSTM neural network to solve the problem of delayed response of the robot to motion changes when tracking a moving object.

[0006] To achieve the above object, according to one aspect of the present invention, a robot visual servoing object delivery method based on an Attention-LSTM neural network is provided, characterized in that the method comprises the following steps:

[0007] Identify the human hand in the image to be processed at different times and determine the position of the human hand in the robot coordinate system at different times;

[0008] The hand positions at multiple moments before the time t to be predicted are input into a sequence-to-sequence model, which outputs the expected hand position at the time to be predicted. The sequence-to-sequence model includes an encoder, an attention mechanism, and a decoder. The encoder input is the hand positions at multiple moments before the time to be predicted, and the encoder output serves as the input of the attention mechanism. The decoder input is the output of the attention mechanism and the output of the decoder at time t-1, and the output is the expected hand position at the time to be predicted.

[0009] At the predicted moment, the robot's end is controlled to move to the expected position of the human hand at the predicted moment, and the object carried by the robot's end is delivered to the human hand, thereby realizing the robot's object delivery process.

[0010] Further preferably, PD control is used to control the movement of the robot end to the desired position of the human hand at the predicted moment.

[0011] Further preferably, when the PD control is adopted, the control force of the robot end is calculated using the following formula:

[0012]

[0013] e p =x d -x c

[0014]

[0015] in, is the error term in the controller, including position error and attitude error Two parts, x d is the expected position, i.e. the predicted hand position, x c is the current position of the robot end, r ik (i∈{d,c},k∈{1,2},)3 represents the rotation matrix R i The kth column of the antisymmetric matrix

[0016] Further preferably, the calculation formulas of Kp and Kd in the PD control are the same, as follows:

[0017]

[0018] in, is the proportionality coefficient, is the differential coefficient, k imin and k imax are all pre-designed fixed values, a and b are constants, which are calculated by ensuring the continuity of critical points, and d is the absolute distance from the human hand to the end of the robot.

[0019] Further preferably, the calculation formulas of Kp and Kd in the PD control are as follows:

[0020]

[0021] Among them, k imax and k imin are the maximum and minimum values ​​of the proportional coefficient or differential coefficient, respectively, max and d min are the maximum and minimum absolute distances from the human hand to the end of the robot, respectively.

[0022] Further preferably, both the decoder and the encoder adopt LSTM neural networks, and the output of the attention mechanism is a context vector.

[0023] Further preferably, the calculation formula of the context vector is as follows:

[0024]

[0025] e tj =v T tanh(W a [h t-1 ;h j ]+b a )

[0026] Among them, α tj is the attention weight, h j (j=1,2,...,t-1) is the historical hidden state, e tj is the historical hidden state and the current hidden state h t-1 The correlation score, v T is the weight vector, W a is the linear transformation matrix, b a is the bias term, c t is the generated context vector.

[0027] Further preferably, the recognition of human hands in the image adopts a YOLOv5 model, the input of the model is an image, and the output is the pixel coordinates of the human hands in the image.

[0028] Further preferably, after obtaining the pixel coordinates of the human hand in the image, the pixel coordinates of the human hand are converted into the coordinates of the human hand in the robot base coordinate system using a transformation matrix, and the transformation matrix is ​​as follows:

[0029]

[0030] Among them, R X is a 3×3 rotation matrix, T Xis a 3×1 translation matrix, O X is a 1×3 zero matrix.

[0031] According to another aspect of the present invention, a robot visual servo object delivery system based on an Attention-LSTM neural network is provided. The system includes a human hand recognition model, a coordinate calculation model, a target trajectory prediction model, and a force controller, wherein:

[0032] The hand recognition model is used to identify the hand in the image and obtain the position of the hand in the pixel coordinate system;

[0033] The coordinate calculation module is used to convert the position of the human hand in the pixel coordinate system into the position of the human hand in the robot coordinate system;

[0034] The target trajectory prediction model is used to predict the expected position of the human hand at the predicted moment;

[0035] The force controller is used to control the movement of the robot end to the desired position of the human hand at the next moment.

[0036] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0037] 1. During the object delivery process, the present invention uses the hand postures at multiple moments in time and combines it with a long short-term memory network with an attention mechanism to predict the hand posture at the time to be predicted. Compared with the existing method of using robots to track human hands, this method can perceive the position of the human hand at the time to be predicted in advance, compensate for the lag in the robot tracking process, and improve the accuracy and timeliness of the response to changes in human hand movement.

[0038] 2. This invention employs adaptive PD control, setting the error term to a fixed value and adjusting the controller coefficient based on the distance between the human hand and the robot end-point. This adaptive PD controller uses a smaller controller coefficient during the initial handover phase to achieve smooth tracking of the robot. Later in the task, a larger controller coefficient is used to improve the robot's accuracy in tracking the desired position.

[0039] 3. This paper adopts a sequence-to-sequence approach in LSTM to achieve variable-step-length prediction of the operator's hand trajectory in human-machine collaboration. Compared with the traditional LSTM network, the seq2seq model can achieve flexibility in the input and output sequence lengths thanks to its encoder-decoder architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a robot visual servoing object delivery method based on an Attention-LSTM neural network constructed according to a preferred embodiment of the present invention.

[0041] Figure 2 4 is a camera calibration scene diagram with the eye outside the hand according to a preferred embodiment of the present invention.

[0042] Figure 3 This is the recognition effect of the YOLOv5 algorithm in the test set according to the preferred embodiment of the present invention.

[0043] Figure 4 This is a position-based visual servo control according to a preferred embodiment of the present invention.

[0044] Figure 5 This is a diagram of track cleaning results according to a preferred embodiment of the present invention, wherein the red track is the retained track after cleaning, and the blue track is the track that was cleaned due to excessive curvature.

[0045] Figure 6 This is an image of loss values ​​after 1000 rounds of neural network training according to a preferred embodiment of the present invention.

[0046] Figure 7 It is the range of change of the predicted coordinates and the robot coordinates on the x-axis when the variable-step-size Attention-LSTM and fixed-step-size Attention-LSTM models are given an x-direction disturbance according to the preferred embodiment of the present invention, wherein (a) is the range of change of the human hand position, predicted position and robot position under the x-direction disturbance of the fixed-step-size Attention-LSTM model; (b) is the range of change of the human hand position, predicted position and robot position under the x-direction disturbance of the variable-step-size Attention-LSTM model.

[0047] Figure 8 This is a visual servoing object delivery experiment process according to a preferred embodiment of the present invention, wherein (a) to (e) are snapshots of the actual delivery experiment within 0-5 seconds respectively.

[0048] Figure 9 1 is a diagram of the robot trajectory, human hand trajectory and predicted human hand trajectory in the visual servoing object delivery experiment according to the preferred embodiment of the present invention, wherein (a), (b) and (c) are the human hand trajectory, human hand predicted trajectory and robot trajectory at t=1s, 2s and 3s respectively. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0050] like Figure 1As shown in FIG, a robot visual servo object delivery method based on an Attention-LSTM neural network includes the following steps:

[0051] (a) Camera calibration is performed using the depth information of the RGB-D camera to obtain the transformation matrix from the camera coordinate system to the robot base coordinate system. Target recognition is performed by training a custom YOLOv5 model and combining it with the RGB-D camera to obtain the real-time position of the recognized target.

[0052] The internal parameters of the RGB-D camera need to be calibrated first to obtain more accurate visual information. The TsaiLenz hand-eye calibration algorithm is used to calibrate the hand on the eye. Represents the rotation matrix from the a coordinate system to the b coordinate system, base is the robot base coordinate system, cam is the camera coordinate system, endn and caln are the robot end coordinate system and the calibration code coordinate system under the nth calibration posture. Figure 2 As shown in Figure 2, in the hand-eye calibration where the eye is outside the hand, the calibration code is close to the end of the robot. Further we can get:

[0053]

[0054] Write it in the form of AX=XB:

[0055]

[0056] In AX=XB, A, X, and B are all homogeneous transformation matrices, which can be decomposed into:

[0057]

[0058] Where R is a 3×3 rotation matrix, T is a 3×1 translation matrix, and O is a 1×3 zero matrix. Thus, AX=XB can be decomposed into:

[0059]

[0060] For the rotation matrix R, it can be regarded as the rotation around the axis P r =[n1,n2,n3] T Rotate by angle θ and redefine using the modified Rodrigues transform From this we can get:

[0061]

[0062] in, Substitute the above formula into (R A -I)·T X =R X ·T B -TA Calculate T X , we can get the matrix X, which is the transformation matrix from the robot base coordinate system to the camera coordinate system, and complete the camera calibration with the eye outside the hand.

[0063] In addition to collecting images of the objects to be recognized, photos of the objects in the experimental environment are also required to train a more accurate model. A total of 1,200 human hand-related images were collected, including 1,000 public human hand image datasets and 200 human hand photos taken in the experimental environment. The images were annotated using a calibration program, and yolov5s was selected as the pre-training weight. A total of 300 rounds of training were performed to obtain a customized YOLOv5 model. The recognition effect is as follows: Figure 3 shown.

[0064] (b) Build a long short-term memory network and collect trajectory datasets for neural network training. Use the trained long short-term memory network model to predict the target trajectory, using the predicted target position as the controller's desired position.

[0065] The seq2seq model's encoder-decoder uses an LSTM network. This architecture is capable of better extracting feature information from historical trajectories. Training is performed using a three-layer LSTM structure, with 512 nodes per layer. LSTM incorporates a forget gate, input gate, and output gate to address issues with traditional RNN networks in handling long-distance dependencies.

[0066] The forget gate determines what information is removed from the memory cell:

[0067] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0068] The input gate determines the extent to which input information is written to the memory cell:

[0069] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0070]

[0071] Combine the results of the forget gate and the input gate to update the memory cell state:

[0072]

[0073] The output gate determines the output information of the current time step and obtains the hidden state output:

[0074] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0075] h t =o t tanh(C t )

[0076] Encoder LSTM is used to process historical trajectories Generate hidden state sequence The attention mechanism dynamically weights the historical trajectory of the input to enhance the model's attention to key features. First, the attention score is calculated for each historical hidden state. Calculation and current hidden state The correlation score e tj :

[0077]

[0078] Then normalize it to the attention weight α tj :

[0079]

[0080] Finally, the context vector c is generated t :

[0081]

[0082] Intermediate vector is the hidden state of the encoder at the last time step.

[0083] The decoder is based on the context vector c t and the previous hidden state Generates the current output:

[0084]

[0085] Finally, the predicted coordinates at time t are generated through linear transformation

[0086]

[0087] Among them, W1 and W2 are weight matrices in the linear transformation, b1 and b2 are bias vectors in the linear transformation, and they are all automatically learned through the model training process.

[0088] The encoder compresses the input sequence into intermediate states, and the decoder gradually generates output based on the intermediate states, thus supporting free changes in the length of the input and output sequences.

[0089] The hand's motion trajectory was captured using an RGB-D camera combined with the YOLOv5 algorithm. Three-dimensional trajectory data, spanning 3900 time steps over 130 seconds at a 30Hz frequency, served as the neural network training dataset. Gaussian filtering was used to preprocess the trajectory dataset to reduce jitter in the trajectory identified by the YOLOv5 algorithm. A Gaussian filter coefficient of sigma = 2 was used to obtain smoother trajectory data.

[0090] Divide the time window into 10, i.e., 1 second of trajectory data, and scroll the time window forward and backward in increments of one time step. Assign the data in each time window to a trajectory point after the desired prediction time step. This means that the data at t∈[T-m+1,T] is used as the network input, and the data at t∈[T+1,T+n] is used as the network output. The target position is predicted for the next n time steps using the trajectory data from the past m time steps.

[0091] Perform trajectory cleaning on the divided time window trajectory and calculate the curvature ρ of all points in each time window:

[0092]

[0093] If the curvature of 5% of the trajectory points in the time window is greater than the threshold ρ max = 20, it will not be input into the network to reduce the impact of low-quality data on model accuracy. Taking a section of the collected sinusoidal trajectory as an example, the effect of trajectory cleaning is as follows: Figure 5 shown.

[0094] Calculate the average distance error between the predicted trajectory and the true trajectory, adjust the parameters and perform multiple training. Ensure the error is as small as possible under the premise of training loss convergence, save the best training model for online prediction of the target trajectory, the loss during the neural network training process is as follows: Figure 6 shown.

[0095] (c) Design a robot visual servo PD controller to control the robot end to move to the desired position.

[0096] like Figure 4 As shown, the robot visual servo method adopts position-based visual servo, and the robot driving force comes from the error The position error e p =x d -x c The robot's desired posture is when the gripper is pointing vertically downward and the gripper's motion direction is parallel to the y-axis of the robot's base coordinate system. Therefore, the posture error where r ik(i∈{d,c},k∈{1,2,3}) represents the rotation matrix R i The kth column of x d The position of the human hand is obtained by identifying and locating it through the YOLO algorithm, x c The end position of the robot is obtained by reading the current state of the robot through ROS.

[0097] Calculate the end control force based on the error term The speed error It is obtained by taking the time derivative of the error e, where the angular velocity of the attitude change is not considered, so

[0098] In human-machine collaborative tasks, it is necessary to ensure the smooth movement of the robot to ensure the safety of the human. In addition, the tracking error of the robot must be minimized as much as possible to ensure the successful delivery of the object. Therefore, the controller coefficient k p 、k d In the design, when the distance d between the human hand and the robot is large, the controller coefficient is set to the minimum value; when d is small, the controller coefficient is set to the maximum value; when d is within the interval, the inverse proportional function is set to achieve a smooth change of the controller coefficient, that is:

[0099]

[0100] Take d min =0.1m,d max =0.5m,

[0101]

[0102] From this we can get:

[0103]

[0104] The end control force F is calculated and then converted into the joint torque τ = J of the robot T F, where J is the robot Jacobian matrix.

[0105] The present invention is further described below with reference to specific embodiments.

[0106] The trained Attention-LSTM neural network model is read, and the position of the human hand is identified and located using an RGB-D camera combined with the YOLOv5 algorithm, capturing the hand's trajectory. When the collected data first fills a time window (i.e., 10 time steps of data per second), the data within the time window is input into the model for the first prediction. Thereafter, at each time step, the first data read into the time window is discarded and the data for the current time step is read in. The new time window data is then input into the model for prediction. The prediction step size is proportional to the distance between the robot end and the human hand. As the task is delivered, the prediction step size decreases. During the physical delivery phase, a certain amount of disturbance is applied to test the disturbance resistance of the variable step size prediction model. The hand is shaken in the x-direction to simulate the operator's fine-tuning during the physical delivery phase. Both a fixed-step Attention-LSTM and a variable-step Attention-LSTM are used to predict the hand's trajectory during the shaking process. The x-direction range of motion of the predicted hand trajectory and the robot trajectory using the two methods is compared, as shown in the following example. Figure 7 Analyzing data within the upper and lower quartiles, the fixed-step prediction method increases the robot's range of motion by 3.446 cm compared to the human hand's range of motion; whereas the variable-step prediction method increases the robot's range of motion by only 0.072 cm. This demonstrates that the proposed variable-step prediction method has strong anti-interference capabilities during the physical delivery phase.

[0107] Install the Franka Hand gripper at the end of the Franka robot arm, hold the ball to be delivered on the gripper, set the end position of the robot as the center point of the gripper, and drive the robot arm at a frequency of 200Hz towards the predicted position of the human hand, such as Figure 8 When the hand moves to the delivery area and the gripper stops above the hand, the gripper is released to complete the object delivery task. Figure 9 As shown in the figure, during the entire mission, the average error between the predicted trajectory and the actual trajectory is 0.0522m.

[0108] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A robot visual servo object delivery method based on Attention-LSTM neural network, characterized in that: The method comprises the following steps: Identify the human hand in the image to be processed at different times and determine the position of the human hand in the robot coordinate system at different times; The hand positions at multiple moments before the time t to be predicted are input into a sequence-to-sequence model, which outputs the expected hand position at the time to be predicted. The sequence-to-sequence model includes an encoder, an attention mechanism, and a decoder. The encoder input is the hand positions at multiple moments before the time to be predicted, and the encoder output serves as the input of the attention mechanism. The decoder input is the output of the attention mechanism and the output of the decoder at time t-1, and the output is the expected hand position at the time to be predicted. At the predicted moment, the robot's end is controlled to move to the expected position of the human hand at the predicted moment, and the object carried by the robot's end is delivered to the human hand, thereby realizing the robot's object delivery process.

2. A robot visual servo object delivery method based on Attention-LSTM neural network as claimed in claim 1, characterized in that: PD control is used to control the movement of the robot end to the expected position of the human hand at the predicted moment.

3. A robot visual servo object delivery method based on Attention-LSTM neural network as claimed in claim 2, characterized in that: When the PD control is used, the control force at the end of the robot is calculated using the following formula: e p =x d -x c in, is the error term in the controller, including position error and attitude error Two parts, x d is the expected position, i.e. the predicted hand position, x c is the current position of the robot end, r ik (i∈{d,c},k∈{1,2},)3 represents the rotation matrix R i The kth column of the antisymmetric matrix 4. A robot visual servo object delivery method based on Attention-LSTM neural network as claimed in claim 3, characterized in that: The calculation formulas for Kp and Kd in the PD control are the same, as follows: in, is the proportionality coefficient, is the differential coefficient, k imin and k imax are all pre-designed fixed values, a and b are constants, which are calculated by ensuring the continuity of critical points, and d is the absolute distance from the human hand to the end of the robot.

5. The robot visual servo object delivery method based on Attention-LSTM neural network according to claim 3, characterized in that: The calculation formulas of Kp and Kd in the PD control are as follows: Among them, k imax and k imin are the maximum and minimum values ​​of the proportional coefficient or differential coefficient, respectively, max and d min are the maximum and minimum absolute distances from the human hand to the robot end, respectively.

6. The robot visual servo object delivery method based on Attention-LSTM neural network according to claim 1, characterized in that: The decoder and encoder both adopt LSTM neural networks, and the output of the attention mechanism is a context vector.

7. The robot visual servo object delivery method based on Attention-LSTM neural network according to claim 1, characterized in that: The calculation formula of the context vector is as follows: e tj =v T tanh(W a [h t-1 ;h j ]+b a ) Among them, α tj is the attention weight, h j (j=1,2,...,t-1) is the historical hidden state, e tj is the historical hidden state and the current hidden state h t-1 The correlation score, v T is the weight vector, W a is the linear transformation matrix, b a is the bias term, c t is the generated context vector.

8. The robot visual servo object delivery method based on Attention-LSTM neural network according to claim 1, characterized in that: The human hand recognition in the image adopts the YOLOv5 model, the input of the model is the image, and the output is the pixel coordinates of the human hand in the image.

9. The robot visual servo object delivery method based on Attention-LSTM neural network according to claim 8, characterized in that: After obtaining the pixel coordinates of the hand in the image, the pixel coordinates of the hand are converted into the coordinates of the hand in the robot base coordinate system using the transformation matrix. The transformation matrix is ​​as follows: Among them, R X is a 3×3 rotation matrix, T X is a 3×1 translation matrix, O X is a 1×3 zero matrix.

10. A robot visual servo object delivery system based on Attention-LSTM neural network, characterized in that: The system includes a hand recognition model, a coordinate calculation model, a target trajectory prediction model, and a force controller, among which: The hand recognition model is used to identify the hand in the image and obtain the position of the hand in the pixel coordinate system; The coordinate calculation module is used to convert the position of the human hand in the pixel coordinate system into the position of the human hand in the robot coordinate system; The target trajectory prediction model is used to predict the expected position of the human hand at the predicted moment; The force controller is used to control the movement of the robot end to the desired position of the human hand at the next moment.

Images