Gaze point based visual grasping method for robot arm

Abstract

The application discloses a mechanical arm visual grasping method based on a gaze point and relates to a mechanical arm grasping method, which comprises the following steps: step 1, human eye gaze point identification and object target identification are performed by a double camera; step 2, a neural network for judging the eye gaze point is optimized by using an artificial bee colony algorithm, and a classification result corresponds to a grasping article operation instruction; step 3, hand-eye calibration is performed, a target detection algorithm is used to position a grasping position, and coordinate conversion is performed, so that the mechanical arm grasping condition can be met; and step 4, gaze point grasping of a UR5E mechanical arm is realized under a ROS system. The application can realize the mechanical arm visual grasping task in a real environment by using the gaze point.

Description

TECHNICAL FIELD

[0001] The application relates to a mechanical arm grabbing method, in particular to a mechanical arm visual grabbing method based on a gaze point. BACKGROUND

[0002] At present, in the demand for an intelligent nursing platform, the existing eye gaze point algorithm uses a data set collected on a tablet, a mobile device or a computer screen, and is limited by devices and physical conditions.

[0003] In order to improve the generality and accuracy of the model, prevent the model from being too large, reduce the redundant parameters from the convolutional layer to the fully connected layer in the convolutional neural network, and improve the inference speed and reduce the storage space of the model while maintaining the accuracy, the artificial bee colony algorithm is introduced to find the optimal pruning structure, and an optimized network model classification method is constructed to obtain the optimal pruning model, so that the accuracy of the human eye image classification is improved.

[0004] It is necessary to collect the user gaze point algorithm to control the mechanical arm grabbing system in a real scene, and the convolutional neural network model judges the human eye gaze point. The network meets the requirements of network lightweight and human eye image classification accuracy, realizes the man-machine interaction system of the eye control mechanical arm, and selects the target object to be grabbed by the user gaze point. The effectiveness of the model and the visual grabbing algorithm is tested on the platform. SUMMARY

[0005] The application aims to provide a mechanical arm visual grabbing method based on a gaze point. The method designs an intelligent platform with an eye control mechanical arm visual grabbing system, and builds a mechanical arm grabbing platform based on vision to assist users in grabbing targets. The application proposes a single-stage YOLOv5 target positioning algorithm, combines a human eye image classification model, and realizes the purpose of gaze point visual grabbing by operating a UR5E mechanical arm through a PC terminal.

[0006] The application aims to realize the following technical solutions:

[0007] The application provides a mechanical arm visual grabbing method based on a gaze point, which comprises the following steps:

[0008] Step 1, building a scene to collect human eye image data sets, and preprocessing the data sets;

[0009] Step 2, training a human eye image classification model;

[0010] Step 3, pruning and optimizing the network structure by using an artificial bee colony algorithm;

[0011] Step 4: Take a picture of the human eye using a camera, and classify the gaze point corresponding to the human eye image:

[0012] Step 5: Collect image information of the object to be grabbed using the mechanical arm vision camera, train the YOLOv5 network model for target recognition, and locate the target to be grabbed;

[0013] Step 6: Calculate the transformation matrix of the camera coordinate system and the end of the mechanical arm, and locate the target. The target position information is converted to the UR5E mechanical arm base coordinate system to perform visual grabbing of the target.

[0014] The data set collection method is:

[0015] a. The human eye image is stable within the field of view of the camera;

[0016] b. Collect image data of each subject's human eye information at 5 horizontal head positions (0°, ±15°, ±30°), 7 horizontal gaze directions (0°, ±5°, ±10°, ±15°), and 3 vertical gaze directions (0°, ±10°);

[0017] Data set processing method:

[0018] a. Each picture in the data set only contains a frontal face task, so the MediaPipe framework is used to detect the face region and define 468 key points of the face features;

[0019] b. The eye position of interest is the six key points (260, 341, 130, 359, 133, 362) in the 468 key points. The eye key points are divided as shown in Figure 1 , where 359 and 362 represent the left side of the left eye, 130 and 133 represent the right side of the right eye, and 360 and 341 represent the corners of the eyes. Each key point corresponds to a set of pixel coordinates, and the minimum and maximum values of the horizontal and vertical coordinates of the six points are selected to obtain the coordinates of the selected rectangular region;

[0020] c. After obtaining the regions corresponding to the left and right eyes, the position map of the two eyes is directly spliced to obtain the picture of the two eye positions.

[0021] The YOLOv5 training steps include:

[0022] a. Take pictures of the target to be grabbed, manually label each image, and obtain the labeled image and label information;

[0023] b. Crop and data augment the collected images to optimize the network model training effect;

[0024] c. input the data set into the YOLOv5 network for training, train the network weight parameters, obtain a target detection model, and use the target detection model for identifying the grabbed target.

[0025] The hand-eye calibration is:

[0026] First, print the ArUco code with a size of 10 cm and an ID code of 100.

[0027] Adjust the angle of the robot, identify the pose of the ArUco code through the ArUco_ros recognition node, and obtain the pose of the calibration board relative to the camera.

[0028] The camera takes pictures of the calibration board at a large inclination, and through path planning, the end of the robot arm moves as little as possible in each group of movements. 17 groups of end poses of the robot arm under the base coordinate are collected, all coordinate systems are visualized through rviz to monitor the movement path of the robot arm, and the hand-eye coordinate relationship of the robot is calculated through two groups of poses.

[0029] The present application has the following beneficial effects and advantages:

[0030] 1. The collected data set in the real environment collects image data under different head poses, avoids the influence caused by head rotation, eliminates redundant information through data processing, and enhances data, avoids the influence caused by light, and improves the generality of the data set.

[0031] 2. The optimized network model of the present application has the effect of lightening, and the performance indicators of the model are improved, which can respond to user intentions in real time, so as to more accurately execute the visual grabbing task of the robot arm, so that the model can run on different device conditions.

[0032] 3. The object detection based on the YOLOv5 algorithm of the present application trains and learns the features of the marked sample model, detects the center point of the sample features, combines the D435i camera, calculates the depth information of the center point coordinates at this moment, calculates the position of the object in the camera coordinate system at this moment according to the coordinate conversion formula, classifies and locates the real object, and the positioning error is within 1cm, which meets the visual grabbing demand.

[0033] 4. In summary, the overall role and technical effect of the present application: the purpose of the gaze point-based visual grabbing algorithm is to give a scene containing an object, take the visual information of the scene as input, the YOLOv5 algorithm classifies and locates the object, a scene information containing an eye image, and successfully applies the eye movement gaze point estimation technology and visual robot arm technology to human-computer interaction tasks. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1Key point marking around eyes in mediapipe face detection;

[0035] Figure 2 Real environment for data set collection;

[0036] Figure 3 Example graph for data set collection;

[0037] Figure 4 Gaze point information contained in data set image data;

[0038] Figure 5 Data set sample redundant information processing;

[0039] Figure 6 Xception network for human eye image classification;

[0040] Figure 7 Artificial bee colony algorithm optimization network structure diagram;

[0041] Figure 8 Model prediction gaze point experiment flow chart;

[0042] Figure 9 YOLOv5 algorithm recognition positioning result;

[0043] Figure 10 Original ArUco code and camera installation position;

[0044] Figure 11 ROS node positioning ArUco code interface and rviz listening robot coordinate visualization interface

[0045] Figure 12 Double camera image data, robot grabbing data, ROS node data flow;

[0046] Figure 13 According to the gaze point information of the method of the present application, the visual robot is guided to grab the experiment. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation method of the present application will be described in detail below. In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, therefore the present application is not limited to the specific implementation disclosed below.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0049] The present application is divided into two parts, the first part is human eye image classification, the second part is visual positioning and grabbing. A method of eye tracking fixation point robot arm grabbing is proposed. Based on the Xception network optimization of artificial bee colony algorithm, the classification of human eye image is realized, the final fixation point information is obtained, the target recognition and positioning based on YOLOv5 are realized, a scene containing objects is given, UR5E robot arm platform, the algorithm takes the visual image information of the scene as input, and the target object is positioned, so that the UR5E robot arm can stably grab the object.

[0050] The human eye image information obtained by the depth camera is described in detail below.

[0051] Human eye image collection and processing:

[0052] Each subject includes 5 horizontal head positions (0°, ±15°, ±30°), 7 horizontal gaze directions (0°, ±5°, ±10°, ±15°) and 3 vertical gaze directions (0°, ±10°). Figure 2 The scene for building the data set collection is shown, Figure 3 Example images of the collected data set are shown, each subject has 5 head poses, and each head pose contains 21 gaze directions. Figure 4 21 fixed points corresponding to each head pose are depicted.

[0053] As Figure 5 In the processing of human eye data set, the irrelevant area of the bridge of the nose is removed, and after the corresponding areas of the left and right eyes are obtained, the positions of the two eyes are reassembled, so as to save more feature information in the feature extraction process.

[0054] The backbone network is Xception network, as Figure 6 As shown, depth separable is mainly used to extract input eye pixel features layer by layer, and a 224x224x3 feature map is input, and 256 1x1x3 convolution kernels are used for convolution operation on the input feature map. The output feature map is the same as the standard convolution. Unlike the standard convolution network, the depth convolution will then split the convolution kernel into a single channel form, and perform convolution operation on each channel without changing the depth of the input feature map. In this way, an output feature map consistent with the number of input feature map channels is obtained.

[0055] The overall pruning system is shown in Table 1:

[0056] Table 1 Pruning Structure

[0057]

[0058]

[0059] In this experiment, Conv_3 has the same number of output channels as Conv_4, and Conv_4 has the same number of output channels as Conv_5. Conv_6_x layers have residual network connections, and the number of output channels in Conv_5 is the same as the number of input channels in Conv_6_x. Therefore, each pruning ratio has three values. At the beginning of the algorithm, 10 pruning ratios are randomly generated, each corresponding to a pruned network structure. The number of channels in each convolutional layer is inconsistent with the number of channels after pruning, so it is necessary to separate the weight parameters of each convolutional layer channel in the pre-trained model to match the pruned network structure. The re-obtained model is trained and tested in small batches to calculate suitability. For each pruned network structure, its corresponding suitability is calculated. For these 10 sets of data, the artificial bee colony algorithm is used to iterate through each pruning ratio to find the optimal pruning structure. In this experiment, we iterate 100 times to find the optimal pruning structure corresponding to the optimal suitability.

[0060] like Figure 7 The overall framework of the Artificial Bee Colony algorithm is shown in the figure: (a) First, an Xception network structure is initialized, where the elements represent the number of channels to be retained. (b) The filter weights of the model are assigned to each optimized structure. We obtain the fitness of the structure by training iteratively according to a specified number of steps. (c) The algorithm updates the set of structures by recalculating the fitness through (b). (b) and (c) will continue for some iterations. (d) The structure with the best fitness is extracted as the optimal pruning structure.

[0061] For the Xception model, which has L layers of convolutional layers and a set W of filters, the pruning structure can be combined into... Where c j Let be the number of channels in the j-th layer. The optimal channel pruning structure selected retains only a certain percentage of channels, thereby reducing the number of channels to:

[0062] For the pruned model N', its structure is represented as: C′=(c′1,c′2,…,c′) L ), where c′ j ≤c j This represents the number of channels in the j-th layer of the Xception pruning model. Given 80% of the training set T... train and 20% of the test set T test, find the optimal combination of C', and get the optimal accuracy of the model through the training of the data set. The following formula is expressed:

[0063]

[0064] where W' is the weight of the pruning model through T train training, and acc(·) is the accuracy of structure C' on T test . The pruned weight is directly updated in the pruned model. Given the Xception network, the pruning structure combination reaches Therefore, the constraint of formula (1) is

[0065]

[0066] where α = 10%, 20%,..., 100% is a preset constant. For the i-th layer, the pre-trained network N retains a certain percentage of the number of channels, that is, c i '≤αc i . We limit the value to [0.1, 1].

[0067] In channel pruning, a certain percentage of filters is retained as an upper bound for each layer, and the introduction of α significantly reduces the combination of pruning structures, but the computational cost is still very large for deep neural networks. Without the need for manual design of the structure, it is not necessary to list all possible structures, and the optimal structure is automatically searched. Initialize a set of n pruning structures The weights are in {0.1c i , 0.2c i ,..., αc i}, and a set of pruning models and a set of pruning weights The goal is to gradually update the model structure and finally select the best pruning structure. It mainly includes three steps:

[0068] (1) Hire bees: the used bees generate a new model structure C j ' to replace each pruned model structure G j ', and define a new model structure G j '.

[0069]

[0070] where r is a random number in the range of [-1, +1], and g≠j indicates that the g-th pruning structure, returns {0.1c i , 0.2c i ,..., αc i}. The bees will generate candidate G according to the fitness.j whether to replace C j is defined as follows:

[0071]

[0072] if the fitness of G j is greater than that of C j , then C j is replaced by G j j , otherwise, C j remains unchanged.

[0073] (2) Observer bees: further update C Unlike this, the probability for fitness-related selection of a pruned structure C j is defined as follows:

[0074]

[0075] (3) Scout bees: if a pruned structure C j is not updated more than M times, the scout bees will generate a new structure C j In order to calculate the fitness in formula (3), N N train are trained on T j This method is time-consuming and labor-intensive in the case of a relatively large data set size, so given a combination of potential C', first select c' j filters from the pre-trained model as the initialization of the jth layer of the pruned model N', and then perform some low-order training on N' to obtain the fitness. Finally, fine-tune the pruned model to have the optimal structure (C') * The model is respectively experimented on the data set of the application, and the data shown in Table 2 is the comparison result before and after the optimization of the model, and the performance index information of the model can be obtained:

[0076] Table 2 Model performance index

[0077]

[0078] The overall model after artificial bee colony optimization removes 42.17% of the channels, 73.28% of the FLOPs, and 64.55% of the parameters, reduces the model size by 68.49%, improves the real-time performance by about 1 second, and improves the accuracy by about 0.6%, thereby improving the performance of the model and promoting the deployment and use of the network model on the mobile nursing platform device.

[0079] Classification experiment on human eye data set:

[0080] ​For the prediction of the gaze point, the trained model is used to predict the gaze point. The gaze point is divided into 21 categories. We first perform face recognition on the input image, segment the recognized eye region, and use the optimized model to estimate the gaze point. The overall structure of the classification of the true gaze point for training and prediction is shown in Figure 8 .

[0081] The YOLOv5 image recognition dataset is self-made, the object to be grabbed is shot by a camera, the object is labeled, after the labeling is completed, the dataset required for training is exported, including labeled pictures and text corresponding to the labeled pictures. The model is trained, and the center point of the detected sample feature is assumed to be (x1, y1), (x1, y2), (x2, y1), and (x2, y2) respectively. The coordinates of the center point of the feature (u1, v1) are represented by equation (5).

[0082] u1=(x1+x2) / 2

[0083] v1=(y1+y2) / 2 (5)

[0084] Combined with the D435i camera, the depth information Z c of the center point coordinates (u1, v1) at this moment is calculated, and according to the coordinate transformation formula, the camera coordinates (X c , Y c , Z c ) of each key point in the camera coordinate system at this moment are calculated.

[0085]

[0086] In the experimental scene, the object to be grabbed is placed corresponding to the gaze point position, and the target positioning algorithm verification is performed, as shown in Figure 9 , the position of each object in the camera coordinate system is calculated.

[0087] The hand-eye calibration experiment:

[0088] Four coordinate systems will be used in calibration, which are: base coordinate system (base), end coordinate system of robot arm (end), camera coordinate system (camera), and target object coordinate system (object).

[0089] Calculate the pose transformation of the robot arm grabbing the target object:

[0090]

[0091] The pose relationship between the camera and the robot base changes with the transformation of the robot end, the calibration board fixed position, the coordinate relationship between the robot base is unchanged, the unknown quantity is the coordinate conversion relationship between the camera and the robot end, even if the robot platform moves the position, the relative position of the camera and the robot end effector does not change. In a certain pose state of the robot, the conversion relationship from the camera to the robot end is:

[0092]

[0093] T represents the transformation matrix. Since the pose of the camera and the robot end on the left side of the equation is always unchanged, the pose relationship between the camera and the robot base on the right side of the equation is always unchanged, so when the robot changes different poses, the following relationship is established by combining the right side of equation (7):

[0094]

[0095] Where the pose of the calibration board to the camera can be obtained by recognizing the ArUco code through the ArUco_ros recognition node, the pose of the robot base to the robot end can be obtained through the robot control box, and the only unknown quantity in equation (9) is After moving the left and right items, we get:

[0096]

[0097] During the movement, the coordinate conversion relationship between the camera and the robot end is fixed and unchanged, and equation (10) is finally converted into the solution of the AX=XB problem, where X represents the coordinate conversion relationship between the camera and the robot end.

[0098] The camera calibration obtains the intrinsic and distortion coefficients of the camera, and the ArUco code is used for hand-eye calibration. The printed ArUco code size is 10cm, and the ID is 100, as shown in (a) of Figure 10 , and the UR5E robot arm and D435i camera are driven, and the ArUco_ros node recognizes the ArUco code.

[0099] The depth camera is fixed on the robot end support, as shown in (b) of Figure 10 , the ArUco code is within the camera shooting range of the robot end, and the rqt visualization interface is opened, and the / aruco_tracker / result topic is selected. Adjust the angle of the robot, and the camera recognizes the pose of the ArUco code, as shown in (a) of Figure 11 . Ensure that in some pictures, the focusing axis is aligned with the calibration board, and 17 groups of robot end motion different positions are realized through path planning, as shown in (b) of Figure 11 , and the robot movement path is listened through rviz visualization.

[0100] The method of the application calibrates the coordinate transformation relationship from the end of the mechanical arm to the camera. The calibration result is as follows:

[0101]

[0102] The image test grasping detection result is as follows:

[0103] During grasping, the visual information of the actual scene is used as input, YOLOv5 target positioning is used, GPU is run, the grasping position in the camera coordinate system is obtained, and the position of the object in the base coordinate system is finally calculated through the coordinate transformation formula. The grasping position and the actual grasping position are calculated as shown in Table 2.

[0104] Table 2 Grasping pose

[0105]

[0106] ROS grasping implementation:

[0107] As Figure 12 is the data transmission flowchart of the overall ROS system. First, the human eye camera collects human eye images, inputs them into the gaze point estimation human eye classification model, outputs gaze point information, places target objects at the gaze point position, and publishes gaze point information through ROS nodes.

[0108] Start the UR5E mechanical arm drive control node, the depth camera node, and the target positioning node. YOLOv5 calculates the center position of the object in the image. The object position is published to the Moveit motion control program through a custom message topic. After coordinate conversion, the pose in the camera coordinate system is converted to the pose under the base of the mechanical arm. According to this pose, the mechanical arm moves to implement grasping.

[0109] The subject needs to observe the objects and then fixate on a certain object. The gaze point is classified by the optimized model to classify the human eye image, and the object to be grasped is selected through the classification of the gaze point. The specified grasping point is generated according to the visual motion grasping model and the user's gaze selection. The object is correctly approached and grasped according to the grasping plan. The object to be grasped is placed at the gaze point position. This experiment selects 7 representative gaze points, as shown in Figure 9 different objects are placed at the selected representative 7 gaze point positions in turn, and different positions are placed. The user needs to grasp the object according to the demand, as shown in Figure 13As shown, the UR5E robot arm is used to complete the visual grasping task of the robot arm under the user's gaze point. The fusion of eye tracking and visual robot technology is successfully applied to human-computer interaction tasks. The method can automatically adjust the pruning strategy according to the characteristics of different tasks and data. This makes the model better adapt to different application scenarios and requirements, and has lower requirements for equipment. Using the gaze point in the real environment for the robot to perform complex tasks has a higher use value for the user.

[0110] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, several improvements and refinements can be made. These improvements and refinements should be considered within the scope of the present application.

Claims

1. A gaze-point-based robot vision grasping method, characterized in that, It comprises the following steps: Step 1, build a scene to collect human eye image dataset, and pretreat the dataset; Step 2, train the classification model of human eye image: artificial bee colony algorithm pruning optimization network structure; Step 3, use the camera to shoot human eye image, and based on artificial bee colony algorithm pruning optimization network structure to classify human eye image, and obtain the final gaze point information; Step 4, collect the image information of the object grabbed by the mechanical arm vision camera, train the YOLOv5 network model for target recognition, and locate the target to be grabbed; Step 5, hand-eye calibration calculates the transformation matrix of camera coordinate system and mechanical arm end, locates the target, and the target position information is converted to UR5E mechanical arm base coordinate system to realize visual grabbing of the target; The step of collecting human eye image dataset is: a. The human eye image is stable in the field of view of the camera; b. Collect human eye information of each subject, and the corresponding head posture 5 horizontal head positions: 0°, ± 15°, ± 30°, 7 horizontal gaze directions: 0°, ± 5°, ± 10°, ± 15° and 3 vertical gaze directions: 0°, ± 10° under the image data; The pretreatment step of the dataset is: a. Only the front face is included in each picture in the dataset, the MediaPipe framework is used to detect the face area, and 468 key points of the face features are defined; b. Focus on the 260, 341, 130, 359, 133, 362 six key points in the 468 key points, 359 and 362 represent the left and right sides of the left eye, 130 and 133 represent the left and right sides of the right eye, which are used as the horizontal coordinates of the right eye, 360 and 341 represent the corners of the eyes, which are used as the vertical coordinates of the eyes. Each key point corresponds to a group of pixel coordinates, and the minimum and maximum values of the horizontal and vertical coordinates of the six key points are selected as the coordinates of the selected rectangular region; c. After obtaining the regions corresponding to the left and right eyes, the position map of the eyes is directly spliced to obtain the picture of the eye position; The artificial bee colony algorithm pruning optimization network structure is: Initialize an Xception network structure; The Xception network structure has L layers of convolutional layers and a filter set W of the L layers of convolutional layers, and the pruning structure is combined as where c j is the number of channels of the jth layer; the optimal channel pruning structure selected only retains a certain percentage of channels: For the pruned model N', the structure is: C' = (c1', c'2,..., c'N) L ), where c' j ≤ c j is the number of channels of the Xception network structure at the jth layer; Given 80% of the training set T train and 20% of the test set T test , find the optimal combination of C', and obtain the optimal accuracy of the model through the training of the data set, as follows: where W' is the weight of the trained model, acc(·) is the accuracy of N' on T train over T test . constraining formula (1) s.t.c i '∈{0.1c i ,0.2c i ,...,αc i} L (2) where a = 10%, 20%,..., 100% is a preset constant, for the ith layer c' i ≤ a c i ; Initialize a set of n pruned structures The weights are in {0.1c i , 0.2c i ,..., αc i}, get a set of pruned models and a set of pruned weights Update the model structure step by step through artificial bee colony algorithm, and finally select the optimal pruning structure.

2. The gaze-point-based mechanical arm visual grasping method according to claim 1, wherein, The hand-eye calibration is: First, print the ArUco code with a size of 10 cm and an ID code of 100; Adjust the angle of the mechanical arm, identify the ArUco code pose through the ArUco_ros recognition node, and obtain the pose of the calibration board relative to the camera; Collect 17 groups of poses of the mechanical arm end in the base coordinate system, visualize all coordinate systems through rviz to listen to the movement path of the mechanical arm, and calculate the hand-eye coordinate relationship of the mechanical arm through two groups of poses.

Images