An intelligent grasping method for mobile robots based on monocular vision guidance

Abstract

The present invention discloses a mobile robot intelligent grasping method based on monocular vision guidance. The method divides the grasping process into two stages: a rough positioning stage and a precise grasping stage, thereby improving the flexibility and intelligence of the mobile robot grasping control system. In the precise grasping stage, the mobile robot grasps the target during the movement, that is, the mobile robot is always in motion and the target is stationary. The position and posture of the target in the robot coordinate system are estimated based on the principle of multi-point perspective imaging. A target tracking method based on position visual servoing is designed, so that the manipulator on the mobile robot end can smoothly reach the top of the target and remain relatively still with the target. Finally, the manipulator adjusts the fixed position and posture offset to grasp the target. The mobile robot intelligent grasping method proposed by the present invention has the characteristics of convenient operation, strong applicability and high execution efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of mobile robot intelligent control in the field of robot control, and specifically is an intelligent grasping method for a mobile robot based on monocular vision guidance. Background Art

[0002] As robots are widely used in various fields such as industry, agriculture, and service industry, robot grasping technology, as one of the main research directions of robot control technology, has also received more and more attention and research. In particular, vision-based robot dexterous grasping has always been the focus of robot intelligent control. Whether in unstructured or unknown dynamic environments, visual sensors can provide real-time information to robots, thereby increasing the flexibility and robustness of robot grasping systems. In recent years, with the rapid development of mobile robot technology, vision-based mobile robot systems composed of mobile platforms, serial manipulators, and vision sensors have been increasingly used in the field of intelligent manufacturing due to their own mobility, high maneuverability of multiple serial manipulators, and real-time perception of visual systems. Operations such as object handling and assembly have improved the degree of mechanization and intelligence in the manufacturing field.

[0003] The key technologies of vision-based mobile robot grasping control system include: target detection technology, image feature extraction technology and vision-based grasping control technology. At present, the target detection technology of most mobile robot grasping systems mostly adopts the traditional scanning target detection method, which detects and extracts the target by analyzing the difference between the target and the background. This type of method requires scanning the entire image. If the image size is relatively large, the target detection time will be relatively long. In addition, the target detection accuracy of this type of method is easily affected by lighting. With the rapid development of deep learning technology, target detection methods based on deep neural networks have received more and more attention. With the help of the strong image feature extraction ability of deep neural networks, the efficiency and accuracy of target detection methods based on deep neural networks have been greatly improved (see the literature: J. Varley, J. Weisz, J. Weiss and P. Allen, "Generating multi-fingered robotic grasps via deep learning," IEEE / RSJ International Conference on Intelligent Robots and Systems, 2015, pp. 4415-4420). Image feature extraction is the basis of target pose measurement, and the selection and extraction accuracy of features affect the accuracy of target pose measurement. Typical features include straight line features, point features, and other shape features. In general, feature selection should be robust and representative. For example, when selecting point features, the point features are obtained by solving two straight lines passing through the point, rather than directly extracting the point features. As one of the core steps of target grasping, target pose measurement has always been a hot topic in the field of robotics. Compared with the pose measurement method based on multi-viewing, the target pose measurement method based on monocular vision is often used in robotic systems due to its low cost, convenient deployment, and other advantages (see literature: Suzuki Y, Koyama K, Ming A, et al. Grasping strategy for moving object using Net-Structure Proximity Sensor and vision sensor [C]. 2015: 1403-1409. and Pence WG, F Farelo, Alqasemi R, et al. Visual servoing control of a 9DoFWMRA to perform ADL tasks. Proceedings-IEEE International Conference on Robotics and Automation, 2012: 916-922.).In addition, most vision-based robot grasping control technologies focus on static grasping scenarios, and the vision system is generally fixedly installed. This installation method of the vision system limits the grasping space to a certain area, greatly reducing the flexibility of the system; in addition, static grasping will reduce the robot's work efficiency. Imitating the process of humans being able to flexibly grasp objects during movement, it is of great significance to study dynamic grasping of robots based on mobile vision. Summary of the invention

[0004] In order to solve the problem that existing robots have poor grasping flexibility and low practicality,

[0005] 1. A mobile robot intelligent grasping method based on monocular vision guidance, characterized in that the method comprises the following steps:

[0006] Step S1: placing the target on the operating platform, the robot with a serial structure is fixedly mounted on the mobile platform, and the visual system is fixed on the mobile platform through a rigid connection device;

[0007] Step S2: In the rough positioning stage, the mobile platform moves to the vicinity of the operating platform through the built-in visual navigation module, so that the visual system can collect the image of the target. The visual system can move with the movement of the mobile platform, which greatly improves its visual perception range.

[0008] Step S3: Precise grasping stage: The robot's precise grasping of the target stage is decomposed into target image detection, target feature extraction, target pose measurement, target tracking and target grasping.

[0009] Step S4: Target image detection. A target detection network based on a single deep neural network is built. The backbone feature extraction network of the target detection network uses VGG-16, which can well extract the features of the target image. Secondly, the target detection network uses convolution to extract detection results from different feature maps, i.e. regression and classification, which improves the detection accuracy of the detection network for targets of different scales. The target detection network introduces an attention residual structure, thereby improving the detection accuracy of the target detection network for high-level semantic features.

[0010] Target detection network training: The goal of target detection network training, that is, the loss function of the network is defined as follows

[0011]

[0012] Among them, x = {0, 1} represents the prediction box indicator, l represents the prediction box, g represents the target real box, c is the confidence of the target classification, L conf (x,c) is the confidence loss function (category loss function), L loc(x, l, g) is the positioning loss function (regression loss function), N is the number of candidate prediction boxes, and α is an adjustable parameter used to adjust the ratio of the two loss functions.

[0013] Regression loss function L loc (x,l,g) uses the typical smooth L1 loss,

[0014]

[0015] Among them, k represents the output result of the target detection network for each candidate prediction box, that is, the center coordinates and width and height (cx, cy, cw, ch) of the prediction box, i represents the number of the prediction box, and j represents the number of the real box. When , it means that the predicted box i matches the real box j. When , it means that the predicted box i does not match the real box j, Pos represents the combination of the predicted boxes of the positive sample. represents the output result of the i-th prediction box, represents the result of the jth true box,

[0016] Category loss function L conf (x,c) is defined as follows

[0017]

[0018] Among them, Neg represents the combination of the prediction box of the negative sample, p represents the category number, and p=0 means that the prediction box output is the background. It means that the i-th predicted box matches the j-th real box, and the category of the real box is p. represents the predicted probability of the i-th prediction box corresponding to category p, The calculation uses softmax to complete the probability normalization operation.

[0019] Attention residual structure: The constructed attention residual structure contains two main branches: the trunk branch Γ i,c (x) and mask branch H i,c (x), the trunk branch is mainly responsible for extracting image features, and the mask branch is mainly responsible for generating attention information of different features, that is, assigning weights.

[0020] Ψ i,c (x) = (1 + H i,c (x))·Γ i,c (x) (4)

[0021] Among them, Γ i,c (x) is similar to the residual function, which contains the features extracted by the backbone feature extraction network, Η i,c (x) Adaptively change the weights based on the extracted features,

[0022] Step S5: target feature extraction. Given a target image, after passing through the target detection network, the target image is output as a rectangular target box containing the target. The target box is used as the target's region of interest for target feature extraction. The target feature of this paper is selected as the edge feature of the target. A feature extraction method based on edge template is designed, which specifically includes the following steps: primitive template creation, target edge detection, and feature of interest calculation.

[0023] Creation of primitive templates: Considering that straight lines and corner points are primitive elements of typical edge features, we define straight line primitive templates and corner point primitive templates.

[0024] The target edge line primitive template is defined as follows:

[0025] au+bv+1=0 (5)

[0026] Among them, (a, b) is the image space parameter of the line primitive, (u, v) is the pixel coordinates of the line primitive template in the image space,

[0027] The target corner primitive template is solved by the intersection of two straight lines and is defined as follows:

[0028]

[0029] Among them, (a i ,b i ) is the parameter of the ith straight line primitive, (a j ,b j ) is the parameter of the jth line primitive. These two lines can be obtained based on the edge line primitive template.

[0030] Target edge detection: First, based on the difference between the target color and the background color, the color feature is used to roughly extract the target edge; then, the image noise is filtered and the texture background is removed by combining morphological operations; finally, the Hough transform is used to calculate the target edge line.

[0031] Use the color space transformation formula to convert the image from RGB color space to HSV color space.

[0032] Assuming that the target image obtained after rough color extraction and color space transformation is X, the opening operation is used to smooth the object contour, disconnect the narrow neck, and eliminate small protrusions.

[0033]

[0034] Among them, Y represents the image after the opening operation, A is the opening operator, Θ represents the corrosion operation, represents the expansion operation,

[0035] Use closing operations to bridge narrow gaps and elongated gullies, eliminate small holes, and fill in the breaks in the contour lines.

[0036]

[0037] Among them, Z represents the image after the opening operation, that is, the image obtained by accurately extracting the edge of the target.

[0038] Using the straight line detection principle of Hough transform, the extracted target edge points are fitted with straight lines and the edge straight line features are calculated.

[0039] Calculation of interesting features: The features of interest in this paper are the corner features of the target. According to the Hough line detection, the straight line edges of the target can be extracted. Then, using (6), the corner features of the target image can be obtained. For a given rectangular target, we can extract 4 edge lines and 4 corner features.

[0040] Step S6: Target pose measurement. The four corner point features on the extracted target can be used to estimate the pose of the target in the visual system coordinate system based on the following multi-point perspective imaging principle: c T t

[0041]

[0042] Among them, M c represents the intrinsic parameter matrix of the visual system, which can be obtained by calibration, i = 1, 2, 3, 4 represents the number of the corner point, Z ci represents the depth of the i-th corner point on the target in the visual system coordinate system, (u i ,v i ) represents the image coordinates of the i-th corner point, ( t X i t Y i t Z i ) represents the physical coordinates of the corner point in the target coordinate system. Taking the center of the rectangular target as the reference object, the physical coordinates of the corner point can be calculated according to the size of the target.

[0043] The position of the target in the robot coordinate system r T t The solution is as follows

[0044] r T t = r T c · c T t (10)

[0045] in, r Tc is the extrinsic parameter matrix of the visual system in the robot coordinate system, which can be obtained through extrinsic parameter calibration.

[0046] Step S7: Target tracking. During the process of the robot grasping the target, the mobile platform is always in motion, which makes the end manipulator of robot 3 move relative to target 5. After converting the position of target 5 to the coordinate system of robot 3, a position-based visual servo controller is designed to adjust the movement of the manipulator so that the manipulator tracks the target. The specific derivation of the target tracking controller based on position visual servo is as follows:

[0047] Define the error e of the target tracker

[0048] e=ss* (11)

[0049] Where s = [s p s θ ] T Represents the state quantity, including the three-dimensional space position and posture, and the position vector is represented by s p =[s px s py s pz ], the attitude angle vector is expressed as r θ g =[ r θ gx r θ gy r θ gz ], s* represents the desired state quantity,

[0050] Position and posture matrix of the manipulator in the target coordinate system t T g It is expressed as follows

[0051]

[0052] in, t R g represents the posture matrix, t p g represents the position vector, t T r = r T t -1 represents the pose matrix of the robot in the target coordinate system, r T g The pose matrix of the manipulator in the robot coordinate system can be obtained directly from the robot controller.

[0053] Considering that the purpose of target tracking is to grasp the target, the ideal state s* is selected as a fixed position above the target coordinate system, that is, Then the state quantity s=[ t p g t p θ ] T , t p g Indicates the position of the manipulator in the target coordinate system, t p θ The posture of the manipulator in the target coordinate system can be expressed by the posture matrix t R g get,

[0054] Assume that the robot's movement speed is ν g , ν g =[ν gt v gr ] T , ν gt ∈R 3×1 represents the translation speed, v gr ∈R 3×1 represents the rotation speed, then the relationship between the movement speed of the manipulator and the change of the state quantity is:

[0055]

[0056] Among them, L e is the interaction matrix,

[0057]

[0058] Then the time derivative of the error is

[0059]

[0060] In order to make the robot quickly track the target, the change of the design error meets the Substituting it into (15), we can deduce that the movement speed of the manipulator is

[0061]

[0062] Among them, λ>0 is the adjustment coefficient,

[0063] Therefore, in the target tracking stage, the movement speed of the manipulator is controlled according to (16) to achieve target tracking.

[0064] Step S8: Target grasping, first define the speed of the manipulator as ν′ when target tracking is completed g =[ν′ gt v′ gr ] TIn target grasping, there is a delay from the time when the visual system obtains the latest image to the time when the manipulator performs grasping motion. Let the delay be t s Considering that the closing of the robot also takes time, let the closing time of the robot be t c In addition, in order to achieve stable grasping, the manipulator needs to be further adjusted. Suppose the position offset is △p o , the attitude offset is △θ o , then in the target grasping, the total adjustment of the position and posture of the manipulator is

[0065]

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] 1) Dynamic grasping of targets under the guidance of monocular vision is realized; 2) The efficiency and grasping success rate of the robot grasping control method are relatively high; 3) The equipment and installation of the robot grasping system are relatively easy, and the practicality is also relatively good.

[0068] The present invention refers to the principle of human target grasping during movement, and divides the entire dynamic grasping process into a rough positioning process and a precise grasping process. In the rough positioning process, the robot moves to the vicinity of the target area so that the vision installed on the robot can capture the target image. In the precise grasping process, a target detection method based on a deep neural network is designed, which can quickly extract the target area, and then combine the proposed primitive template extraction method to extract the features of the target area of ​​interest, and then calculate the position and posture of the target. Considering dynamic target grasping, a target tracking method based on position visual servoing and a stable grasping strategy are designed, thereby improving the efficiency and success rate of grasping. With the rapid development of robotics technology, the application prospects and social and economic benefits of the present invention are considerable. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 The figure is a schematic diagram of the structure of the mobile robot intelligent grasping system based on monocular vision guidance of the present invention.

[0070] Figure 2 This is a flow chart of the intelligent grasping of a mobile robot based on monocular vision guidance according to the present invention.

[0071] Figure 3 Schematic diagram of the target detection network of the present invention.

[0072] Figure 4 Schematic diagram of target feature extraction and posture measurement of the present invention.

[0073] Figure 5 The present invention is a flow chart for dynamically tracking and capturing the target.

[0074] Figure 6 Schematic diagram of target position change of the present invention.

[0075] Figure 7 Schematic diagram of target attitude angle change of the present invention.

[0076] Figure 8 For the purpose of the present invention, R The translation speed of the axis.

[0077] Fig. 9 For the purpose of the present invention, R The translation speed of the axis.

[0078] Fig.10 For Z R The rotation speed of the shaft.

[0079] In the attached figure: computer-1, visual system-2, robot-3, mobile platform-4, target-5, operating platform-9, DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0081] As shown in the figure, it is a structural schematic diagram of the mobile robot intelligent grasping method based on monocular vision guidance of the present invention. Figure 1 As shown, the device used in the invention includes: a computer 1, a visual system 2, a robot 3, a mobile platform 4, a target 5 and an operating platform 9, wherein:

[0082] The computer 1 is used to receive the target image collected by the visual system 2, perform target detection, feature extraction and target pose calculation; and the computer 1 controls the mobile platform 4 to move to the target vicinity area according to the target area to complete rough positioning; in addition, the computer 1 controls the robot 3 according to the designed visual servo controller to complete the grasping operation of the target 5.

[0083] The visual system 2 is fixedly mounted on the mobile platform 4 and is used to provide real-time visual information when the robot 3 is grasping the target 5 .

[0084] The robot 3 is fixedly mounted on the mobile platform 4. The robot 3 is a 7-axis serial structured mechanical arm, including three translational degrees of freedom and three rotational degrees of freedom, and a manipulator is fixedly equipped at the end of the robot. Dynamic target grasping is achieved by controlling the movement of the robot 3 and the opening and closing of the manipulator.

[0085] The target 5 is placed on the operating platform, and the target 5 considered by the present invention is a common rectangular structure.

[0086] The operating platform 9 is used to place the target 5, and the operating platform is fixedly installed horizontally.

[0087] The visual system 5 is connected to the computer 1 via a visual connection line; the robot 3 is connected to the computer 1 via a control line; and the mobile platform 4 is connected to the computer 1 via a first control line.

[0088] In one embodiment of the present invention, the robot 3 is an ABB IRB 14050 YuMi single-arm robot, which includes 7 motion axes. The visual system 2 is a Basler color vision sensor with a frame rate of 60fps / s. The mobile platform is a GeekM200C, with a maximum operating speed of 1.5m / s and a positioning accuracy of 10mm.

[0089] Figure 2 The present invention is a flow chart of the intelligent grasping of a mobile robot based on monocular vision guidance, and the method comprises the following steps:

[0090] Step S1: Place the target 5 on the operating platform 9, the serially-structured robot 3 is fixedly mounted on the mobile platform 4, and the visual system 2 is fixed on the mobile platform via a rigid connection device. The visual system 2 can move with the mobile platform, thereby greatly improving its visual perception range.

[0091] Step S2: Rough positioning stage: The mobile platform 4 moves to the vicinity of the operating platform 9 through the built-in visual navigation module, so that the visual system 2 can capture the image of the target 5, and the rough positioning stage is completed.

[0092] Step S3: Precise grasping stage. The stage of robot 3 accurately grasping target 5 is decomposed into target image detection, target feature extraction, target posture measurement, target tracking and target grasping.

[0093] Step S4: Target image detection. Figure 3 The target detection network based on a single deep neural network shown in the figure has a backbone feature extraction network using VGG-16, which can extract the features of the target image well. Secondly, the target detection network uses convolution to extract detection results from different feature maps, namely regression and classification, which improves the detection accuracy of the detection network for targets of different scales. The target detection network introduces an attention residual structure, thereby improving the detection accuracy of the target detection network for high-level semantic features.

[0094] Target detection network training: The goal of target detection network training, that is, the loss function of the network is defined as follows

[0095]

[0096] Among them, x = {0, 1} represents the prediction box indicator, l represents the prediction box, g represents the target real box, c is the confidence of the target classification, L conf (x,c) is the confidence loss function (category loss function), L loc (x, l, g) is the positioning loss function (regression loss function), N represents the number of candidate prediction boxes, and α is an adjustable parameter used to adjust the ratio of the two loss functions.

[0097] Regression loss function L loc (x,l,g) uses the typical smooth L1 loss.

[0098]

[0099] Among them, k represents the output result of the target detection network for each candidate prediction box, that is, the center coordinates and width and height (cx, cy, cw, ch) of the prediction box, i represents the number of the prediction box, and j represents the number of the real box. When , it means that the predicted box i matches the real box j. When , it means that the predicted box i does not match the real box j, Pos represents the combination of the predicted boxes of the positive sample. Represents the output result of the i-th prediction box, Represents the result of the jth ground-truth box.

[0100] Category loss function L conf (x,c) is defined as follows

[0101]

[0102] Among them, Neg represents the combination of the prediction box of the negative sample, p represents the category number, and p=0 means that the prediction box output is the background. It means that the i-th predicted box matches the j-th real box, and the category of the real box is p. represents the predicted probability of the i-th prediction box corresponding to category p, The calculation uses softmax to complete the probability normalization operation.

[0103] Attention residual structure: The constructed attention residual structure contains two main branches: the trunk branch Γ i,c (x) and mask branch H i,c (x). The trunk branch is mainly responsible for extracting image features, and the mask branch is mainly responsible for generating attention information of different features, that is, assigning weights.

[0104] Ψ i,c (x) = (1 + H i,c (x))·Γ i,c (x) (20)

[0105] Among them, Γ i,c (x) is similar to the residual function, which contains the features extracted by the backbone feature extraction network. i,c (x) Adaptively change the weights based on the extracted features.

[0106] Based on the designed target detection network, the embodiment detects 4 different rectangular targets.

[0107] Step S5: Target feature extraction. Given a target image, after passing through the target detection network, the output is a rectangular target box containing the target. The target box is used as the target's region of interest for target feature extraction. The target feature of this paper is selected as the edge feature of the target, and a feature extraction method based on edge template is designed. The target feature extraction process is as follows: Figure 4 As shown, the specific steps include: primitive template creation, target edge detection, and feature of interest calculation.

[0108] Primitive template creation: Considering lines and points as primitive elements of typical edge features, we define line primitive templates and point primitive templates.

[0109] The target edge line primitive template is defined as follows:

[0110] au+bv+1=0 (21)

[0111] Among them, (a, b) are the image space parameters of the line primitive, and (u, v) are the pixel coordinates of the line primitive template in the image space.

[0112] The target point primitive template is solved by the intersection of two straight lines and is defined as follows:

[0113]

[0114] Among them, (a i ,b i ) is the parameter of the ith straight line primitive, (a j ,b j ) are the parameters of the j-th line primitive, and these two lines can be obtained based on the edge line primitive template.

[0115] Target edge detection: First, based on the difference between the target color and the background color, the color features are used to roughly extract the target edge; then, the image noise is filtered and the texture background is removed by combining morphological operations; finally, the Hough transform is used to calculate the target edge line.

[0116] Use the color space transformation formula to convert the image from RGB color space to HSV color space.

[0117] Assuming that the target image obtained after rough color extraction and color space transformation is X, the opening operation is used to smooth the object contour, disconnect the narrow neck, and eliminate small protrusions.

[0118]

[0119] Among them, Y represents the image after the opening operation, A is the opening operator, Θ represents the corrosion operation, Represents an expansion operation.

[0120] Use closing operations to bridge narrow gaps and long gullies, eliminate small holes, and fill in breaks in contour lines.

[0121]

[0122] Wherein, Z represents the image after the opening operation, that is, the image obtained by accurately extracting the edge of the target.

[0123] Using the line detection principle of Hough transform, the extracted target edge points are fitted with lines and the edge line features are calculated.

[0124] Calculation of interesting features: The interesting features in this paper are the corner features of the target. According to Hough line detection, the straight line edges of the target can be extracted, and then (6) can be used to obtain the corner features of the target image. For a given rectangular target, we can extract 4 edge lines and 4 corner features.

[0125] Step S6: Target pose measurement. Figure 4 As shown in the figure, the four corner features of the target are extracted, and then the position and posture of the target in the visual system coordinate system can be estimated based on the following multi-point perspective imaging principle c T t

[0126]

[0127] Among them, M c represents the intrinsic parameter matrix of the visual system 2, which can be obtained by calibration, i = 1, 2, 3, 4 represents the number of the corner point, Z ci represents the depth of the i-th corner point on the target in the visual system coordinate system, (u i ,v i ) represents the image coordinates of the i-th corner point, ( t X i t Y i t Z i) represents the physical coordinates of the corner point in the target coordinate system. Taking the center of the rectangular target as the reference object, the physical coordinates of the corner point can be calculated according to the size of the target. Using Zhang Zhengyou's calibration method, the intrinsic parameter matrix of the visual system 2 is calibrated as

[0128]

[0129] The position of target 5 in the robot 3 coordinate system r T t The solution is as follows

[0130] r T t = r T c · c T t (26)

[0131] in, r T c is the extrinsic parameter matrix of the visual system in the machine 3D coordinate system, which can be obtained by extrinsic parameter calibration.

[0132]

[0133] Step S7: Target tracking. During the process of robot 3 grasping target 5, mobile platform 4 is always in motion, which makes the end manipulator of robot 3 move relative to target 5. After converting the position of target 5 to the coordinate system of robot 3, a position-based visual servo controller is designed to adjust the movement of the manipulator so that the manipulator tracks the target.

[0134] In the embodiment, the target dynamic tracking and grabbing process is as follows Figure 5 As shown. In the target tracking stage, the target image is first collected, and then the target detection network is built to detect the target. Then, the features of interest are extracted according to the designed primitive template method, and then the position and posture of the target in the visual system coordinate system are solved. According to the conversion relationship between the visual system coordinate system and the robot coordinate system - the external parameter matrix and the position and posture of the manipulator in the robot coordinate system (which can be read directly from the robot end), the position and posture of the manipulator in the target coordinate system can be obtained. If the selected state error is less than the set threshold, the target tracking ends. If the state error is greater than the set threshold, the movement speed of the manipulator needs to be adjusted, and then the position of the manipulator is indicated. The next cycle is performed in sequence until the target tracking is completed.

[0135] The specific derivation of the target tracking controller based on position visual servoing is as follows:

[0136] Define the error e of the target tracker

[0137] e=ss* (27)

[0138] Where s = [s p s θ ] T Represents the state quantity, including the three-dimensional space position and posture. The position vector is represented by s p =[s px s py s pz ], the attitude angle vector is expressed as r θ g =[ r θ gx r θ gy r θ gz ], s* represents the desired state quantity.

[0139] Position and posture matrix of the manipulator in the target coordinate system t T g It is expressed as follows

[0140]

[0141] in, t R g represents the posture matrix, t p g represents the position vector, t T r = r T t -1 Represents the pose matrix of robot 3 in the target 5 coordinate system, r T g The pose matrix representing the manipulator in the robot 5 coordinate system can be directly obtained from the robot 3 controller.

[0142] Considering that the purpose of target tracking is to capture target 5, the ideal state s* is selected as a fixed position above the coordinate system of target 5, that is, Then the state quantity s=[ t p g t p θ ] T , t p g Indicates the position of the manipulator in the target 5 coordinate system, t p θ The posture of the manipulator in the target 5 coordinate system can be expressed by the posture matrix t R g get.

[0143] Assume that the robot's movement speed is ν g , ν g =[νgt v gr ] T , ν gt ∈R 3×1 represents the translation speed, v gr ∈R 3×1 represents the rotation speed, then the relationship between the movement speed of the manipulator and the change of the state quantity is:

[0144]

[0145] Among them, L e is the interaction matrix,

[0146]

[0147] Then the time derivative of the error is

[0148]

[0149] In order to make the manipulator quickly track the target 5, the change of the design error satisfies Substituting it into (15), we can deduce that the movement speed of the manipulator is

[0150]

[0151] Among them, λ>0 is the adjustment coefficient,

[0152] Therefore, in the target tracking stage, the movement speed of the manipulator is controlled according to (33) to achieve target tracking. The position and posture error changes during target tracking are as follows: Figure 6-7 As shown. And the speed of the manipulator and the speed of the target relative to the mobile platform are shown in 8-10. Figure 6-10 It can be seen that it takes about 2500ms to complete target tracking, so that the 2nd robot can track the upper target 5, and the speeds of the two are consistent.

[0153] Step S8: Target grabbing. First, define the speed of the manipulator as ν′ when target tracking is completed. g =[ν′ gt v′ gr ] T In target grasping, there is a delay between the time when the visual system obtains the latest image and the time when the robot performs grasping motion. Let this delay be t s Considering that the closing of the robot also takes time, let the closing time of the robot be t c In addition, in order to achieve stable grasping, the manipulator needs to be further adjusted, assuming the position offset is △p o , the attitude offset is △θ o. In the target grasping, the total adjustment of the position and posture of the manipulator is

[0154]

[0155] Among them, △p in the embodiment o =[12.5 0 -20] T mm, △θ o =[60 0 0 0 0 0 ] T

[0156] In the embodiment, the results of the mobile robot before and after grasping the target are as follows: Figure 7 In addition, Table 1 gives the dynamic grasping experimental results under different states.

[0157] Table 1 Results of 10 dynamic grasping experiments under different motion states

[0158]

[0159] The present invention realizes dynamic grasping of a mobile robot based on monocular vision. The efficiency and success rate of dynamic grasping are relatively high, and the entire robot grasping system constructed is also highly flexible and portable. The system can be tried for grasping in the field of intelligent manufacturing.

[0160] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A mobile robot intelligent grasping method based on monocular vision guidance, characterized in that: The method comprises the following steps: Step S1: placing the target on the operating platform, the robot with a serial structure is fixedly mounted on the mobile platform, and the visual system is fixed on the mobile platform through a rigid connection device; Step S2: During the rough positioning phase, the mobile platform moves to the vicinity of the operating platform through the built-in visual navigation module, so that the visual system can capture the image of the target. The visual system moves with the movement of the mobile platform. Step S3: Grasping stage: the robot grasping target stage is decomposed into target image detection, target feature extraction, target posture measurement, target tracking and target grasping process. Step S4: Target image detection. A target detection network based on a single deep neural network is built. The backbone feature extraction network of the target detection network uses VGG-16. Secondly, the target detection network uses convolution to extract detection results from different feature maps. The target detection network introduces an attention residual structure. Target detection network training: The goal of target detection network training, that is, the loss function of the network is defined as follows Among them, x = {0, 1} represents the prediction box indicator, l represents the prediction box, g represents the target real box, c is the confidence of the target classification, L conf (x,c) is the category loss function, L loc (x, l, g) is the regression loss function, N is the number of candidate prediction boxes, and α is an adjustable parameter used to adjust the ratio of the two loss functions. Regression loss function L loc (x,l,g) uses smooth L1 loss, Among them, k represents the output result of the target detection network for each candidate prediction box, the center coordinates (x, y), width, and height (cx, cy, cw, ch) of the prediction box, i represents the number of the prediction box, and j represents the number of the real box. When , it means that the predicted box i matches the real box j. When , it means that the predicted box i does not match the real box j, Pos represents the combination of the predicted boxes of the positive sample. represents the output result of the i-th prediction box, represents the result of the jth true box, Category loss function L conf (x,c) is defined as follows Among them, Neg represents the combination of the predicted boxes of negative samples, p represents the category number, p = 0 means that the predicted box output is the background, and the category of the real box is p. represents the predicted probability of the i-th prediction box corresponding to category p, The calculation uses softmax to complete the probability normalization operation. Attention residual structure: The constructed attention residual structure consists of two branches: the main branch Γ i,c (x) and mask branch H i,c (x), the trunk branch is responsible for extracting image features, and the mask branch generates attention information of different features, that is, assigning weights. P i,c (x)=(1+H i,c (x))·C i,c (x) (4) Among them, Γ i,c (x) Residual function, which contains the features extracted by the backbone feature extraction network, Η i,c (x) Adaptively change the weights based on the extracted features, Step S5: target feature extraction. Given a target image, after passing through a target detection network, the target image is output as a rectangular target frame containing the target. The target frame is used as the target's region of interest for target feature extraction. The target feature is selected as the target's edge feature. The feature extraction method based on the edge template specifically includes the following steps: primitive template creation, target edge detection, and feature of interest calculation. Primitive template creation: straight lines and corner points are primitive elements of edge features, and straight line primitive templates and corner point primitive templates are defined. The target edge line primitive template is defined as follows: au+bv+1=0 (5) Among them, (a, b) is the image space parameter of the line primitive, (u, v) is the pixel coordinates of the line primitive template in the image space, The target corner primitive template is solved by the intersection of two straight lines and is defined as follows: Among them, (a i ,b i ) is the parameter of the ith straight line primitive, (a j ,b j ) is the parameter of the jth line primitive, and these two lines are obtained based on the edge line primitive template. Target edge detection: First, based on the difference between the target color and the background color, the color feature is used to roughly extract the target edge; then, the image noise is filtered and the texture background is removed by combining morphological operations; finally, the Hough transform is used to calculate the target edge line. Use the color space transformation formula to convert the image from RGB color space to HSV color space. Assuming that the target image obtained after rough color extraction and color space transformation is X, the object contour is smoothed by opening operation, the narrow neck is disconnected, and small protrusions are eliminated. Among them, Y represents the image after opening operation, A is the opening operator, Θ represents the corrosion operation, ⊕ represents the expansion operation, Use closing operations to bridge narrow gaps and elongated gullies, eliminate small holes, and fill in breaks in contour lines. Among them, Z represents the image after the opening operation, that is, the image obtained by extracting the edge of the target. Using the straight line detection principle of Hough transform, the extracted target edge points are fitted with straight lines and the edge straight line features are calculated. Calculation of interesting features: The interesting features are the corner features of the target. Based on the Hough line detection, the straight line edges of the target are extracted. Then, using (6), the corner features of the target image are obtained. For a given rectangular target, four edge lines and four corner features are extracted. Step S6: Target pose measurement: The four corner point features on the extracted target are used to estimate the pose of the target in the visual system coordinate system based on the following multi-point perspective imaging principle: c T t Among them, M c represents the intrinsic parameter matrix of the visual system, obtained through calibration, i = 1, 2, 3, 4 represents the number of the corner point, Z ci represents the depth of the i-th corner point on the target in the visual system coordinate system, (u i ,v i ) represents the image coordinates of the i-th corner point, ( t X i t Y i t Z i ) represents the physical coordinates of the corner point in the target coordinate system. Taking the center of the rectangular target as the reference object, the physical coordinates of the corner point are calculated according to the size of the target. The position of the target in the robot coordinate system r T t The solution is as follows r T t = r T c · c T t (10) in, r T c is the extrinsic parameter matrix of the visual system in the robot coordinate system, which is obtained by extrinsic parameter calibration. Step S7: Target tracking. During the process of the robot grasping the target, the mobile platform is always in motion, which makes the robot end manipulator move relative to the target. After converting the target's posture to the robot coordinate system, a position-based visual servo controller is designed to make the manipulator track the target. The specific derivation of the target tracking controller based on position visual servo is as follows: Define the error e of the target tracker e=ss* (11) Where s = [s p s θ ] T Represents the state quantity, including the three-dimensional space position and posture, and the position vector is represented by s p =[s px s py s pz ], the attitude angle vector is expressed as r θ g =[ r θ gx r θ gy r θ gz ],s * represents the expected state quantity, Position and posture matrix of the manipulator in the target coordinate system t T g It is expressed as follows in, t R g represents the posture matrix, t p g represents the position vector, t T r = r T t -1 represents the pose matrix of the robot in the target coordinate system, r T g The pose matrix of the manipulator in the robot coordinate system is obtained directly from the robot controller. The purpose of target tracking is to capture the target. The ideal state s* is selected as a fixed position above the target coordinate system, that is, Then the state quantity s=[ t p g t p θ ] T , t p g Indicates the position of the manipulator in the target coordinate system, t p θ Represents the posture of the manipulator in the target coordinate system through the posture matrix t R g get, Assume that the robot's movement speed is ν g , ν g =[ν gt v gr ] T , ν gt ∈R 3×1 represents the translation speed, v gr ∈R 3×1 represents the rotation speed, then the relationship between the movement speed of the manipulator and the change of the state quantity is: Among them, L e is the interaction matrix, Then the time derivative of the error is In order to make the robot quickly track the target, the change of the design error meets the Substituting it into (15), we can deduce that the movement speed of the manipulator is: Among them, λ>0 is the adjustment coefficient, In the target tracking stage, the movement speed of the manipulator is controlled according to (16) to achieve target tracking. Step S8: Target grasping, first define the speed of the manipulator as ν′ when target tracking is completed g =[ν′ gt v′ gr ] T In target grasping, there is a delay from the time when the visual system obtains the latest image to the time when the manipulator performs grasping motion. Let the delay be t s , assuming the closing time of the manipulator is t c , assuming the position offset is Δp o , the attitude offset is Δθ o , then in the target grasping, the total adjustment of the position and posture of the manipulator is

Images