Assembly line mechanical arm identifying and grabbing system based on Rein-Inception network
Through the recognition and grabbing system based on the Rein-Inception network, the problem of low recognition and grabbing efficiency of assembly line robotic arms is solved, and accurate identification of material types, positions and postures is realized and independent grabbing path planning is improved, thus improving the recognition of robotic arms and grab efficiency.
Patent Information
- Application Number
- CN202411235268.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-04
AI Technical Summary
Existing assembly line robotic arms are less efficient in identifying and grabbing materials, especially in dealing with unknown or complex scenarios.
The recognition and grabbing system based on the Rein-Inception network is adopted, and the loss function is constructed and the network is trained by combining the feature extraction module, attention mechanism module and prediction module to achieve accurate identification of material types, positions and postures by the robotic arm, and based on the identification As a result, the crawling path was independently planned.
It improves the efficiency of the robotic arm in identifying and grabbing materials, overcomes the defects of missing types of prediction modules in traditional networks, achieves more accurate material identification and grabbing decisions, and improves the efficiency of assembly line production.
Smart Images

Figure CN118977246A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of industrial robots, and in particular to an assembly line robot arm recognition and grasping system based on a Rein-Inception network. Background Art
[0002] The requirements for the intelligence of robotic arms in industrial production are getting higher and higher. Traditional robotic arms rely on preset programs to perform tasks and cannot adapt to the ever-changing working environment. Especially when dealing with object grasping tasks in unknown or complex scenes, they lack effective autonomous recognition and adaptability. Deep learning is a method of machine learning. By building a multi-layer neural network, it can automatically learn and extract high-level features of data. In robotic arm recognition and grasping tasks, deep learning models are widely used to extract features of target objects from images. Therefore, the development of a robotic arm intelligent recognition and grasping system based on a deep learning model is of great significance to improving the application value of robotic arms and improving the efficiency of logistics production lines. Summary of the invention
[0003] The purpose of the present invention is to provide a pipeline robot arm recognition and grasping system based on the Rein-Inception network, thereby solving the problem of poor recognition and grasping efficiency of the existing pipeline robot arm.
[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a pipeline robot arm recognition and grasping system based on the Rein-Inception network, comprising a robot arm base, a collection unit, a processing unit, a control unit and an end effector, wherein the robot arm base is fixed on one side of the pipeline, and the processing unit, the control unit and the end effector are fixedly installed in the robot arm base, and a collection unit is fixedly arranged on one side of the end effector, and the collection unit is aligned with the pipeline together with the end effector of the robot arm, and the collection unit is used to scan and obtain the video of the material, and the processing unit is used to extract and preprocess the high-definition 3D image of the material to obtain a plurality of 3D images of the material, and the processing unit is used to use the Rein-Inception network model to identify the type, position and posture of the target material, and the control unit is used to drive the end effector of the robot arm to perform corresponding actions according to the recognition result of the processing unit, and the system also includes a recognition module and a grasping module, and the implementation of the recognition module specifically includes the following steps:
[0005] S1.1: The acquisition unit and processing unit perform 3D image scanning and annotation of the material respectively:
[0006] S1.2: Establish a Rein-Inception network, which includes: feature extraction module Col, attention mechanism module Ate, prediction module Ved and end effector;
[0007] S1.3: Construct the loss function Loss(s,u) and train the Rein-Inception network;
[0008] The grasping module controls the mechanical arm to complete the grasping task according to the result of the recognition module.
[0009] Further, the specific steps of S1.1 are as follows:
[0010] S1.1.1: The acquisition unit scans and obtains the video of the material, and the processing unit extracts and preprocesses the 3D image of the material in the video to obtain a 3D image sequence L of the material. Any frame of the 3D image of the material in the 3D image sequence L is recorded as L r ;
[0011] S1.1.2: For 3D image L r The positions and types K_IN of the M materials in the image are marked to obtain a feature image L with M position labels. pro , wherein the types K_IN of the M materials include: metal K_M, plastic K_P and wood K_W;
[0012] Let the feature image L pro The true type feature of the i-th material is Label i , and Label i ∈{K_M, K_P, K_W},i∈M;
[0013] Let the feature image L pro The predicted category feature of the i-th material is Label out_i , and Label out_i ∈{K_M, K_P, K_W};
[0014] Let the feature image L pro The position of the i-th material in is recorded as POS_IN i , and POS_IN i =(c i_x ,c i_y ,c i_z ,c i_r ,c i_p ,c i_w ), where (c i_x ,c i_y ,c i_z ) indicates that the i-th material is in the feature image L pro The coordinates of (c i_r ,c i_p ,c i_w ) indicates that the i-th material is in the feature image L pro posture.
[0015] Furthermore, the specific steps of S1.2 are as follows:
[0016] S1.2.1: The feature extraction module Col is composed of B sub-units connected in parallel, wherein any b-th sub-unit is composed of h1 convolutional layers Conv and h1 normalization layers Stand in sequence;
[0017] The feature image L pro It is input into the feature extraction module Col and is processed by the h1 convolutional layers Conv of each sub-unit to obtain the sub-feature set {L b,col |b=1,2,…,B}, where L b,Col Represents the sub-feature output by the b-th basic unit;
[0018] {L b,Col |b=1,2,…,B} are processed by h1 normalization layers Stand of each subunit to obtain a normalized feature set I Std = {I b,Std |b=1,2,…,B}, where I b,Std Represents the normalized feature output by the b-th basic unit;
[0019] S1.2.2: The attention module Ate is composed of t3 random fusion branches in parallel;
[0020] The normalized feature set I Std They are input into t3 random fusion branches respectively, among which the t-th feature random fusion branch uses the formula to calculate the t-th fusion feature I t,comb_act , thus the fusion feature set I is obtained by t3 feature random fusion branches comb_act = {I t,comb_act |t=1,2,…t3}, the formula is;
[0021]
[0022] in, represents the weight of the t-th branch, represents the bias of the t-th branch, I i',act represents the i'th activated feature, I j',act represents the j'th activated feature, i' <t3,j'<t3;
[0023] S1.2.3: The prediction module Ved is composed of Q basic units, wherein any qth basic unit is composed of h2 convolutional layers Conv, h2 pooling layers Pool, h2 fully connected layers Link, and 1 activation layer sig in sequence;
[0024] The fusion feature set Icomb_act The input is sent to the verification module Ved, and is processed by the h2 convolutional layers of each basic unit to obtain the convolutional feature set {I q,comb_conv |q=1,2,…Q}, where I q,comb_conv Represents the convolutional features output by the h2th convolutional layer of the qth basic unit;
[0025] {I q,comb_conv |q=1,2,…Q} are upsampled by h2 pooling layers of each basic unit to obtain the upsampled feature set {I q,comb_pool |q=1,2,…Q}, where I q,comb_pool Represents the upsampled features of the h2th pooling layer Pool output of the qth basic unit;
[0026] {I q,comb_pool |q=1,2,…Q} are processed by the h2 fully connected layers Link of each basic unit to obtain the standard feature set {I q,comb_link |q=1,2,…Q}, the standard feature set is passed through the activation function F of the activation layer sig sig The predicted type characteristics of the i-th material obtained after processing {I q,lab |q=1,2,…Q}, and the probability that the true type of the i-th material is equal to the predicted type characteristics {p q,i |q=1,2,…Q}, where I q,comb_link I represents the feature output by the h2th fully connected layer Link of the qth basic unit, q,lab Label indicating the type feature of the i-th material predicted by the q-th basic unit out_q , and Label out_q =Label out_i ∈{K_M, K_P, K_W}, where the activation function F sig The formula shown is:
[0027]
[0028] S1.2.4: End effector: autonomously plans the path and performs grasping according to the material type characteristics predicted by the prediction module Ved.
[0029] Further, the specific steps of S1.3 are as follows:
[0030] S1.3.1: Use the formula to construct the loss function Loss_L of the i-th material type i (p r,i ,u), the loss function formula is:
[0031]
[0032] Among them, I i,lab is the predicted type feature of the i-th material, p r,i is the probability that the predicted type feature of the i-th material is correct, and α and β are hyperparameters;
[0033] S1.3.2: Use the formula to construct the total loss function Loss i_sum , the total loss function formula is:
[0034]
[0035] S1.3.3: Iteratively train the Rein-Inception network through the SGD optimizer to calculate the total loss function Loss i_sum , and adjust the network parameters according to the calculation results. When the total loss function Loss i_sum When it converges to 0, the assembly line material recognition model is obtained.
[0036] Furthermore, S1.2.3 further obtains the type feature Label of the i-th material predicted by the q-th basic unit out_q The corresponding type m:
[0037] Let K_IN[m] represent the type of the m-th material, and K_IN[m] = {K_M, K_P, K_W}, when m = 1, it means that the type of the m-th material K_IN[m] is metal, that is, K_IN[1] = K_M; when m = 2, it means that the type of the m-th material K_IN[m] is plastic, that is, K_IN[2] = K_P; when m = 3, it means that the type of the m-th material K_IN[m] is wood, that is, K_IN[3] = K_W.
[0038] Furthermore, the end effector of the robot arm performs material grabbing, which specifically includes the following steps:
[0039] S2.1: The acquisition unit of the robotic arm acquires the feature image L pro The position POS_IN of the i-th material in i ;
[0040] S2.2: The processing unit of the robot arm takes the characteristic image L of the i-th material obtained in the previous step S2.1 pro Input into the assembly line material identification model obtained in step S1.3.3, and obtain the predicted type m of the i-th material i ;
[0041] S2.3: Determine the type m of the i-th material iIs it the target type that the robot arm can grasp? If yes, proceed to the next step; if no, execute the previous step S2.1;
[0042] S2.4: Initialize the crawling path S init , and the current position S of the end effector of the robot cur =[t cur |p cur ]Put in S init ;
[0043] S2.5: The processing unit of the robot arm determines the position POS_IN of the i-th material according to S2.1 i , calculate the position S that the end effector of the robot arm needs to transform next =[t' cur |p' cur ];
[0044] S2.6: The control unit of the robot arm drives the end effector of the robot arm to transform to the posture S described in the previous step S2.5 next , and the pose S next Put in S init ;
[0045] S2.7: Determine the posture S obtained in the previous step S2.6 next Is it consistent with the position POS_IN of the i-th material? i coincide with each other, if they coincide, then the S init That is, the grasping path of the end effector of the robot arm. If they do not overlap, repeat the above steps S2.5 to S2.7.
[0046] Furthermore, in the position to be transformed by the end effector of the current robot arm in S2.5, S next The specific calculation method includes the following:
[0047] S2.5.1: The acquisition unit of the robot arm obtains the position and posture of the i-th material And the posture POS_IN ii Converted to the coordinates based on the robot arm coordinate system Arm_Img i =(Arm_img i_x ,Arm_img i_y ,Arm_img i_z ) T , where the conversion method is shown as follows;
[0048] (Img i_x ,Img i_y ,Img i_z ) T =(R,T)T (Arm_img i_x ,Arm_img i_y ,Arm_img i_z ,1) T
[0049] Among them, R and T represent the rotation matrix and translation vector from the acquisition unit coordinate system of the robot to the coordinate system of the robot base;
[0050] S2.5.2: Convert the coordinates of the i-th material based on the robot base coordinate system into coordinates based on the robot end effector as the reference coordinate system, the transformation method being:
[0051] (Img i_x ,Img i_y ,Img i_z ) T =(R',T') T (G_img i_x ,G_img i_y ,G_img i_z ,1) T
[0052] S2.5.3: According to the coordinates of the i-th material in the end effector coordinate system of the robot arm (G_img i_x ,G_img i_y ,G_img i_z ,1) T , and the current position of the end effector of the robot arm t cur and posture cur , use the formula to generate the position S that the end effector of the robot arm needs to transform to grab the i-th material next , the posture formula is:
[0053] S next ={[p cur ,t cur ] T +t i *((G_img i_x ,G_img i_y ,G_img i_z ,1) T -[p cur ,t cur ] T )}
[0054] Among them, t i is a random factor. Each time the position of the end effector of the robot arm is calculated, t i will be randomly assigned, and t i ∈(0,1].
[0055] Furthermore, the Rein-Inception network structure includes: a feature extraction module Col, an attention mechanism module Ate, a prediction module Ved and an end effector;
[0056] Feature extraction module Col: It is composed of parallel sub-units, where any sub-unit is composed of convolutional layers Conv and normalization layers Stand in sequence. Compared with Inception-V4, this module extracts the geometric features of materials at a deeper level, providing more refined features for the attention mechanism module Ate, which facilitates Ate to further integrate geometric features, thereby helping the Rein-Inception network model to more accurately identify different types of materials and improve the grasping efficiency;
[0057] Attention mechanism module Ate: receives the fine features provided by the feature extraction module Col and further integrates the geometric features;
[0058] Prediction module Ved: It is composed of small units connected in parallel, which are composed of a convolution layer Conv, a pooling layer Pool, a fully connected layer Link, and an activation layer. After feature extraction and upsampling of each small unit, it is processed by the activation layer to predict the type characteristics of the material and the probability that the predicted type characteristics are correct.
[0059] End effector: autonomously plans the path and performs grasping based on the material type characteristics predicted by the prediction module Ved.
[0060] Furthermore, the feature extraction module Col processes the input feature image in sequence through subunits connected in parallel, and the convolution layer Conv and the normalization layer Stand of each subunit extract the geometric features of the material and provide fine features for the attention mechanism module Ate.
[0061] Furthermore, the prediction module Ved processes the input features in turn through parallel micro-units, and the convolution layer Conv, pooling layer Pool and full connection layer Link of each micro-unit are processed, and finally predicts the type characteristics of the material and the probability that the predicted type characteristics are correct through activation layer processing.
[0062] Furthermore, the specific working mode of the end effector is to predict the material type characteristics according to the prediction module Ved. The types of materials that can be grasped include wood, metal, and plastic. It also autonomously plans the path and executes the grasping action. The execution of the grasping action adjusts the grasping force and angle according to the characteristics and requirements of the material to ensure that the material can be grasped stably.
[0063] Compared with the prior art, the beneficial effects of the present invention are:
[0064] The present invention proposes a pipeline robot arm recognition and grasping system based on the Rein-Inception network. The existing pipeline robot arm recognition and grasping efficiency is poor. The present invention uses the feature extraction module Col to be composed of T sub-units in parallel, wherein any b-th sub-unit is sequentially composed of h1 convolutional layers Conv and h1 normalization layers Stand. Compared with Inception-V4, the feature extraction module of the Rein-Inception network can extract the geometric features of the material at a deeper level, and provide more refined features for the attention mechanism module Ate, which is convenient for the attention mechanism module Ate to further integrate the geometric features, thereby helping the Rein-Inception network model to more accurately identify different types of materials and improve the grasping efficiency.
[0065] The prediction module Ved is composed of small units connected in parallel, which are h2 convolutional layers Conv, h2 pooling layers Pool, h2 fully connected layers Link, and 1 activation layer sig. After feature extraction and upsampling of each small unit, and then processing by the activation layer, the type characteristics of the material and the probability of the predicted type characteristics being correct can be predicted, which overcomes the defect of the prediction module in the original Inception-V4 network that the type is missing, so as to more accurately predict the type characteristics of the material and the probability of the predicted type characteristics being correct, so that the robot arm can choose whether to grasp the material according to the type characteristics, optimize the grasping decision, and improve the grasping efficiency;
[0066] The end effector of the robot arm can autonomously plan the path and perform grasping based on the predicted type characteristics of the material. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 This is an architecture diagram of a robotic arm grasping system based on a Rein-Inception network pipeline of the present invention;
[0068] Figure 2 It is the Rein-Inception network structure diagram in the present invention;
[0069] Figure 3 It is the Rein-Inception network training flow chart in the present invention;
[0070] Figure 4 It is a flow chart of the assembly line mechanical arm grabbing materials in the present invention. DETAILED DESCRIPTION
[0071] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0072] The present invention proposes an assembly line robot arm recognition and grasping system based on the Rein-Inception network, comprising a robot arm base, a collection unit, a processing unit, a control unit and an end effector, wherein the robot arm base is fixed on one side of the assembly line, the processing unit, the control unit and the end effector are fixedly installed in the robot arm base, a collection unit is fixedly arranged on one side of the end effector, the collection unit is aligned with the assembly line together with the end effector of the robot arm, the collection unit is used to scan and obtain a video of the material, the processing unit is used to extract and preprocess the high-definition 3D image of the material to obtain a plurality of 3D images of the material, the processing unit is used to use the Rein-Inception network model to identify the type, position and posture of the target material, the control unit is used to drive the end effector of the robot arm to perform corresponding actions according to the recognition result of the processing unit, the system further comprises a recognition module and a grasping module, and the implementation of the recognition module specifically comprises the following steps:
[0073] S1.1: The acquisition unit and processing unit perform 3D image scanning and annotation of the material respectively:
[0074] S1.1 The specific steps are as follows: first, the acquisition unit scans and obtains the video of the material, and the processing unit extracts and preprocesses the 3D image of the material in the video to obtain a 3D image sequence L of the material. Any frame of the 3D image of the material in the 3D image sequence L is recorded as L r , and then for the 3D image L r The positions and types K_IN of the M materials in the image are marked to obtain a feature image L with M position labels. pro , wherein the types K_IN of the M materials include: metal K_M, plastic K_P and wood K_W, let the feature image L pro The true type feature of the i-th material is Label i , and Label i ∈{K_M, K_P, K_W},i∈M, let the feature image L pro The predicted category feature of the i-th material is Label out_i , and Label out_i ∈{K_M, K_P, K_W}, let the feature image L pro The position of the i-th material in is recorded as POS_INi , and POS_IN i =(c i_x ,c i_y ,c i_z ,c i_r ,c i_p ,c i_w ), where (c i_x ,c i_y ,c i_z ) indicates that the i-th material is in the feature image L pro The coordinates of (c i_r ,c i_p ,c i_w ) indicates that the i-th material is in the feature image L pro In this embodiment, the posture of the marked object can help the machine vision system to accurately distinguish and identify the target object, thereby realizing automated target detection and recognition tasks.
[0075] S1.2: Establish a Rein-Inception network, which includes: feature extraction module Col, attention mechanism module Ate, prediction module Ved and end effector in sequence; S1.2 specific steps are as follows: the feature extraction module Col is composed of B sub-units in parallel, wherein any b-th sub-unit is composed of h1 convolutional layers Conv and h1 normalization layers Stand in sequence, and the feature image L pro It is input into the feature extraction module Col and is processed by the h1 convolutional layers Conv of each sub-unit to obtain the sub-feature set {L b,col |b=1,2,…,B}, where L b,Col Represents the sub-feature output by the b-th basic unit;
[0076] {L b,Col |b=1,2,…,B} are processed by h1 normalization layers Stand of each subunit to obtain a normalized feature set I Std = {I b,Std |b=1,2,…,B}, where I b,Std represents the normalized feature output by the bth basic unit. In this embodiment, each normalization layer Stand has a sub-feature set {L b,col |b=1,2,…,B} is normalized to make the sub-feature set I Std = {I b,Std |b=1,2,…,B} is limited to a certain range of sub-feature data size, thereby eliminating the adverse effects caused by singular sub-feature data. S1.2.2: The attention module Ate is composed of t3 random fusion branches in parallel, which normalizes the feature set I StdThey are respectively input into t3 random fusion branches. Among them, the t-th feature random fusion branch calculates the t-th fusion feature I using the formula t,comb_act so as to obtain the fusion feature set I from t3 feature random fusion branches comb_act ={I t,comb_act |t = 1, 2, … t3}, and the formula for the fusion feature set is;
[0077]
[0078] wherein, represents the weight of the t-th branch, represents the bias of the t-th branch, I i',act represents the i'-th activation feature, I j',act represents the j'-th activation feature, i' < t3, j' < t3. S1.2.3: The prediction module Ved is composed of Q basic units. Among them, any q-th basic unit is successively composed of h2 convolutional layers Conv, h2 pooling layers Pool, h2 fully connected layers Link, and 1 activation layer sig. The fusion feature set I comb_act is input into the verification module Ved and processed by the h2 convolutional layers of each basic unit respectively to obtain the convolutional feature set {I q,comb_conv |q = 1, 2, … Q}, wherein, I q,comb_conv represents the convolutional feature output by the h2-th convolutional layer of the q-th basic unit;
[0079] {I q,comb_conv |q = 1, 2, … Q} are respectively processed by the h2 pooling layers Pool of each basic unit for upsampling to obtain the upsampling feature set {I q,comb_pool |q = 1, 2, … Q}, wherein, I q,comb_pool represents the upsampling feature output by the h2-th pooling layer Pool of the q-th basic unit;
[0080] {I q,comb_pool |q = 1, 2, … Q} are respectively processed by the h2 fully connected layers Link of each basic unit to obtain the standard feature set {I q,comb_link |q = 1, 2, … Q}. After the standard feature set is processed by the activation function F sig of the activation layer sig, the predicted category feature {I q,lab |q = 1, 2, … Q} of the i-th material is obtained, and the probability {p q,i |q = 1, 2, … Q} that the true category of the i-th material is equal to the predicted category feature, wherein, I q,comb_link represents the feature output by the h2-th fully connected layer Link of the q-th basic unit, I q,labLabel indicating the type feature of the i-th material predicted by the q-th basic unit out_q , and Label out_q =Label out_i ∈{K_M, K_P, K_W}, where the activation function F sig The formula shown is:
[0081]
[0082] S1.2.4: End effector: autonomously plans the path and performs grasping according to the material type characteristics predicted by the prediction module Ved.
[0083] S1.3: Construct the loss function Loss(s,u) and train the Rein-Inception network; The specific steps of S1.3 are as follows: S1.3.1: Use the formula to construct the loss function Loss_L of the type of the i-th material i (p r,i ,u), the loss function formula is:
[0084]
[0085] Among them, I i,lab is the predicted type feature of the i-th material, p r,i is the probability that the predicted type characteristics of the i-th material are correct, α and β are hyperparameters; S1.3.2: Use the formula to construct the total loss function Loss i_sum , the total loss function formula is:
[0086]
[0087] S1.3.3: Iteratively train the Rein-Inception network through the SGD optimizer to calculate the total loss function Loss i_sum , and adjust the network parameters according to the calculation results. When the total loss function Loss i_sum When it converges to 0, the assembly line material recognition model is obtained. In this embodiment, through the training of the neural network model, the recognition accuracy of the robot arm to the target object can be improved, and the misidentification and missed identification can be reduced, thereby improving the success rate of grasping. From the above step S1.2.3, the type feature Label of the i-th material predicted by the q-th basic unit is further obtained. out_qCorresponding type m: Let K_IN[m] represent the type of the m-th material, and K_IN[m] = {K_M, K_P, K_W}, when m = 1, it means that the type of the m-th material K_IN[m] is metal, that is, K_IN[1] = K_M; when m = 2, it means that the type of the m-th material K_IN[m] is plastic, that is, K_IN[2] = K_P; when m = 3, it means that the type of the m-th material K_IN[m] is wood, that is, K_IN[3] = K_W. In this embodiment, through the one-to-one correspondence between type features and material types, the robotic arm can accurately and effectively identify the material types in the assembly line, providing a basis for the subsequent grasping tasks.
[0088] The end effector of the robot arm grabs the material, which specifically includes the following steps: S2.1: The acquisition unit of the robot arm obtains the feature image L pro The position POS_IN of the i-th material in i ; S2.2: The processing unit of the robot arm takes the characteristic image L of the i-th material obtained in the previous step S2.1 pro Input into the assembly line material identification model obtained in step S1.3.3, and obtain the predicted type m of the i-th material i ; S2.3: Determine the type m of the i-th material i Is it the target type that the robot can grasp? If yes, proceed to the next step. If not, execute the previous step S2.1; S2.4: Initialize the grasping path S init , and the current position S of the end effector of the robot cur =[t cur |p cur ]Put in S init ; S2.5: The processing unit of the robot arm determines the position POS_IN of the i-th material according to S2.1 i , calculate the position S that the end effector of the robot arm needs to transform next =[t' cur |p' cur ]; S2.6: The control unit of the robot arm drives the end effector of the robot arm to transform to the posture S described in S3.5 next , and the pose S next Put in S init ; S2.7: Determine the posture S obtained in the previous step S2.6 next Is it consistent with the position POS_IN of the i-th material? i coincide with each other, if they coincide, then the S init is the grasping path of the end effector of the robot arm. If they do not overlap, the above steps S2.5 to S2.7 are repeated. In this embodiment, the grasping path S initThe recorded position S is the position S that the end effector of the robot arm needs to transform each time it executes the above step S2.5. next The set of S init ={S next_times}, where next_times represents the number of times the above step S2.5 is executed, and next_times≥1.
[0089] The position S that the end effector of the current robot arm needs to transform in S2.5 next The calculation method specifically includes the following: S2.5.1: The acquisition unit of the robot arm obtains the position and posture of the i-th material And the posture Converted to the coordinates based on the robot arm coordinate system Arm_Img i =(Arm_img i_x ,Arm_img i_y ,Arm_img i_z ) T , where the conversion method is shown as follows;
[0090] (Img i_x ,Img i_y ,Img i_z ) T =(R,T) T (Arm_img i_x ,Arm_img i_y ,Arm_img i_z ,1) T
[0091] Among them, R and T represent the rotation matrix and translation vector from the acquisition unit coordinate system of the robot to the coordinate system of the robot base;
[0092] S2.5.2: Convert the coordinates of the i-th material based on the robot base coordinate system into coordinates based on the robot end effector as the reference coordinate system, the transformation method being:
[0093] (Img i_x ,Img i_y ,Img i_z ) T =(R',T') T (G_img i_x ,G_img i_y ,G_img i_z ,1) T
[0094] S2.5.3: According to the coordinates of the i-th material in the end effector coordinate system of the robot arm (G_img i_x ,G_imgi_y ,G_img i_z ,1) T , and the current position of the end effector of the robot arm t cur and posture cur , use the formula to generate the position S that the end effector of the robot arm needs to transform to grab the i-th material next , the formula is:
[0095] S next ={[p cur ,t cur ] T +t i *((G_img i_x ,G_img i_y ,G_img i_z ,1) T -[p cur ,t cur ] T )}
[0096] Among them, t i is a random factor. Each time the position of the end effector of the robot arm is calculated, t i will be randomly assigned, and t i ∈(0,1], in this embodiment, adjusting the change of the end posture can help optimize the path planning of the robot arm, so that it can complete the task more efficiently and smoothly, reduce unnecessary movement and time waste, and in addition, by controlling the change of the end posture of the robot arm, it can ensure that the robot arm avoids obstacles when executing trajectory planning, reduce the risk of collision, and improve safety.
[0097] Based on the feature extraction module Col mentioned in S1.2 above: it is composed of sub-units in parallel, where any sub-unit is composed of convolutional layers Conv and normalization layers Stand in sequence. Compared with Inception-V4, this module can extract the geometric features of the material at a deeper level, and provide more refined features for the attention mechanism module Ate, so that Ate can further integrate the geometric features, thereby helping the Rein-Inception network model to more accurately identify different types of materials and improve the grasping efficiency. The attention mechanism module Ate: receives the refined features provided by the feature extraction module Col, and further integrates the geometric features. The prediction module Ved: is composed of small units in parallel that are composed of convolutional layers Conv, pooling layers Pool, fully connected layers Link, and 1 activation layer. After the feature extraction and upsampling of each small unit, the type characteristics of the material and the predicted type characteristics can be predicted after processing by the activation layer. The prediction module Ved can predict the material type and the probability of the material being grabbed accurately, which overcomes the defect of the missing prediction type in the prediction module of the original Inception-V4 network. It can predict the material type characteristics and the probability of the predicted type characteristics being correct more accurately, so that the robot arm can choose whether to grab the material according to the material type characteristics, optimize the grabbing decision, and improve the grabbing efficiency. The end effector can autonomously plan the path and perform the grabbing according to the material type characteristics predicted by the prediction module Ved. The feature extraction module Col processes the input feature image in turn through parallel sub-units. The convolution layer Conv and the normalization layer Stand of each sub-unit are used to extract the geometric features of the material and provide fine features for the attention mechanism module Ate. The prediction module Ved processes the input features in turn through parallel tiny units. The convolution layer Conv, the pooling layer Pool and the fully connected layer Link of each tiny unit are used to process the input features in turn. Finally, the material type characteristics and the probability of the predicted type characteristics being correct are predicted after the activation layer processing.
[0098] Specifically, the feature extraction module Col is composed of sub-units in parallel, and each sub-unit contains a convolution layer Conv and a normalization layer Stand in sequence. Compared with Inception-V4, it can extract the geometric features of materials at a deeper level and provide fine features for the attention mechanism module Ate to help the network model more accurately identify the type of materials and improve the grasping efficiency; the prediction module Ved is composed of tiny units in parallel, which are composed of a convolution layer Conv, a pooling layer Pool, a fully connected layer Link and an activation layer in sequence. Through the feature extraction, upsampling and activation layer processing of each tiny unit, the material type characteristics and the probability of correct prediction of the type characteristics can be predicted, which overcomes the defect of missing prediction types of the Inception-V4 network, so that the robot arm can select and grasp according to the material type characteristics, optimize the grasping decision and improve efficiency. The end effector of the robot arm autonomously plans the grasping path and executes the grasping action according to the material type characteristics predicted by the prediction module Ved.
[0099] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0100] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A pipeline robot arm recognition and grasping system based on Rein-Inception network, comprising a robot arm base, a collection unit, a processing unit, a control unit and an end effector, wherein the robot arm base is fixed on one side of the pipeline, the processing unit, the control unit and the end effector are fixedly installed in the robot arm base, the collection unit is fixedly arranged on one side of the end effector, and the collection unit is aligned with the pipeline together with the end effector of the robot arm, characterized in that: The acquisition unit is used to scan and obtain the video of the material, the processing unit is used to extract and preprocess the high-definition 3D image of the material to obtain a plurality of 3D images of the material, the processing unit is used to use the Rein-Inception network model to identify the type, position and posture of the target material, and the control unit is used to drive the end effector of the robot arm to perform corresponding actions according to the recognition result of the processing unit. The system also includes a recognition module and a grasping module. The implementation of the recognition module specifically includes the following steps: S1.1: The acquisition unit and processing unit perform 3D image scanning and annotation of the material respectively: S1.2: Establish a Rein-Inception network, which includes: feature extraction module Col, attention mechanism module Ate, prediction module Ved and end effector; S1.3: Construct the loss function Loss(s,u) and train the Rein-Inception network; The grasping module controls the mechanical arm to complete the grasping task according to the result of the recognition module.
2. According to claim 1, a pipeline robot arm recognition and grasping system based on Rein-Inception network is characterized in that: The specific steps of S1.1 are as follows: S1.1.1: The acquisition unit scans and obtains the video of the material, and the processing unit extracts and preprocesses the 3D image of the material in the video to obtain a 3D image sequence L of the material. Any frame of the 3D image of the material in the 3D image sequence L is recorded as L r ; S1.1.2: For 3D image L r The location and type K_IN of the M materials are marked. Get the feature image L with M position labels pro , wherein the types K_IN of the M materials include: metal K_M, plastic K_P and wood K_W; Let the feature image L pro The true type feature of the i-th material is Label i , and Label i ∈{K_M, K_P, K_W},i∈M; Let the feature image L pro The predicted type feature of the i-th material is Label out_i , and Label out_i ∈{K_M, K_P, K_W}; Let the feature image L pro The position of the i-th material in is recorded as POS_IN i , and POS_IN i =(c i_x ,c i_y ,c i_z ,c i_r ,c i_p ,c i_w ), where (c i_x ,c i_y ,c i_z ) indicates that the i-th material is in the feature image L pro The coordinates of (c i_r ,c i_p ,c i_w ) indicates that the i-th material is in the feature image L pro posture.
3. According to claim 1, a pipeline robot arm recognition and grasping system based on Rein-Inception network is characterized in that: The specific steps of S1.2 are as follows: S1.2.1: The feature extraction module Col is composed of B sub-units connected in parallel, where: Any b-th subunit is composed of h1 convolutional layers Conv and h1 normalization layers Stand in sequence; The feature image L pro It is input into the feature extraction module Col and is processed by the h1 convolutional layers Conv of each sub-unit to obtain the sub-feature set {L b,col |b=1,2,…,B}, where L b,Col Represents the sub-feature output by the b-th basic unit; {L b,Col |b=1,2,…,B} are processed by h1 normalization layers Stand of each subunit to obtain a normalized feature set I Std = {I b,Std |b=1,2,…,B}, where I b,Std Represents the normalized feature output by the b-th basic unit; S1.2.2: The attention module Ate is composed of t3 random fusion branches in parallel; the normalized feature set I Std They are input into t3 random fusion branches respectively, among which t feature random fusion branches use the formula to calculate the tth fusion feature I t,comb_act , thus the fusion feature set I is obtained by t3 feature random fusion branches comb_act = {I t,comb_act |t=1,2,…t3}, the fusion feature set formula is: in, represents the weight of the t-th branch, represents the bias of the t-th branch, I i',act represents the i'th activated feature, I j',act represents the j'th activated feature, i' <t3,j'<t3; S1.2.3: The prediction module Ved is composed of Q basic units, wherein any qth basic unit is composed of h2 convolutional layers Conv, h2 pooling layers Pool, h2 fully connected layers Link, and 1 activation layer sig in sequence; The fusion feature set I comb_act The input is sent to the verification module Ved, and is processed by the h2 convolutional layers of each basic unit to obtain the convolutional feature set {I q,comb_conv |q=1,2,…Q}, where I q,comb_conv Represents the convolutional features output by the h2th convolutional layer of the qth basic unit; {I q,comb_conv |q=1,2,…Q} are upsampled by h2 pooling layers of each basic unit to obtain the upsampled feature set {I q,comb_pool |q=1,2,…Q}, where I q,comb_pool Represents the upsampled features of the h2th pooling layer Pool output of the qth basic unit; {I q,comb_pool |q=1,2,…Q} are processed by the h2 fully connected layers Link of each basic unit to obtain the standard feature set {I q,comb_link |q=1,2,…Q}, the standard feature set is passed through the activation function F of the activation layer sig sig The predicted type characteristics of the i-th material obtained after processing {I q,lab |q=1,2,…Q}, and the probability that the true type of the i-th material is equal to the predicted type characteristics {p q,i |q=1,2,…Q}, where I q,comb_link I represents the feature output by the h2th fully connected layer Link of the qth basic unit, q,lab Label indicating the type feature of the i-th material predicted by the q-th basic unit out_q , and Label out_q =Label out_i ∈{K_M, K_P, K_W}, where the activation function F sig The formula shown is: S1.2.4: End effector: autonomously plans the path and performs grasping according to the material type characteristics predicted by the prediction module Ved.
4. According to claim 1, a pipeline robot arm recognition and grasping system based on Rein-Inception network is characterized in that: The specific steps of S1.3 are as follows: S1.3.1: Use the formula to construct the loss function Loss_L of the i-th material type i (p r,i ,u), the loss function formula is: Among them, I i,lab is the predicted type feature of the i-th material, p r,i is the probability that the predicted type feature of the i-th material is correct, and α and β are hyperparameters; S1.3.2: Use the formula to construct the total loss function Loss i_sum , the total loss function formula is: S1.3.3: Iteratively train the Rein-Inception network through the SGD optimizer to calculate the total loss function Loss i_sum , and adjust the network parameters according to the calculation results. When the total loss function Loss i_sum When it converges to 0, the assembly line material recognition model is obtained.
5. According to claim 3, a pipeline robot arm recognition and grasping system based on Rein-Inception network is characterized in that: S1.2.3 further obtains the type feature Label of the i-th material predicted by the q-th basic unit out_q The corresponding type m: Let K_IN[m] represent the type of the m-th material, and K_IN[m] = {K_M, K_P, K_W}, when m = 1, it means that the type of the m-th material K_IN[m] is metal, that is, K_IN[1] = K_M; when m = 2, it means that the type of the m-th material K_IN[m] is plastic, that is, K_IN[2] = K_P; when m = 3, it means that the type of the m-th material K_IN[m] is wood, that is, K_IN[3] = K_W.
6. According to claim 4, a pipeline robot arm recognition and grasping system based on Rein-Inception network is characterized in that: The end effector of the robotic arm performs material grabbing, which specifically includes the following steps: S2.1: The acquisition unit of the robotic arm acquires the feature image L pro The position POS_IN of the i-th material in i ; S2.2: The processing unit of the robot arm takes the characteristic image L of the i-th material obtained in the previous step S2.1 pro Input into the assembly line material identification model obtained in step S1.3.3, and obtain the predicted type m of the i-th material i ; S2.3: Determine the type m of the i-th material i Is it the target type that the robot arm can grasp? If yes, proceed to the next step; if no, execute the previous step S2.1; S2.4: Initialize the crawling path S init , and the current position S of the end effector of the robot cur =[t cur |p cur ]Put in S init ; S2.5: The processing unit of the robot arm determines the position POS_IN of the i-th material according to S2.1 i , calculate the position S that the end effector of the robot arm needs to transform next =[t' cur |p' cur ]; S2.6: The control unit of the robot arm drives the end effector of the robot arm to transform to the posture S described in the previous step S2.5 next , and the pose S next Put in S init ; S2.7: Determine the posture S obtained in the previous step S2.6 next Is it consistent with the position POS_IN of the i-th material? i coincide with each other, if they coincide, then the S init That is, the grasping path of the end effector of the robot arm. If they do not overlap, repeat the above steps S2.5 to S2.
7.
7. The assembly line robot arm recognition and grasping system based on Rein-Inception network according to claim 6 is characterized by: The position S to be transformed by the end effector of the current robot arm in S2.5 next The specific calculation method includes the following: S2.5.1: The acquisition unit of the robot arm obtains the position POS_IN of the i-th material ii =(Img i_x ,Img i_y ,Img i_z ) T , and the posture POS_IN ii Converted to the coordinates based on the robot arm coordinate system Arm_Img i =(Arm_img i_x ,Arm_img i_y ,Arm_img i_z ) T , where the conversion method is shown as follows; (Img i_x ,Img i_y ,Img i_z ) T =(R,T) T (Arm_img i_x ,Arm_img i_y ,Arm_img i_z ,1) T Among them, R and T represent the rotation matrix and translation vector from the acquisition unit coordinate system of the robot to the coordinate system of the robot base; S2.5.2: Convert the coordinates of the i-th material based on the robot base coordinate system into coordinates based on the robot end effector as the reference coordinate system, the transformation method being: (Img i_x ,Img i_y ,Img i_z ) T =(R',T') T (G_img i_x ,G_img i_y ,G_img i_z ,1) T S2.5.3: According to the coordinates of the i-th material in the end effector coordinate system of the robot arm (G_img i_x ,G_img i_y ,G_img i_z ,1) T , and the current position of the end effector of the robot arm t cur and posture cur , use the formula to generate the position S that the end effector of the robot arm needs to transform to grab the i-th material next , the posture S next The formula is: S next ={[p cur ,t cur ] T +t i *((G_img i_x ,G_img i_y ,G_img i_z ,1) T -[p cur ,t cur ] T )} Among them, t i is a random factor. Each time the position of the end effector of the robot arm is calculated, t i will be randomly assigned, and t i ∈(0,1].
8. According to claim 2, a pipeline robot arm recognition and grasping system based on Rein-Inception network is characterized in that: The feature extraction module Col processes the input feature image in sequence through sub-units connected in parallel. The convolution layer Conv and the normalization layer Stand of each sub-unit extract the geometric features of the material and provide fine features for the attention mechanism module Ate.
9. The assembly line robot arm recognition and grasping system based on Rein-Inception network according to claim 2 is characterized in that: The prediction module Ved processes the input features in turn through parallel micro-units, and the convolution layer Conv, pooling layer Pool and full connection layer Link of each micro-unit are used to process the input features, and finally predicts the type features of the material and the probability that the predicted type features are correct through the activation layer processing.
10. The assembly line robot arm recognition and grasping system based on Rein-Inception network according to claim 2, characterized in that: The specific working mode of the end effector is to predict the material type characteristics according to the prediction module Ved. The types of materials that can be grasped include wood, metal, and plastic. It also plans the path autonomously and executes the grasping action. The execution of the grasping action adjusts the grasping force and angle according to the characteristics and requirements of the material to ensure that the material can be grasped stably.
Citation Information
Patent Citations
Pedestrian re-identification method based on global features and local features of attention mechanism
CN110070073A
Cross-domain scene recognition method based on private information
CN110084141A
Robot arm grabbing control system and method based on intelligent stereoscopic vision
CN111823223A
Small sample ship target recognition method based on global attention relation network
CN113869418A
Control device, method and equipment for pole-climbing detection robot
CN117444953A
