A pipeline robotic arm recognition and grasping system based on Rein-Inception network

By using the feature extraction and prediction module of the Rein-Inception network, combined with the autonomous planning of the end effector, the problem of low efficiency in object recognition and grasping of robotic arms in complex scenarios is solved, and more efficient material recognition and grasping is achieved.

CN118977246BActive Publication Date: 2026-11-17CHANGZHOU INST OF LIGHT IND TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411235268.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2026-11-17
Estimated Expiration
2044-09-04

AI Technical Summary

Technical Problem

Existing robotic arms on production lines lack autonomous recognition and adaptability when identifying and grasping objects in unknown or complex scenes, resulting in poor recognition and grasping efficiency.

Method used

A robotic arm identification and grasping system based on the Rein-Inception network is adopted, which includes a feature extraction module Col, an attention mechanism module Ate, and a prediction module Ved. The system identifies the type and pose of materials through a deep learning model, and the end effector autonomously plans the grasping path.

Benefits of technology

It improved the accuracy of the robotic arm in identifying materials and the success rate of grasping, optimized grasping decisions, and improved the production efficiency of the assembly line.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118977246B_ABST
    Figure CN118977246B_ABST
Patent Text Reader

Abstract

This invention relates to the field of industrial robot technology and discloses a robotic arm recognition and grasping system based on a Rein-Inception network for assembly lines. This invention solves the problem of poor recognition and grasping efficiency in existing assembly line robotic arms. The invention involves scanning and annotating 3D images of materials using the robotic arm's acquisition unit, establishing a Rein-Inception network, setting a loss function Loss(s,u), training the Rein-Inception network, using the model generated after training the Rein-Inception network to recognize materials, and finally having the robotic arm's end effector plan a grasping path based on the type and posture of the material and its own pose. By using an optimized model with superior recognition performance, the system efficiently grasps materials, significantly reducing the waiting time and error rate of the robotic arm during material handling, thereby improving the efficiency of the robotic arm in grasping materials on an assembly line.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of industrial robots, specifically to a robotic arm identification and grasping system for production lines based on Rein-Inception networks. Background Technology

[0002] The demand for intelligent robotic arms in industrial production is increasing. Traditional robotic arms rely on preset programs to perform tasks and cannot adapt to ever-changing working environments. In particular, when handling object grasping tasks in unknown or complex scenarios, they lack effective autonomous recognition and adaptation capabilities. Deep learning, a method of machine learning, can automatically learn and extract high-level features of data by constructing multi-layer neural networks. In robotic arm recognition and grasping tasks, deep learning models are widely used to extract features of target objects from images. Therefore, developing an intelligent recognition and grasping system for robotic arms based on deep learning models is of great significance for improving the application value of robotic arms and increasing the efficiency of logistics production lines. Summary of the Invention

[0003] The purpose of this invention is to provide a robotic arm recognition and grasping system based on Rein-Inception network for production lines, thereby solving the problem of poor recognition and grasping efficiency of existing robotic arms for production lines.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a robotic arm identification and grasping system for a production line based on a Rein-Inception network, comprising a robotic arm base, a data acquisition unit, a processing unit, a control unit, and an end effector. The robotic arm base is fixed to one side of the production line. The processing unit, control unit, and end effector are fixedly installed inside the robotic arm base. A data acquisition unit is fixedly disposed on one side of the end effector. The data acquisition unit is aligned with the production line along with the end effector of the robotic arm. The data acquisition unit is used to scan and acquire video of the material. The processing unit is used to extract and preprocess the high-definition 3D image of the material to obtain several 3D images of the material. The processing unit is used to identify the type, position, and orientation of the target material using a Rein-Inception network model. The control unit is used to drive the end effector of the robotic arm to perform corresponding actions based on the identification results of the processing unit. The system also includes an identification module and a grasping module. The implementation of the identification module specifically includes the following steps:

[0005] S1.1: The acquisition unit and processing unit respectively perform 3D image scanning and annotation of the material:

[0006] S1.2: Establish the Rein-Inception network, which includes, in sequence: feature extraction module Col, attention mechanism module Ate, prediction module Ved, and end effector;

[0007] S1.3: Construct the loss function Loss(s,u) and train the Rein-Inception network;

[0008] The grasping module controls the robotic arm to complete the grasping task based on the results of the recognition module.

[0009] Furthermore, the specific steps of S1.1 are as follows:

[0010] S1.1.1: The acquisition unit scans and acquires video of the material, and the processing unit extracts and preprocesses the 3D image of the material in the video to obtain a 3D image sequence L of the material. Any frame of the 3D image of the material in the 3D image sequence L is denoted as L. r ;

[0011] S1.1.2: For 3D image L r The locations and types K_IN of the M materials in the image are labeled to obtain a feature image L with M location labels. pro The types of the M materials K_IN include: metal K_M, plastic K_P, and wood K_W;

[0012] Let feature image L pro The true type characteristic of the i-th material is Label. i and Label i ∈{K_M, K_P, K_W}, i∈M;

[0013] Let feature image L pro The predicted category feature of the i-th material is Label. out_i and Label out_i ∈{K_M, K_P, K_W};

[0014] Let feature image L pro The pose of the i-th material is denoted as POS_IN. i And POS_IN i =(c i_x ,c i_y ,c i_z ,c i_r ,c i_p ,c i_w ), where (c i_x ,c i_y ,c i_z ) represents the i-th material in the feature image L pro The coordinates, (c i_r ,c i_p ,c i_w ) represents the i-th material in the feature image L pro The posture.

[0015] Furthermore, the specific steps of S1.2 are as follows:

[0016] S1.2.1: The feature extraction module Col consists of B parallel sub-units, wherein any b-th sub-unit is composed of h1 convolutional layers Conv and h1 normalization layers Stand in sequence;

[0017] The feature image L pro The input is fed into the feature extraction module Col, and then processed by h1 convolutional layers Conv in each sub-unit to obtain the sub-feature set {L}. b,col |b=1,2,…,B}, where L b,Col This represents the sub-feature output by the b-th basic unit;

[0018] {L b,Col |b=1,2,…,B} are processed by h1 normalization layers (Stand) of each subunit to obtain the normalized feature set I. Std ={I b,Std |b=1,2,…,B}, where I b,Std This represents the normalized characteristic of the output of the b-th basic unit;

[0019] S1.2.2: The attention module Ate consists of t3 randomly fused branches connected in parallel;

[0020] The normalized feature set I Std The inputs are respectively fed into t3 random fusion branches, where the t-th feature random fusion branch calculates the t-th fusion feature I using a formula. t,comb_act Thus, the fused feature set I is obtained by randomly fusing t3 feature branches. comb_act ={I t,comb_act The formula is: |t=1,2,…t3};

[0021]

[0022] in, This represents the weight of the t-th branch. I represents the bias of the t-th branch. i',act I represents the i'th activation feature. j',act Let j' represent the j'th activation feature, and i' represent the i'th activation feature. <t3,j'<t3;

[0023] S1.2.3: The prediction module Ved consists of Q basic units, wherein any q-th basic unit is composed of h2 convolutional layers Conv, h2 pooling layers Pool, h2 fully connected layers Link, and 1 activation layer sig in sequence;

[0024] The fusion feature set Icomb_act The input is fed into the validation module Ved, and processed by h² convolutional layers of each basic unit to obtain the convolutional feature set {I}. q,comb_conv |q=1,2,…Q}, where I q,comb_conv This represents the convolutional feature output by the h2th convolutional layer of the qth basic unit;

[0025] {I q,comb_conv |q=1,2,…Q} are each processed by h2 pooling layers of each basic unit for upsampling, resulting in the upsampled feature set {I}. q,comb_pool |q=1,2,…Q}, where I q,comb_pool This represents the upsampled feature of the output of the pooling layer Pool of the q-th basic unit;

[0026] {I q,comb_pool |q=1,2,…Q} are processed by h2 fully connected layers Link of each basic unit to obtain the standard feature set {I q,comb_link |q=1,2,…Q}, the standard feature set is passed through the activation function F of the activation layer sig. sig The predicted type feature of the i-th material after processing {I q,lab |q=1,2,…Q}, and the probability {p} that the actual type of the i-th material is equal to the predicted type characteristic. q,i |q=1,2,…Q}, where I q,comb_link I represents the feature of the Link output of the h2th fully connected layer of the qth basic unit. q,lab Label represents the type feature predicted by the q-th basic unit for the i-th material. out_q and Label out_q =Label out_i ∈{K_M, K_P, K_W}, where the activation function F sig The formula shown is:

[0027]

[0028] S1.2.4: End effector: Based on the material type characteristics predicted by the prediction module Ved, it autonomously plans the path and performs the grabbing.

[0029] Furthermore, the specific steps of S1.3 are as follows:

[0030] S1.3.1: Construct the loss function Loss_L for the i-th material type using the formula. i (p r,i The loss function formula is:

[0031]

[0032] Among them, I i,lab p is the predicted type feature of the i-th material. r,i Let α and β be the probability that the predicted type feature of the i-th material is correct, and α and β are hyperparameters.

[0033] S1.3.2: Constructing the total loss function Loss using the formula i_sum The total loss function formula is as follows:

[0034]

[0035] S1.3.3: Iteratively train the Rein-Inception network using the SGD optimizer and calculate the total loss function Loss. i_sum And adjust the network parameters based on the calculation results, when the total loss function Loss i_sum When the convergence reaches 0, the material recognition model for the production line is obtained.

[0036] Furthermore, step S1.2.3 further obtains the type feature Label of the i-th material predicted by the q-th basic unit. out_q The corresponding type m:

[0037] Let K_IN[m] represent the type of the m-th material, and K_IN[m] = {K_M, K_P, K_W}. When m = 1, it means that the type of the m-th material K_IN[m] is metal, i.e., K_IN[1] = K_M; when m = 2, it means that the type of the m-th material K_IN[m] is plastic, i.e., K_IN[2] = K_P; when m = 3, it means that the type of the m-th material K_IN[m] is wood, i.e., K_IN[3] = K_W.

[0038] Furthermore, the end effector of the robotic arm performs material gripping, specifically including the following steps:

[0039] S2.1: The robotic arm's acquisition unit acquires the feature image L pro The pose POS_IN of the i-th material i ;

[0040] S2.2: The robotic arm's processing unit processes the feature image L of the i-th material obtained in the previous step S2.1. pro Input the data into the production line material identification model obtained in step S1.3.3, and obtain the predicted type m of the i-th material. i ;

[0041] S2.3: Determine the type m of the i-th material. iIs it the type of target that the robotic arm is grasping? If yes, continue to the next step; otherwise, execute the previous step S2.1.

[0042] S2.4: Initialize the crawling path S init And the current pose S of the end effector of the robotic arm cur =[t cur |p cur Put S init ;

[0043] S2.5: The robotic arm's processing unit calculates the pose POS_IN of the i-th material as described in S2.1. i The pose S to be transformed by the end effector of the computational robotic arm. next =[t' cur |p' cur ];

[0044] S2.6: The control unit of the robotic arm drives the end effector of the robotic arm to change to the pose S described in the previous step S2.5. next and the pose S next Put S init ;

[0045] S2.7: Determine the pose S obtained in the previous step S2.6. next Is it related to the pose POS_IN of the i-th material? i If they overlap, then the S... init This refers to the grasping path of the robotic arm's end effector. If they do not overlap, repeat steps S2.5 to S2.7.

[0046] Furthermore, in S2.5, the pose to be changed by the end effector of the current robotic arm, S next The specific calculation methods include the following:

[0047] S2.5.1: The robotic arm's data acquisition unit obtains the pose of the i-th material. and the pose POS_IN ii Converted to Arm_Img coordinates based on the robotic arm coordinate system i =(Arm_img i_x ,Arm_img i_y ,Arm_img i_z ) T The conversion method is shown in the formula below;

[0048] (Img i_x 1mg i_y 1mg i_z ) T =(R,T)T (Arm_img i_x ,Arm_img i_y ,Arm_img i_z ,1) T

[0049] Where R and T represent the rotation matrix and translation vector from the coordinate system of the robotic arm's acquisition unit to the coordinate system of the robotic arm's base, respectively;

[0050] S2.5.2: Convert the coordinates of the i-th material based on the coordinate system of the robot arm base to the coordinates based on the end effector of the robot arm. The transformation method is as follows:

[0051] (Img i_x 1mg i_y 1mg i_z ) T =(R',T') T (G_img i_x ,G_img i_y ,G_img i_z ,1) T

[0052] S2.5.3: Based on the coordinates (G_img) of the i-th material in the coordinate system of the end effector of the robotic arm. i_x ,G_img i_y ,G_img i_z ,1) T And the current position t of the end effector of the robotic arm. cur and posture p cur The formula is used to generate the pose S that the end effector of the robotic arm needs to change to grasp the i-th material. next The pose formula is:

[0053] S next ={[p cur ,t cur ] T +t i *((G_img i_x ,G_img i_y ,G_img i_z ,1) T -[p cur ,t cur ] T )}

[0054] Among them, t i As a random factor, t is used each time the pose to be transformed by the end effector of the robotic arm is calculated. i It will be randomly assigned a value, and t i ∈(0,1).

[0055] Furthermore, the Rein-Inception network structure includes: a feature extraction module Col, an attention mechanism module Ate, a prediction module Ved, and an end effector;

[0056] The feature extraction module Col consists of parallel sub-units, where any i-th sub-unit is composed of i-th convolutional layers Conv and i-th normalization layers Stand. Compared with Inception-V4, this module extracts the geometric features of materials at a deeper level, providing more refined features for the attention mechanism module Ate, which facilitates Ate to further integrate geometric features, thereby helping the Rein-Inception network model to more accurately identify different types of materials and improve grasping efficiency.

[0057] Attention mechanism module Ate: Receives fine features provided by feature extraction module Col and further integrates geometric features;

[0058] The prediction module Ved consists of parallel micro-units consisting of one convolutional layer (Conv), one pooling layer (Pool), one fully connected layer (Link), and one activation layer. After feature extraction and upsampling of each micro-unit, the activation layer then processes the data to predict the type of material and the probability that the predicted type of material is correct.

[0059] End effector: Based on the material type characteristics predicted by the prediction module Ved, it autonomously plans the path and performs the grabbing.

[0060] Furthermore, the feature extraction module Col processes the input feature image through parallel sub-units, with each sub-unit having a convolutional layer Conv and a normalization layer Stand, in turn to extract the geometric features of the material and provide fine features for the attention mechanism module Ate.

[0061] Furthermore, the prediction module Ved uses parallel micro-units, where each micro-unit's convolutional layer (Conv), pooling layer (Pool), and fully connected layer (Link) process the input features sequentially. Finally, after processing by the activation layer, it predicts the type features of the material and the probability that the predicted type features are correct.

[0062] Furthermore, the end effector works by predicting the material type characteristics based on the prediction module Ved. The types of materials that can be grasped include wood, metal, and plastic. It autonomously plans its path and executes the grasping action. The grasping force and angle are adjusted according to the characteristics and requirements of the material to ensure that the material can be grasped stably.

[0063] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0064] This invention proposes a robotic arm recognition and grasping system based on the Rein-Inception network. Existing robotic arms on production lines have poor recognition and grasping efficiency. The present invention utilizes a feature extraction module Col, which consists of T parallel sub-units. Each b-th sub-unit is composed of h1 convolutional layers (Conv) and h1 normalization layers (Stand). Compared with Inception-V4, the feature extraction module of the Rein-Inception network can extract the geometric features of materials at a deeper level, providing more refined features for the attention mechanism module Atte. This facilitates the Atte module to further integrate geometric features, thereby helping the Rein-Inception network model to more accurately identify different types of materials and improve grasping efficiency.

[0065] The prediction module Ved is composed of parallel micro-units consisting of h2 convolutional layers (Conv), h2 pooling layers (Pool), h2 fully connected layers (Link), and 1 activation layer (sig). After feature extraction and upsampling of each micro-unit, the activation layer processes the data to predict the type of material and the probability of the predicted type being correct. This overcomes the defect of missing prediction types in the original Inception-V4 network, thus achieving more accurate prediction of the type of material and the probability of the predicted type being correct. This allows the robotic arm to choose whether to grasp the material based on its type, optimizing the grasping decision and improving grasping efficiency.

[0066] The end effector of the robotic arm can autonomously plan its path and perform grasping based on the predicted characteristics of the material. Attached Figure Description

[0067] Figure 1 This is an architecture diagram of a robotic arm grasping system based on a Rein-Inception network pipeline according to the present invention;

[0068] Figure 2 This is a diagram of the Rein-Inception network structure in this invention;

[0069] Figure 3 This is a flowchart of the Rein-Inception network training process in this invention;

[0070] Figure 4 This is a flowchart of the material grasping process of the robotic arm on the production line in this invention. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0072] This invention proposes a robotic arm recognition and grasping system for an assembly line based on a Rein-Inception network. The system includes a robotic arm base, a data acquisition unit, a processing unit, a control unit, and an end effector. The robotic arm base is fixed to one side of the assembly line. The processing unit, control unit, and end effector are fixedly installed inside the robotic arm base. A data acquisition unit is fixedly mounted on one side of the end effector. The data acquisition unit is aligned with the assembly line along with the end effector of the robotic arm. The data acquisition unit is used to scan and acquire video of the material. The processing unit is used to extract and preprocess the high-definition 3D images of the material to obtain several 3D images of the material. The processing unit is used to identify the type, location, and orientation of the target material using a Rein-Inception network model. The control unit is used to drive the end effector of the robotic arm to perform corresponding actions based on the identification results of the processing unit. The system also includes a recognition module and a grasping module. The implementation of the recognition module specifically includes the following steps:

[0073] S1.1: The acquisition unit and processing unit respectively perform 3D image scanning and annotation of the material:

[0074] The specific steps of S1.1 are as follows: First, the acquisition unit scans and acquires the video of the material. The processing unit extracts and preprocesses the 3D image of the material in the video to obtain a 3D image sequence L of the material. Any frame of the 3D image of the material in the 3D image sequence L is denoted as L. r Secondly, regarding 3D images L r The locations and types K_IN of the M materials in the image are labeled to obtain a feature image L with M location labels. pro Wherein, the types K_IN of the M materials include: metal K_M, plastic K_P, and wood K_W, and let the feature image L pro The true type characteristic of the i-th material is Label. i and Label i Let the feature image L ∈{K_M, K_P, K_W}, i∈M. pro The predicted category feature of the i-th material is Label. out_i and Label out_i Let L ∈{K_M, K_P, K_W}, and let the feature image L pro The pose of the i-th material is denoted as POS_IN.i And POS_IN i =(c i_x ,c i_y ,c i_z ,c i_r ,c i_p ,c i_w ), where (c i_x ,c i_y ,c i_z ) represents the i-th material in the feature image L pro The coordinates, (c i_r ,c i_p ,c i_w ) represents the i-th material in the feature image L pro In this embodiment, marking the object pose can help the machine vision system accurately identify and recognize the target object, thereby achieving automated target detection and recognition tasks.

[0075] S1.2: Establish a Rein-Inception network, which includes: a feature extraction module Col, an attention mechanism module Ate, a prediction module Ved, and an end effector. The specific steps of S1.2 are as follows: The feature extraction module Col consists of B parallel sub-units, where any b-th sub-unit consists of h1 convolutional layers (Conv) and h1 normalization layers (Stand) in sequence. The feature image L... pro The input is fed into the feature extraction module Col, and then processed by h1 convolutional layers Conv in each sub-unit to obtain the sub-feature set {L}. b,col |b=1,2,…,B}, where L b,Col This represents the sub-feature output by the b-th basic unit;

[0076] {L b,Col |b=1,2,…,B} are processed by h1 normalization layers (Stand) of each subunit to obtain the normalized feature set I. Std ={I b,Std |b=1,2,…,B}, where I b,Std This represents the normalized feature output by the b-th basic unit. In this embodiment, each normalization layer Stand represents the input sub-feature set {L}. b,col Normalize |b=1,2,…,B} to make the sub-feature set I Std ={I b,Std The size of the sub-feature data in |b=1,2,…,B} is limited to a certain range, thereby eliminating the adverse effects caused by singular sub-feature data. S1.2.2: The attention module Ate is composed of t3 random fusion branches connected in parallel, which integrates the normalized feature set I Stdare respectively input into t3 random fusion branches, wherein the t-th feature random fusion branch uses a formula to calculate the t-th fusion feature I t,comb_act , so as to obtain a fusion feature set I from the t3 feature random fusion branches comb_act ={I t,comb_act |t=1,2,…t3}, the formula of the fusion feature set is;

[0077]

[0078] wherein, represents the weight of the t-th branch, represents the bias of the t-th branch, I i',act represents the i'-th activation feature, I j',act represents the j'-th activation feature, i'<t3,j'<t3, S1.2.3: The prediction module Ved is composed of Q basic units, wherein any q-th basic unit is sequentially composed of h2 convolutional layers Conv, h2 pooling layers Pool, h2 fully-connected layers Link, and 1 activation layer sig, the fusion feature set I comb_act is input into the verification module Ved, and respectively processed by the h2 convolutional layers of each basic unit to obtain a convolutional feature set {I q,comb_conv |q=1,2,…Q}, wherein I q,comb_conv represents the convolutional feature output by the h2-th convolutional layer of the q-th basic unit;

[0079] {I q,comb_conv |q=1,2,…Q} are respectively subjected to upsampling processing by the h2 pooling layers Pool of each basic unit to obtain an upsampled feature set {I q,comb_pool |q=1,2,…Q}, wherein I q,comb_pool represents the upsampled feature output by the h2-th pooling layer Pool of the q-th basic unit;

[0080] {I q,comb_pool |q=1,2,…Q} are respectively processed by the h2 fully-connected layers Link of each basic unit to obtain a standard feature set {I q,comb_link |q=1,2,…Q}, the standard feature set is processed by the activation function F of the activation layer sig sig to obtain the predicted category feature {I of the i-th material q,lab |q=1,2,…Q}, and the probability {p that the real category of the i-th material is equal to the predicted category feature q,i |q=1,2,…Q}, wherein I q,comb_link represents the feature output by the h2-th fully-connected layer Link of the q-th basic unit, I q,labLabel represents the type feature predicted by the q-th basic unit for the i-th material. out_q and Label out_q =Label out_i ∈{K_M, K_P, K_W}, where the activation function F sig The formula shown is:

[0081]

[0082] S1.2.4: End effector: Based on the material type characteristics predicted by the prediction module Ved, it autonomously plans the path and performs the grabbing.

[0083] S1.3: Construct the loss function Loss(s,u) and train the Rein-Inception network; the specific steps of S1.3 are as follows: S1.3.1: Construct the loss function Loss_L for the i-th material type using the formula. i (p r,i The loss function formula is:

[0084]

[0085] Among them, I i,lab p is the predicted type feature of the i-th material. r,i Let be the probability that the predicted type feature of the i-th material is correct, and α and β are hyperparameters; S1.3.2: Construct the total loss function Loss using the formula. i_sum The total loss function formula is as follows:

[0086]

[0087] S1.3.3: Iteratively train the Rein-Inception network using the SGD optimizer and calculate the total loss function Loss. i_sum And adjust the network parameters based on the calculation results, when the total loss function Loss i_sum When convergence reaches 0, the material recognition model for the production line is obtained. In this embodiment, training the neural network model can improve the accuracy of the robotic arm in recognizing target objects, reduce false recognition and missed recognition, thereby improving the success rate of grasping. From the above step S1.2.3, the type feature Label of the i-th material predicted by the q-th basic unit is further obtained. out_qThe corresponding type m: Let K_IN[m] represent the type of the m-th material, and K_IN[m] = {K_M, K_P, K_W}. When m = 1, it means that the type of the m-th material K_IN[m] is metal, that is, K_IN[1] = K_M; when m = 2, it means that the type of the m-th material K_IN[m] is plastic, that is, K_IN[2] = K_P; when m = 3, it means that the type of the m-th material K_IN[m] is wood, that is, K_IN[3] = K_W. In this embodiment, by the one-to-one correspondence between the type characteristics and the material types, the robotic arm can accurately and effectively identify the types of materials in the production line, providing a basis for the subsequent grasping tasks.

[0088] The end effector of the robotic arm performs material gripping, specifically including the following steps: S2.1: The robotic arm's acquisition unit acquires the feature image L. pro The pose POS_IN of the i-th material i S2.2: The robotic arm's processing unit processes the feature image L of the i-th material obtained in the previous step S2.1. pro Input the data into the production line material identification model obtained in step S1.3.3, and obtain the predicted type m of the i-th material. i S2.3: Determine the type m of the i-th material. i Is this the type of target the robotic arm is grasping? If yes, proceed to the next step; otherwise, execute the previous step S2.1. S2.4: Initialize the grasping path S init And the current pose S of the end effector of the robotic arm cur =[t cur |p cur Put S init S2.5: The robotic arm's processing unit, based on the pose POS_IN of the i-th material as described in S2.1, i The pose S to be transformed by the end effector of the computational robotic arm. next =[t' cur |p' cur S2.6: The control unit of the robotic arm drives the end effector of the robotic arm to change to the pose S described in S3.5. next and the pose S next Put S init S2.7: Determine the pose S obtained in the previous step S2.6. next Is it related to the pose POS_IN of the i-th material? i If they overlap, then the S... init This refers to the grasping path of the robotic arm's end effector. If the paths do not overlap, steps S2.5 to S2.7 above are repeated. In this embodiment, the grasping path S... initThe recorded position S is the pose change that the robotic arm end effector needs to perform each time it executes step S2.5 above. next The set of, i.e.: S init ={S next_times}, where next_times represents the number of times the above step S2.5 is executed, and next_times≥1.

[0089] In S2.5, the pose S that the end effector of the current robotic arm needs to change is... next The calculation method specifically includes the following: S2.5.1: The robotic arm's acquisition unit obtains the pose of the i-th material. and the pose Converted to Arm_Img coordinates based on the robotic arm coordinate system i =(Arm_img i_x ,Arm_img i_y ,Arm_img i_z ) T The conversion method is shown in the formula below;

[0090] (Img i_x 1mg i_y 1mg i_z ) T =(R,T) T (Arm_img i_x ,Arm_img i_y ,Arm_img i_z ,1) T

[0091] Where R and T represent the rotation matrix and translation vector from the coordinate system of the robotic arm's acquisition unit to the coordinate system of the robotic arm's base, respectively;

[0092] S2.5.2: Convert the coordinates of the i-th material based on the coordinate system of the robot arm base to the coordinates based on the end effector of the robot arm. The transformation method is as follows:

[0093] (Img i_x 1mg i_y 1mg i_z ) T =(R',T') T (G_img i_x ,G_img i_y ,G_img i_z ,1) T

[0094] S2.5.3: Based on the coordinates (G_img) of the i-th material in the coordinate system of the end effector of the robotic arm. i_x ,G_imgi_y ,G_img i_z ,1) T And the current position t of the end effector of the robotic arm. cur and posture p cur The formula is used to generate the pose S that the end effector of the robotic arm needs to change to grasp the i-th material. next The formula is:

[0095] S next ={[p cur ,t cur ] T +t i *((G_img i_x ,G_img i_y ,G_img i_z ,1) T -[p cur ,t cur ] T )}

[0096] Among them, t i As a random factor, t is used each time the pose to be transformed by the end effector of the robotic arm is calculated. i It will be randomly assigned a value, and t i In this embodiment, adjusting the end-effector pose can help optimize the path planning of the robotic arm, making it more efficient and smooth in completing tasks, reducing unnecessary movement and time waste. In addition, by controlling the end-effector pose, the robotic arm can avoid obstacles when performing trajectory planning, reducing the risk of collision and improving safety.

[0097] Based on the feature extraction module Col mentioned in S1.2 above, which consists of parallel sub-units, where any i-th sub-unit is composed of i-th convolutional layers (Conv) and i-th normalization layers (Stand) in sequence, compared with Inception-V4, this module can extract the geometric features of materials at a deeper level, providing more refined features to the attention mechanism module Ate, which facilitates Ate's further fusion of geometric features, thereby helping the Rein-Inception network model to more accurately identify different types of materials and improve grasping efficiency. The attention mechanism module Ate receives the refined features provided by the feature extraction module Col and further fuses the geometric features. The prediction module Ved is composed of parallel micro-units consisting of i-th convolutional layers (Conv), i-th pooling layers (Pool), i-th fully connected layers (Link), and 1 activation layer in sequence. After feature extraction and upsampling by each micro-unit, and then processing by the activation layer, the type features of the material can be predicted, as well as the positive value of the predicted type features. This method achieves a more accurate prediction of material type characteristics and the probability of correct prediction, overcoming the deficiency of missing prediction types in the original Inception-V4 network. This allows the robotic arm to choose whether to grasp the material based on its type characteristics, optimizing grasping decisions and improving grasping efficiency. The end effector can autonomously plan its path and perform grasping based on the material type characteristics predicted by the prediction module Ved. The feature extraction module Col processes the input feature image through parallel sub-units, with each sub-unit's convolutional layer Conv and normalization layer Stand processing sequentially to extract the geometric features of the material and provide fine-grained features for the attention mechanism module Ate. The prediction module Ved processes the input features through parallel micro-units, with each micro-unit's convolutional layer Conv, pooling layer Pool, and fully connected layer Link processing sequentially. Finally, the activation layer processes the data to predict the material type characteristics and the probability of correct prediction.

[0098] Specifically, the feature extraction module Col consists of parallel sub-units, each containing one convolutional layer (Conv) and one normalization layer (Stand). Compared to Inception-V4, it can extract material geometric features at a deeper level, providing fine features for the attention mechanism module Ate, thus helping the network model to more accurately identify material types and improve grasping efficiency. The prediction module Ved consists of parallel micro-units consisting of one convolutional layer (Conv), one pooling layer (Pool), one fully connected layer (Link), and one activation layer. Through feature extraction, upsampling, and activation layer processing of each micro-unit, it can predict material type characteristics and the probability of correct prediction, overcoming the deficiency of missing prediction types in the Inception-V4 network. This allows the robotic arm to select materials to grasp based on their characteristics, optimizing grasping decisions and improving efficiency. The robotic arm's end effector autonomously plans the grasping path and executes the grasping action based on the material type characteristics predicted by the prediction module Ved.

[0099] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0100] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A pipeline mechanical arm recognition and grabbing system based on a Rein-Inception network, comprising a mechanical arm base, an acquisition unit, a processing unit, a control unit and an end effector, the mechanical arm base is fixed on one side of a pipeline, the processing unit, the control unit and the end effector are fixedly installed in the mechanical arm base, the acquisition unit is fixedly arranged on one side of the end effector, and the acquisition unit is aligned with the pipeline together with the end effector of the mechanical arm, characterized in that: The acquisition unit is used to scan and acquire video of the material. The processing unit is used to extract and preprocess the high-definition 3D images of the material to obtain several 3D images of the material. The processing unit is used to identify the type, location, and orientation of the target material using a Rein-Inception network model. The control unit is used to drive the end effector of the robotic arm to perform corresponding actions based on the identification results of the processing unit. The system also includes an identification module and a grasping module. The implementation of the identification module specifically includes the following steps: ​ S1.1: The acquisition unit and processing unit respectively perform 3D image scanning and annotation of the material: S1.2: Establish the Rein-Inception network, which includes, in sequence: feature extraction module Col, attention mechanism module Ate, prediction module Ved, and end effector; S1.3: Constructing the loss function And train the Rein-Inception network; The grasping module controls the robotic arm to complete the grasping task based on the results of the recognition module; The specific steps of S1.1 are as follows: S1.1.1: The acquisition unit scans and acquires video of the material, and the processing unit extracts and preprocesses the 3D image of the material in the video to obtain a 3D image sequence of the material. The 3D image sequence A 3D image of any frame of material in the document is denoted as ; S1.1.2: For 3D images The location and type K_IN of the M materials are labeled. Obtain feature images with M location labels The types of the M materials K_IN include: metal K_M, plastic K_P, and wood K_W; Let feature image The Middle The true type characteristics of each material are ,and ∈{K_M, K_P, K_W}, i∈M; Let feature image The Middle The predicted type characteristics of each material are as follows: ,and ∈{K_M, K_P, K_W}; Let feature image The Middle The pose of each material is denoted as ,and ,in,( Indicates the first The material in the feature image coordinates, ( Indicates the first The material in the feature image The posture; The specific steps of S1.2 are as follows: S1.2.1: The feature extraction module Col consists of parallel... It consists of sub-units, where any _th _ ... Each subunit consists of Conv, a convolutional layer The normalized Stand layers are arranged sequentially. The feature image The input is fed into the feature extraction module Col, and then processed by each sub-unit. Feature extraction processing of each convolutional layer (Conv) yields a sub-feature set { | =1,2,…, },in, Indicates the first Sub-features output by each basic unit; { | =1,2,…, } passing through each sub-unit respectively A normalization layer (Stand) is used to obtain a normalized feature set. { },in, Indicates the first The normalized feature of the output of each basic unit; S1.2.2: The attention mechanism module Ate is composed of It consists of several randomly fused branches connected in parallel; The normalized feature set Enter them separately In the random fusion branches, among which, The feature random fusion branch is calculated using the formula. One fusion feature Thus by The fused feature set is obtained by randomly fusing the features along the branch. The formula for the fusion feature set is: in, Indicates the first The weight of each branch, Indicates the first The bias of each branch Indicates the first One activation feature, Indicates the first One activation feature, ; S1.2.3: The prediction module Ved is composed of... It consists of 3 basic units, where any 3 ... Each basic unit consists of Conv, a convolutional layer Pooling layer One fully connected layer (Link), one activation layer. Consisting in sequence; The fusion feature set The input is fed into the verification module Ved, and then passes through each basic unit. The processing of each convolutional layer yields a set of convolutional features. ,in, Indicates the first The first basic unit The convolutional features output by each convolutional layer; Each basic unit Each pooling layer performs upsampling to obtain an upsampled feature set. ,in, Indicates the first The first basic unit Upsampled features of the output of each pooling layer; Each basic unit Processing each fully connected layer Link yields a standard feature set. The standard feature set is passed through an activation layer. activation function The result after processing The predicted type characteristics of each material { }, and the first The probability that the actual type of a material is equal to the predicted type characteristic. ,in, Indicates the first The first basic unit Features of the output of a fully connected layer Link. Indicates the first The material was the first Types of predictions for each basic unit ,and ∈{K_M, K_P, K_W}, where the activation function is... The formula shown is: S1.2.4: End effector: Based on the material type characteristics predicted by the prediction module Ved, it autonomously plans the path and performs the grabbing; The specific steps of S1.3 are as follows: S1.3.1: Constructing the first using formula Loss function for each type of material The loss function formula is: in, It is the first The predicted characteristics of each material type For the first The probability that the predicted type of a material is correct. , For hyperparameters; S1.3.2: Constructing the total loss function using the formula The total loss function formula is as follows: S1.3.3: Iteratively train the Rein-Inception network using the SGD optimizer and calculate the total loss function. And adjust the network parameters based on the calculation results, when the total loss function When the convergence reaches 0, the material recognition model for the production line is obtained.

2. The pipeline robotic arm identification and grasping system based on Rein-Inception network according to claim 1, characterized in that: S1.2.3 further yields the first indivual by Prediction of basic units Corresponding types : Let K_IN[ ] indicates the first The types of materials, and K_IN[ ]={K_M, K_P, K_W}, when When =1, it means the first... Types of materials K_IN[ ] is a metal, i.e., K_IN[1]=K_M; when When =2, it means the first... Types of materials K_IN[ ] is plastic, that is, K_IN[2]=K_P, when When =3, it means the first... Types of materials K_IN[ ] represents wood, i.e., K_IN[3]=K_W.

3. The pipeline robotic arm identification and grasping system based on Rein-Inception network according to claim 1, characterized in that: The end effector of the robotic arm performs material gripping, specifically including the following steps: S2.1: The robotic arm's acquisition unit acquires the feature image. The Middle The pose of a material ; S2.2: The robotic arm's processing unit will process the data obtained in the previous step S2.

1. Feature image of a material Input it into the production line material identification model obtained in step S1.3.3, and obtain the first... The types of materials predicted ; S2.3: Determine the first... Types of materials Is it the type of target that the robotic arm is grasping? If yes, continue to the next step; otherwise, execute the previous step S2.

1. S2.4: Initialize the crawling path and the current pose of the end effector of the robotic arm. =[ Put it in ; S2.5: The processing unit of the robotic arm, according to the procedure described in S2.1... The pose of a material The pose to be transformed by the end effector of the computational robotic arm. ; S2.6: The control unit of the robotic arm drives the end effector of the robotic arm to change to the pose described in the previous step S2.

5. and the pose Put in ; S2.7: Determine the pose obtained in the previous step S2.

6. Is it related to the first The pose of a material If they overlap, then the aforementioned This is the grasping path of the end effector of the robotic arm. If they do not overlap, repeat steps S2.5 to S2.7 above.

4. The pipeline robotic arm identification and grasping system based on Rein-Inception network according to claim 2, characterized in that: The pose to be changed by the end effector of the current robotic arm in S2.5 The specific calculation methods include the following: S2.5.1: The robotic arm's data acquisition unit acquires the first... The pose of a material and the pose Converted to coordinates based on the robotic arm's coordinate system The conversion method is shown in the formula below; in, T and T represent the rotation matrix and translation vector from the coordinate system of the robotic arm's acquisition unit to the coordinate system of the robotic arm's base, respectively. S2.5.2: The first... The coordinates of a material are converted from the coordinates in the coordinate system of the robot arm base to the coordinates in the coordinate system with the end effector of the robot arm as the reference. The transformation method is as follows: S2.5.3: According to the first The coordinates of a material in the end effector coordinate system of the robotic arm and the current position of the end effector of the robotic arm. and posture The end effector of the robotic arm is generated using a formula to grasp the first... The pose change of a material , the described pose The formula is: + *( )} in, As a random factor, when calculating the pose to be transformed by the end effector of the robotic arm each time, It will be randomly assigned a value, and .

5. The pipeline robotic arm identification and grasping system based on Rein-Inception network according to claim 1, characterized in that: The feature extraction module Col processes the input feature image through parallel sub-units, with each sub-unit having a convolutional layer Conv and a normalization layer Stand, in turn to extract the geometric features of the material and provide fine features for the attention mechanism module Ate.

6. The pipeline robotic arm identification and grasping system based on Rein-Inception network according to claim 1, characterized in that: The prediction module Ved uses parallel micro-units. Each micro-unit has a convolutional layer (Conv), a pooling layer (Pool), and a fully connected layer (Link) that process the input features in sequence. Finally, after processing by the activation layer, it predicts the type of material and the probability that the predicted type of material is correct.

7. The pipeline robotic arm identification and grasping system based on Rein-Inception network according to claim 1, characterized in that: The end effector works by predicting the material type characteristics based on the prediction module Ved. The types of materials that can be grasped include wood, metal, and plastic. It autonomously plans its path and executes the grasping action. The grasping force and angle are adjusted according to the characteristics and requirements of the material to ensure that the material can be grasped stably.

Citation Information

Patent Citations

  • Pedestrian re-identification method based on global features and local features of attention mechanism

    CN110070073A

  • Cross-domain scene recognition method based on private information

    CN110084141A

  • Robot arm grabbing control system and method based on intelligent stereoscopic vision

    CN111823223A

  • Control device, method and equipment for pole-climbing detection robot

    CN117444953A