A deep learning-based part sorting method

Abstract

The present application is directed to the problems of high-precision and intelligent mechanical arm sorting of parts on an industrial production line, and discloses a part sorting method based on deep learning. The method constructs a part sorting model, introduces a CA attention mechanism in the Backbone network, and then extracts the feature grabbing point and the pose angle feature of the part image through the feature fusion layer. The best grabbing point and grabbing pose angle of the part are obtained by combining the convolution layer and the detection layer, and are transmitted to the mechanical arm after coordinate conversion, finally realizing real-time sorting of the part. The present application combines the grabbing point coordinates with the part pose angle, realizes fast, accurate and efficient sorting of the part, and improves the accuracy and efficiency of part sorting.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of automatic part sorting, and particularly relates to a part sorting method based on deep learning. BACKGROUND

[0002] The part sorting speed on the assembly line has an important influence on production efficiency, manual sorting is not only low in efficiency and high in cost, but also has problems such as single labor content, repetitive labor, and is not conducive to the physical and mental health of workers. Therefore, it is an urgent problem to be solved to realize automatic sorting of parts on the production line.

[0003] With the development of artificial intelligence and control technology, the robot arm has been widely used in the fields of intelligent sorting, garbage recycling, and intelligent assembly. However, the robot arm still faces certain challenges in grasping parts in a non-structured environment, and detecting objects and predicting grasping poses are key links for the success of the robot arm in grasping parts. Thanks to the rapid development of computer technology, computer vision technology based on deep learning has been closely combined with robot technology. If the above technology is applied to the automatic part sorting system on the production line, it can not only improve production efficiency and increase industrial output, but also release labor resources and reduce the labor intensity of workers. SUMMARY

[0004] The application discloses a part sorting method based on deep learning to solve the problems of high-precision and intelligent robot arm sorting of parts on an industrial production line. The method constructs a part sorting model, introduces a CA attention mechanism in the Backbone network, extracts the feature grasping points and pose angle features of the part image through a feature fusion layer, obtains the best grasping points and grasping pose angles of the part in combination with a convolution layer and a detection layer, and finally realizes real-time sorting of the part after the coordinates are converted and transmitted to the robot arm, thereby improving the accuracy and efficiency of part sorting.

[0005] The application adopts the following technical solutions:

[0006] A part sorting method based on deep learning adopts the following steps:

[0007] Step 1: Collecting part images and determining the best grasping elements A depth camera is used to collect images P of the upward surface of part i on a static conveying belt directly below the depth camera ij , i is the label of all kinds of parts involved in the production line, i is an integer greater than or equal to 30, j is the number of different upward surfaces of a certain part placed on the conveying belt, j is an integer greater than or equal to 1; the best spatial coordinates of the mechanical arm grasping part i on the j surface are (x ij ,y ij ), z ij , wherein x ij and yij are the horizontal position of the centroid of part i mapped to the horizontal and vertical coordinates in image P ij , respectively; the coordinate system of image P ij is a pixel coordinate system: the upper left corner of image P ij is the coordinate origin O UV ; the direction horizontally to the right of image P ij is the x-axis, and the direction vertically downward perpendicular to the x-axis is the y-axis; z ij is the distance from the camera to the horizontal plane where the centroid of part i is located when the j surface of part i is upward; the optimal pose angle of the robot arm when grabbing part i when the j surface of part i is upward is θ ij ;

[0008] Step 2: Labeling the feature grabbing points and pose angle information of the part in image P ij in the pixel coordinate system using the labelme labeling software Label the json label of the feature grabbing points of the part in image P ij : [label ij , x ij , y ij ], wherein label ij is the character recognition label of the j surface of part i; Label the yaml label of the pose angle of the part in image P ij using the roLabelImg labeling software: [class i , cx ij , cy ij , w ij , h ij , θ ij ], wherein class i is the kind recognition label of part i, and a rectangle R' is constructed in image P ij , the four edges of R' are parallel to the x-axis and y-axis of the pixel coordinate system of image P ij , and the size of R' is the same as the size of the smallest rectangle that encloses part i, and R is obtained by rotating R', so that R is the smallest rectangle that encloses part i, w ij and h ij are the width and height of R, respectively, wherein w ij is the length of the edge of R' parallel to the x-axis; cx ij and cy ij are the center coordinates of R, and θ ij is the angle between w ij of R and the x-axis in the opposite direction, 0°≤θ ij <180°;

[0009] Step 3: Data label conversion of feature points and pose angle Convert the json label in step 2 to a feature point data label: [label ij , x ij , yij ,lw ij ,lh ij ], where lw ij lh ij For pixel values ​​of the same size, 10 ≤ lw ij ≤30, 10≤lh ij ≤30; Convert the YAML tags in step 2 into attitude angle data tags: [(x ij1 ,y ij1 ),(x ij2 ,y ij2 ),(x ij3 ,y ij3 ),(x ij4 ,y ij4 ),θ ij ,class i ,z ij ], where (x ij1 ,y ij1 ), (x ij2 ,y ij2 ), (x ij3 ,y ij3 ), (x ij4 ,y ij4 The images P and P represent the images of part i with surface j facing upwards. ij The coordinates of the four vertices of R.

[0010] When 0°≤θ ij When ≤90°,

[0011] x ij1 =cx ij -(w ij / 2)×cosθ ij -(h ij / 2)×sinθ ij y ij1 =cy ij -(w ij / 2)×sinθ ij +(h ij / 2)×cosθ ij ,

[0012] x ij2 =cx ij -(w ij / 2)×cosθ ij +(h ij / 2)×sinθ ij y ij2 =cy ij -(w ij / 2)×sinθ ij -(h ij / 2)×cosθ ij ,

[0013] x ij3 = cx ij +(w ij / 2)×cosθ ij +(h ij / 2)×sinθ ij , y ij3 = cy ij +(w ij / 2)×sinθ ij -(h ij / 2)×cosθ ij ,

[0014] x ij4 = cx ij +(w ij / 2)×cosθ ij -(h ij / 2)×sinθ ij , y ij4 = cy ij +(w ij / 2)×sinθ ij +(h ij / 2)×cosθ ij ;

[0015] When 90° < θ ij < 180°, θ ij ' = θ ij - 90°,

[0016] x ij1 = cx ij -(w ij / 2)×cosθ ij '-(h ij / 2)×sinθ ij , y ij1 = cy ij -(w ij / 2)×sinθ ij '+(h ij / 2)×cosθ ij ',

[0017] x ij2 = cx ij -(w ij / 2)×cosθ<​​​​​​​​​ / 2)×sinθ ij ′-(h ij / 2)×cosθ ij ′,

[0018] x ij3 =cx ij +(w ij / 2)×cosθ ij ′+(h ij / 2)×sinθ ij ′,y ij3 =cy ij +(w ij / 2)×sinθ ij ′-(h ij / 2)×cosθ ij ′,

[0019] x ij4 =cx ij +(w ij / 2)×cosθ ij ′-(h ij / 2)×sinθ ij ′,y ij4 =cy ij +(w ij / 2)×sinθ ij ′+(h ij / 2)×cosθ ij ′;

[0020] Image P of the part obtained from all steps 1 ij The parts sorting model dataset consists of the corresponding feature point data labels and attitude angle data labels. The parts sorting model dataset is divided into training set and validation set according to an 8:2 ratio.

[0021] Step 4: Establish a parts sorting model to train the P in the set. ij The image is used as input, and the class values ​​in the image's corresponding feature point data labels and attitude angle data labels are used as input. i x ij y ij z ij and θ ijFor output, a part sorting model is trained; the part sorting model comprises a feature extraction network and an output layer, wherein the feature extraction network further comprises a BackBone network and a feature fusion layer, a coordinate attention mechanism is particularly introduced in the BackBone network, the coordinate attention mechanism is used for further feature extraction of part feature information, the feature fusion layer comprises four groups of sub-modules, two groups of the sub-modules are composed of a CBS convolution module, an up-sampling module, a concat module and a C3CBAM module, the other two groups of the sub-modules are composed of a CBS convolution module, a concat module and a C3CBAM module, different dimension feature maps output by the BackBone network are input into the feature fusion layer for feature fusion, P ij After the image is subjected to the feature extraction network, part feature maps are obtained; the output layer further comprises a convolution layer, a part posture angle detection layer and a feature point detection layer; the part feature maps output by the feature extraction network are input into the output layer, and class i , x ij , y ij , z ij and θ ij are output after classification of the convolution layer, the part posture angle detection layer and the feature point detection layer; related parameters of the part sorting model: batchsize = 16 or 32, epoch = 200-400, momentum = 0.732-0.895, 0.01≤lr≤0.02, 2≤r≤8, wherein r is an even integer;

[0022] In step 5, the x ij and y ij output by the part sorting model are subjected to coordinate conversion respectively to obtain corresponding coordinates x ij ' and y ij ' in a world coordinate system, and then class i , x ij ', y ij ', z ij , θ ij are transmitted to a mechanical arm, and finally the mechanical arm sorts the parts.

[0023] Compared with the prior art, the part sorting model has the following beneficial effects:

[0024] (1) The part sorting model is used for accurately judging part categories, grabbing points and posture angles, combining the grabbing point coordinates with the part posture angles, and realizing rapid, accurate and efficient sorting of the parts;

[0025] (2) The coordinate attention mechanism (CA) is introduced into the BackBone network, and the effect of part image posture angle detail feature extraction is further improved;

[0026] (3) The Backbone network, the feature fusion layer and the convolution layer are combined to realize detailed extraction of part grabbing features. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A part sorting detection method flowchart of the present application;

[0028] Figure 2 An image P ij of the present application in a pixel coordinate system;

[0029] Figure 3 A network structure diagram of a part sorting model of the present application. DETAILED DESCRIPTION

[0030] The present application will be further described below in combination with the drawings.

[0031] A part sorting method based on deep learning, as shown in the figure, adopts the following steps: Figures 1-3

[0032] Step 1: Collecting part images and determining the best grabbing element ij A depth camera is used to collect the image P ij of the upward surface of the part i on the static conveying belt directly below the depth camera, i is the label of all kinds of parts involved in the production line, i is an integer greater than or equal to 30, j is the number of different upward surfaces of a certain part on the conveying belt, j is an integer greater than or equal to 1; the best spatial coordinates of the mechanical arm grabbing the part i on the j upward surface are (x ij ,y ij ), z ij , wherein x ij and y ij are the horizontal position of the mass center of the part i mapped into the horizontal and vertical coordinates of the image P ij , the coordinate system of the image P ij is a pixel coordinate system: the upper left corner of the P UV image is the coordinate origin O ij , the horizontal right direction of the image P ij is the x-axis, and the direction perpendicular to the x-axis downward is the y-axis; z ij is the distance from the camera to the horizontal plane where the mass center of the part i is located when the j upward surface of the part i is grabbed by the mechanical arm; the best attitude angle of the mechanical arm grabbing the part i on the j upward surface is θ ij .

[0033] Step 2: Labeling the feature grabbing points and attitude angle information of the part in P ij in the pixel coordinate system ij using the labelme labeling software to label the json label of the part feature grabbing points in P[Label , xij ,y ij ], wherein label ij is a character recognition label of the surface of the i-th part; the P ij ixel coordinates of the i-th part are labeled by the roLabelImg software; the yaml label of the pose angle of the i-th part is: [class i ,cx ij ,cy ij ,w ij ,h ij ,θ ij ], wherein class i is a kind recognition label of the i-th part, a rectangle R' is constructed in the image P ij , four edges of R' are parallel to the x-axis and the y-axis of the pixel coordinate system of the image P ij , and the size of R' is the same as the size of the smallest rectangle that frames the i-th part, R' is rotated to obtain R, so that R is the smallest rectangle that frames the i-th part, w ij and h ij are the width and the height of R, respectively, wherein w ij is the length of the edge of R' that is parallel to the x-axis; cx ij and cy ij are the center coordinates of R, and θ ij is the angle between w ij of R and the x-axis in the opposite direction, 0°≤θ ij <180°;

[0034] Step 3: Data label conversion of feature points and pose angles In step 2, the json label is converted into a feature point data label: [label ij ,x ij ,y ij ,lw ij ,lh ij ], wherein lw ij and lh ij are pixel values of the same size, 10≤lw ij ≤30, and 10≤lh ij ≤30; and the yaml label in step 2 is converted into a pose angle data label: [(x ij1 ,y ij1 ),(x ij2 ,y ij2 ),(x ij3 ,y ij3 ),(x ij4 ,y ij4 ),θ ij ,class i ,z ij ], wherein (x ij1 ,y ij1 ), (xij2 ,y ij2 ), (x ij3 ,y ij3 ), (x ij4 ,y ij4 The images P and P represent the images of part i with surface j facing upwards. ij The coordinates of the four vertices of R.

[0035] When 0°≤θ ij When ≤90°,

[0036] x ij1 =cx ij -(w ij / 2)×cosθ ij -(h ij / 2)×sinθ ij y ij1 =cy ij -(w ij / 2)×sinθ ij +(h ij / 2)×cosθ ij ,

[0037] x ij2 =cx ij -(w ij / 2)×cosθ ij +(h ij / 2)×sinθ ij y ij2 =cy ij -(w ij / 2)×sinθ ij -(h ij / 2)×cosθ ij ,

[0038] x ij3 =cx ij +(w ij / 2)×cosθ ij +(h ij / 2)×sinθ ij y ij3 =cy ij +(w ij / 2)×sinθ ij -(h ij / 2)×cosθ ij ,

[0039] x ij4 =cx ij +(w ij / 2)×cosθ ij -(h ij(w / 2) x sin θ ij , y ij4 = cy ij +(w ij / 2) x sin θ ij +(h ij / 2) x cos θ ij ;

[0040] when 90° < θ ij < 180°, θ ij ' = θ ij - 90°,

[0041] x ij1 = cx ij -(w ij / 2) x cos θ ij '-(h ij / 2) x sin θ ij ', y ij1 = cy ij -(w ij / 2) x sin θ ij '+(h ij / 2) x cos θ ij ',

[0042] x ij2 = cx ij -(w ij / 2) x cos θ ij '+(h ij / 2) x sin θ ij ', y ij2 = cy ij -(w ij / 2) x sin θ ij '-(h ij / 2) x cos θ ij ',

[0043] x ij3 = cx ij +(w ij / 2) x cos θ ij '+(h ij / 2) x sin θ ij ', y ij3 = cy ij +(w ij / 2) x sin θ ij '-(h ij / 2) x cos θ ij ',

[0044] x ij4 = cx ij +(wij (w / 2) x cos θ ij h ij (w / 2) x sin θ ij y ij4 = cy ij +(w ij / 2) x sin θ ij h ij / 2) x cos θ ij ;

[0045] images P of the parts obtained by all step 1 ij and their corresponding feature point data labels and pose angle data labels constitute a part sorting model data set, and the part sorting model data set is divided into a training set and a verification set according to a quantity ratio of 8:2;

[0046] Step 4 establishes a part sorting model with P ij images in the training set as input, and class i , x ij , y ij , z ij and θ ij in the image corresponding feature point data labels and pose angle data labels as output, to train the part sorting model; the part sorting model includes a feature extraction network and an output layer, wherein the feature extraction network further includes a BackBone network and a feature fusion layer, a coordinate attention mechanism is specially introduced in the BackBone network, further feature extraction of part feature information is realized through the coordinate attention mechanism, the feature fusion layer includes 4 groups of sub-modules, wherein 2 groups are composed of a CBS convolution module, an up-sampling module, a concat module and a C3CBAM module, and the other 2 groups are composed of a CBS convolution module, a concat module and a C3CBAM module, different dimension feature maps output by the BackBone network are input into the feature fusion layer for feature fusion, and part feature maps are obtained after the P ij images pass through the feature extraction network; the output layer further includes a convolution layer, a part pose angle detection layer and a feature point detection layer; the part feature maps output by the feature extraction network are input into the output layer, and class i , x ij , y ij , z ij and θ ij are output after classification by the convolution, part pose angle detection layer and feature point detection layer; the related parameters of the part sorting model: batchsize = 16 or 32, epoch = 200-400, momentum = 0.732-0.895, 0.01≤lr≤0.02, 2≤r≤8, wherein r is an even integer;

[0047] Step 5: The robotic arm performs coordinate transformation to convert the x-coordinates output by the parts sorting model. ij and y ij After coordinate transformation, the corresponding x coordinates in the world coordinate system are obtained. ij ′ and y ij ', then class i x ij ′,y ij ′,z ij θ ij The data is transmitted to the robotic arm, which then sorts the parts.

[0048] Example 1

[0049] A deep learning-based parts sorting method comprises the following steps:

[0050] Step 1: Acquire part images and determine optimal gripping elements. Use a depth camera to acquire images of all different surfaces (j) of part i, located directly below the depth camera, on a stationary conveyor belt. ij Let i be the number of all 35 parts involved in the production line, i = 1, 2, ..., 35, and j be the number of different upper surfaces that a certain part has when placed on the conveyor belt. Among the 35 parts, the cube structure has the fewest upper surfaces: 1, and the irregular hexahedral part has the most upper surfaces: 6. The upper surface refers to the image P captured by the depth camera. ij The corresponding surface, in this embodiment, is adopted. Figure 2 The screw shown, with part number i=4, has three upper surfaces: when j=1, both the screw head and tail are in contact with the conveyor belt, and the central axis of the screw forms an acute angle with the conveyor belt; when j=2, the screw head is placed on the conveyor belt, and the central axis of the screw is perpendicular to the conveyor belt; when j=3, the screw tail is placed on the conveyor belt, and the central axis of the screw is perpendicular to the conveyor belt. Figure 2 As shown, the optimal spatial coordinates for the robotic arm to grasp the screw when it is facing upwards on the surface where j=1 are (x 41 ,y 41 ),z 41 , where x 41 and y 41 The horizontal position of the screw's centroid is mapped to image P. 41 x and y coordinates in the figure 41 =579, y 41 =641, z 41 Let z be the distance from the camera to the horizontal plane containing the centroid of the screw when the surface of the screw (j=1) is facing upwards. 41 =61 cm; the optimal gripping angle for the robotic arm when the screw with j=1 surface is facing upwards is θ. 41 ;

[0051] Step 2: Label P in the pixel coordinate system41 The feature capture points and attitude angle information of the parts were annotated using labelme annotation software. 41 JSON labels for feature capture points of parts: [label 41 ,x 41 ,y 41 ], where label 41 For the character identification label of screw j=1, label 41 ='screw1'; Use roLabelImg annotation software to annotate P 41 YAML tags for part attitude angles: [class4,cx 41 ,cy 41 ,w 41 ,h 41 ,θ 41 ],like Figure 2 As shown, class4 is the screw type identification label, class4=4, w 41 and h 41 Image P is shown when the surface of the screw j=1 is facing upwards. 41 The width and height of the minimum rectangular frame R that holds the screw in the middle frame, where w 41 =486,h 41 =301; cx 41 and cy 41 Let cx be the center coordinate of R when the surface of screw j=1 is facing upwards. 41 =612, cy 41 =650, θ 41 When the surface of screw j=1 is facing upwards, R's w 41 The angle θ with respect to the opposite direction of the x-axis 41 =104°;

[0052] Step 3: Data Label Conversion for Feature Points and Attitude Angles. The JSON labels from Step 2 are converted into feature point data labels: [label] 41 ,x 41 ,y 41 ,lw 41 ,lh 41 ], where lw 41 lh 41 For pixel values ​​of the same size, lw 41 =15,lh 41 =15; for example Figure 2 As shown, the YAML tags in step 2 are converted into attitude angle data tags: [(x 411 ,y 411 ),(x 412 ,y 412 ),(x 413 ,y413 ),(x 414 ,y 414 ),θ 41 ,class4,z 41 ], where (x 411 ,y 411 ), (x 412 ,y 412 ), (x 413 ,y 413 ), (x 414 ,y 414 The images P and P represent the screw with the surface facing upwards when j=1. 41 The coordinates of the four vertices of R.

[0053] Because θ 41 = 104°, therefore, θ ij ′=104°-90°=14°,

[0054] x 411 =cx 41 -(w 41 / 2)×cosθ 41 ′-(h 41 / 2)×sinθ 41 =340,

[0055] y 411 =cy 41 -(w 41 / 2)×sinθ 41 ′+(h 41 / 2)×cosθ 41 =738

[0056] x 412 =cx 41 -(w 41 / 2)×cosθ 41 ′+(h 41 / 2)×sinθ 41 =413,

[0057] y 412 =cy 41 -(w 41 / 2)×sinθ 41 ′-(h 41 / 2)×cosθ 41 ′=445,

[0058] x 413 =cx 41 +(w 41 / 2)×cosθ 41 ′+(h 41 / 2)×sinθ41 ′ = 885,

[0059] y 413 ′ = cy 41 +(w 41 / 2) x sin θ 41 ′ -(h 41 / 2) x cos θ 41 ′ = 563,

[0060] x 414 ′ = cx 41 +(w 41 / 2) x cos θ 41 ′ -(h 41 / 2) x sin θ 41 ′ = 812,

[0061] y 414 ′ = cy 41 +(w 41 / 2) x sin θ 41 ′ +(h 41 / 2) x cos θ 41 ′ = 855;

[0062] The images P ij of the parts obtained by all step 1 collection, and the corresponding feature point data labels and pose angle data labels thereof constitute a part sorting model data set, 6420 images are collected for 35 images, the part sorting model data set is divided into a training set and a verification set according to an 8:2 quantity ratio, the training set includes 5136 samples, and the verification set includes 1284 samples;

[0063] Step 4 establishes a part sorting model as shown in Figure 3 , takes the P ij images in the training set as input, and takes the class i , x ij , y ij , z ij and θ ij in the image corresponding feature point data labels and pose angle data labels as labels.For output, a parts sorting model is trained. The parts sorting model includes a feature extraction network and an output layer. The feature extraction network further includes a Backbone network and a feature fusion layer. The Backbone network incorporates a coordinate attention mechanism to further extract features from the parts. The feature fusion layer consists of four sub-modules: two groups comprise a CBS convolutional module, an upsampling module, a concat module, and a C3CBAM module; the other two groups also comprise a CBS convolutional module, a concat module, and a C3CBAM module. The CBS module uses a lightweight Ghost convolution. Feature maps of different dimensions output from the Backbone network are input to the feature fusion layer for feature fusion. ij The image is processed by a feature extraction network to obtain a part feature map; the output layer includes a convolutional layer, a feature point detection layer, and a part pose angle detection layer; the part feature map output by the feature extraction network is input to the output layer, and after classification by the convolutional, feature point detection, and part pose angle detection layers, the class is output. i x ij y ij , z ij and θ ij The relevant parameters of the parts sorting model are: batchsize = 16, epoch = 300, momentum = 0.895, lr = 0.015, r = 6.

[0064] Step 5: The robotic arm performs coordinate transformation to convert the x-coordinates output by the parts sorting model. 41 and y 41 After coordinate transformation, the corresponding x coordinates in the world coordinate system are obtained. 41 ′ and y 41 ', then class4, x 41 ′,y 41 ′,z 41 θ 41 The data is transmitted to the robotic arm, which then grasps the part.

[0065] like Figure 3 As shown, in step 4, a CA attention mechanism is added to the last layer of the Backbone network. First, positional information is incorporated into channel attention. To obtain attention in both the width and height of the image, precise positional information is encoded, and features are aggregated along both spatial dimensions. The feature map x output from the last layer of the Backbone network is used as input, and two pooling kernels (H,1) and (1,W) are used to encode each channel along the horizontal and vertical directions, respectively. The output at height h of channel c can be expressed as:

[0066]

[0067] Similarly, the output of the c channel at width w can be represented as:

[0068]

[0069] After the conversion by information embedding, in order to better utilize the precise information with global receptive field generated by coordinate information embedding, a second conversion method is added to generate coordinate attention:

[0070] The position information output in the horizontal and vertical directions is embedded and spliced, and is subjected to convolution, normalization and nonlinear activation processing, as shown in equation (3),

[0071] f = δ (F1([Z h ,Z W ])) (3)

[0072] In the above equation, F1 is a convolution function, δ is a nonlinear activation function sigmoid function, which can effectively reduce the model complexity and reduce the computational overhead, and f is the intermediate feature mapping of independent coding in two directions, which is decomposed into two independent tensors f h and f w along the spatial dimension, and then two 1x1 convolutions F h and F w are used to transform it to the same number of channels as the input features, and then nonlinear activation is performed as shown in equations (4) and (5).

[0073] g h = δ (F h (f h )) (4)

[0074] g w = δ (F w (f w )) (5)

[0075] Finally, the outputs g h and g w are respectively subjected to graph expansion, and the coordinate attention module outputs y c (i,j) as shown in equation (6).

[0076]

[0077] As Figure 3As shown in the Backbone network, the original main layer CBS module is replaced by a Ghost convolution module. The first step of the Ghost convolution operation is similar to the ordinary convolution, generating a small number of feature maps, and then applying a series of linear operations to further generate similar feature maps. Finally, the two sets of feature map information are spliced to obtain all the feature information. In the case of not changing the output feature mapping size, the total number of parameters required by the Ghost module and the computational complexity are reduced, effectively optimizing the network parameters.

[0078] As shown in the Backbone network, the original main layer CBS module is replaced by a Ghost convolution module. The first step of the Ghost convolution operation is similar to the ordinary convolution, generating a small number of feature maps, and then applying a series of linear operations to further generate similar feature maps. Finally, the two sets of feature map information are spliced to obtain all the feature information. In the case of not changing the output feature mapping size, the total number of parameters required by the Ghost module and the computational complexity are reduced, effectively optimizing the network parameters. Figure 3 As shown in the Backbone network, the original main layer CBS module is replaced by a Ghost convolution module. The first step of the Ghost convolution operation is similar to the ordinary convolution, generating a small number of feature maps, and then applying a series of linear operations to further generate similar feature maps. Finally, the two sets of feature map information are spliced to obtain all the feature information. In the case of not changing the output feature mapping size, the total number of parameters required by the Ghost module and the computational complexity are reduced, effectively optimizing the network parameters.

[0079] As shown in the Backbone network, the original main layer CBS module is replaced by a Ghost convolution module. The first step of the Ghost convolution operation is similar to the ordinary convolution, generating a small number of feature maps, and then applying a series of linear operations to further generate similar feature maps. Finally, the two sets of feature map information are spliced to obtain all the feature information. In the case of not changing the output feature mapping size, the total number of parameters required by the Ghost module and the computational complexity are reduced, effectively optimizing the network parameters. Figure 3 As shown in the Backbone network, the original main layer CBS module is replaced by a Ghost convolution module. The first step of the Ghost convolution operation is similar to the ordinary convolution, generating a small number of feature maps, and then applying a series of linear operations to further generate similar feature maps. Finally, the two sets of feature map information are spliced to obtain all the feature information. In the case of not changing the output feature mapping size, the total number of parameters required by the Ghost module and the computational complexity are reduced, effectively optimizing the network parameters.

[0080] The CSL expression is as follows:

[0081]

[0082] Where g(x) is a window function; r is the radius of the window function; and θ represents the current boundary box angle. In this algorithm, a Gaussian function is used as the window function.

[0083] After the real box angle label is processed by the CSL, the binary cross-entropy loss function is used for loss calculation to accelerate the model convergence. The angle loss function expression is as follows:

[0084] P i j =Sigmoid(x i ) (8)

[0085]

[0086] Where N represents the total number of categories, x i is the predicted value of the current category, and where p is the probability of the current class after passing through the activation function, is the real value (0 or 1) of the current class, and L cls is the angle loss. After adding the angle classification, the number of detection head channels is: (5+nc+180)×na), nc=35, na=3.

[0087] The CIoU_Loss (Generalized Intersection over Union Loss) function is used to estimate the recognition loss of the detection target rectangular frame of the detection layer.

[0088] The CIoU loss function is L CIoU :

[0089]

[0090] In the formula, a is a balance parameter:

[0091]

[0092] The parameter is used to measure the consistency of the aspect ratio,

[0093]

[0094] In the formula, p is the distance between the center points of the predicted frame and the real frame, b,b gt are the real frame and the predicted frame, respectively; w, h are the width and height of the predicted frame, respectively; w gt , h gt are the width and height of the real frame, respectively.

[0095] The part image P ij collected by the depth camera in step 1 has a size of 640×640 pixels. In the training, the data set is preprocessed by flipping, scaling, etc. The adam optimizer is used for training optimization; the feature map sizes output by the three detection layers in step 4 are 128×128, 64×64, and 32×32, respectively.

[0096] The part sorting accuracy index is as follows:

[0097]

[0098]

[0099]

[0100]

[0101] In the formula, P is the accuracy rate, R is the recall rate, TP is the number of correctly detected samples, FP is the number of false detection samples, FN is the number of missed detection samples, AP represents the area under the P-R curve, the accuracy rate and the recall rate are used as comprehensive indexes, mAP represents the average value of AP of each category, represents the average good and bad degree in all categories, Q is the total number of all kinds of parts involved in the production line, Q=i.

[0102] The part sorting method based on deep learning can realize real-time intelligent high-precision part sorting detection on parts of different sizes and different specifications, realizes high-precision positioning of part coordinates and feature grabbing points, realizes part attitude angle prediction and part category prediction accurately, has the advantages of fast detection speed, strong stability, low consumption cost, easy operation and the like, compared with other part detection, has lower model parameter quantity, faster detection speed and higher positioning accuracy, and can meet the real-time and unmanned requirements of part intelligent sorting under the background of intelligent manufacturing.

Claims

1. A deep learning based part sorting method, characterized by: The following steps are employed: Step 1 Collecting part image and determining the best grabbing element Collecting the image of part i on the static conveyer right below the depth camera with the surface of part i facing upward P ij , i is the label of all kinds of parts involved in the production line, i is an integer greater than or equal to 30, j is the number of different upper surfaces of a certain kind of part when placed on the conveyer, j is an integer greater than or equal to 1; the best spatial coordinates of the mechanical arm grabbing part i with the surface j facing upward is (x ij ,y ij ), z ij , wherein x ij and y ij are the horizontal position of the mass center of part i mapped into the horizontal and vertical coordinates of image P ij , the coordinate system of image P ij is the pixel coordinate system: taking the upper left corner of image P ij as the coordinate origin O UV , taking the horizontal right direction of image P ij as the x-axis, and taking the direction perpendicular to the x-axis downward as the y-axis; z ij is the distance from the camera to the horizontal plane where the mass center of part i is located when the surface j of part i faces upward; the best attitude angle of the mechanical arm grabbing part i with the surface j facing upward is θ ij ; Step 2 annotates P in the pixel coordinate system ij The feature grabbing point and pose angle information of the middle part are annotated by labelme annotation software ij The json label of the feature grabbing point of the middle part: [label ij ,x ij ,y ij ], wherein label ij is the character recognition label of the i part j surface; the pose angle of the middle part is annotated by roLabelImg annotation software ij The yaml label of the pose angle: [class i ,cx ij ,cy ij ,w ij ,h ij ,θ ij ], wherein class i is the kind recognition label of the i part, and a rectangle R' is constructed in the image P ij The four edges of R' are respectively parallel to the x and y axes of the pixel coordinate system of the image P ij , and the size of R' is the same as the size of the smallest rectangle that frames the i part, and R is obtained by rotating R', so that R is the smallest rectangle that frames the part i, w ij and h ij are the width and height of R respectively, wherein w ij is the edge length of the edge of R' parallel to the x axis; cx ij and cy ij are the center coordinates of R, and θ ij is the included angle of w ij of R and the x axis in the opposite direction, 0°≤θ ij <180°. Step 3 Data label conversion of feature points and attitude angle The json label in step 2 is converted into feature point data label: [label ij ,x ij ,y ij ,lw ij ,lh ij ] wherein lw ij , lh ij are pixel values of the same size, 10≤lw ij ≤30, 10≤lh ij ≤30; The yaml label in step 2 is converted into attitude angle data label: [(x ij1 ,y ij1 ),(x ij2 ,y ij2 ),(x ij3 ,y ij3 ),(x ij4 ,y ij4 ), θ ij , class i , z ij ] wherein (x ij1 ,y ij1 ), (x ij2 ,y ij2 ), (x ij3 ,y ij3 ), (x ij4 ,y ij4 ) are respectively 4 vertex coordinates of R in image P ij when i part j surface is upward, when 0° < θ < 90°, ij when 0° < θ < 90°, x ij1 = cx ij - (w ij / 2) x cos θ ij - (h ij / 2) x sin θ ij , y ij1 = cy ij - (w ij / 2) x sin θ ij + (h ij / 2) x cos θ ij , x ij2 = cx ij - (w ij / 2) x cos θ ij + (h ij / 2) x sin θ ij , y ij2 = cy ij - (w ij / 2) x sin θ ij - (h ij / 2) x cos θ ij , x ij3 = cx ij + (w ij / 2) × cos θ ij + (h ij / 2) × sin θ ij , y ij3 = cy ij + (w ij / 2) × sin θ ij - (h ij / 2) × cos θ ij , x ij4 = cx ij + (w ij / 2) × cos θ ij - (h ij / 2) × sin θ ij , y ij4 = cy ij + (w ij / 2) × sin θ ij + (h ij / 2) × cos θ ij ; when 90° < θ < 180°, θ ij when 90° < θ < 180°, θ ij ′ = θ ij - 90°, x ij1 = cx ij -(w ij / 2) x cos θ ij ′+(h ij / 2) x sin θ ij ′, y ij1 = cy ij -(w ij / 2) x sin θ ij ′+(h ij / 2) x cos θ ij ′, x ij2 = cx ij - (w ij / 2) x cos θ ij ' + (h ij / 2) x sin θ ij ', y ij2 = cy ij - (w ij / 2) x sin θ ij ' - (h ij / 2) x cos θ ij ', x ij3 = cx ij +(w ij / 2) x cos θ ij ′+(h ij / 2) x sin θ ij ′, y ij3 = cy ij +(w ij / 2) x sin θ ij ′-(h ij / 2) x cos θ ij ′, x ij4 = cx ij + (w ij / 2) x cos θ ij ′ - (h ij / 2) x sin θ ij ′, y ij4 = cy ij + (w ij / 2) x sin θ ij ′ + (h ij / 2) x cos θ ij ′; Images P of the parts obtained from all step 1 ij and their corresponding feature point data labels and pose angle data labels constitute a part sorting model dataset, and the part sorting model dataset is divided into a training set and a validation set according to an 8:2 quantity ratio. Step 4 establishes a part sorting model to train P ij The image is input, and class i , x ij , y ij , z ij and θ ij are output, and the part sorting model is trained; the part sorting model comprises a feature extraction network and an output layer, wherein the feature extraction network further comprises a BackBone network and a feature fusion layer, a coordinate attention mechanism is particularly introduced in the BackBone network, further feature extraction of part feature information is realized through the coordinate attention mechanism, the feature fusion layer comprises four groups of sub-modules, two groups of which are composed of a CBS convolution module, an up-sampling module, a concat module and a C3CBAM module, and the other two groups are composed of a CBS convolution module, a concat module and a C3CBAM module, different dimension feature maps output by the BackBone network are input into the feature fusion layer for feature fusion, and P ij The image is input into the feature extraction network to obtain a part feature map; the output layer further comprises a convolution layer, a part attitude angle detection layer and a feature point detection layer; the part feature map output by the feature extraction network is input into the output layer, and class i , x ij , y ij , z ij and θ ij are output after classification by the convolution layer, the part attitude angle detection layer and the feature point detection layer; related parameters of the part sorting model: batchsize=16 or 32, epoch=200-400, momentum=0.732-0.895, 0.01≤lr≤0.02, 2≤r≤8, wherein r is an even integer; Step 5 Coordinate transformation of mechanical arm grabbing The x ij and y ij coordinates output by the part sorting model are respectively converted to obtain corresponding coordinates x ij ′ and y ij ′ in the world coordinate system, and then class i , x ij ′, y ij ′, z ij , θ ij are transmitted to the mechanical arm, and finally the mechanical arm sorts the parts.

Images