A part sorting and recognition system and method based on deep learning

Abstract

The present invention discloses a part sorting and recognition system and method based on deep learning, which relates to the fields of computer vision and part sorting. The system includes: a training module and an application module. A model is established in the training module, and the model is applied to the application module to obtain a recognition result and output the recognition result. The training module includes: a synthetic image production module, an actual image acquisition module, a data set generation module, and a deep learning module. Among them, the deep learning module is connected to the data set generation module, receives a training set and a validation set, and constructs a deep learning model, including an object recognition network and a domain adaptation network. The object recognition network is used for feature extraction, object classification, and generation of candidate boxes; the domain adaptation network is used for aligning the features of images in the source domain and the target domain.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and part sorting, and in particular to a part sorting and recognition system and method based on deep learning. Background Art

[0002] In recent years, with the substantial increase in the demand for production capacity, the process of industrial intelligence has gradually accelerated, and the use of industrial sorting equipment has gradually entered major fields led by the logistics industry. The most traditional sorting method is to use an assembly line plus manual sorting, which is a huge consumption of human resources and difficult to ensure the continuity of sorting quality. Currently, most sorting equipment uses a sensor-based sorting method, judging the size of an object through an infrared sensor and the weight of an object through an acceleration sensor, etc. This method often has certain limitations in the objects to be recognized and can only recognize relatively general objects.

[0003] With the development of artificial intelligence, deep learning methods have also begun to be applied to the field of part sorting. The part sorting method based on deep learning can handle more complex scenarios and has no special requirements for the shape and style of the parts themselves.

[0004] In the Chinese invention patent application "Industrial Part Recognition and Sorting Method, Medium and System Based on FD-Densenet Network" (publication number CN112347863), An Kang et al. used a target detection network based on FD-Densenet to recognize parts to be sorted. The present invention realizes the integration of part recognition and production line sorting, improves the sorting efficiency, and is a lightweight model, which is more convenient for the configuration and use of industrial equipment. However, during the network training process, it is necessary to perform a preliminary cropping of parts with different sizes: enlarging or reducing the effective area, and the process of image acquisition is relatively complex.

[0005] In the Chinese invention patent application "Deep Learning-Based Small Part Sorting Method and Its Sorting System" (publication number CN112170233A), Zhang Liguo et al. used the YOLOv4 target detection network to train and recognize part objects, which can realize the sorting of small parts, improve the range of part categories and styles that can be recognized on the production line, and mainly achieve a certain accuracy and speed in sorting for irregular complex parts. However, this invention trains and tests with fixed sample data, and the recognized objects it satisfies are limited to the selected part categories. If other parts need to be trained, it is necessary to remake the labeled data set, and the process is too complicated.

[0006] Therefore, the technical personnel in this field are committed to developing a part sorting and recognition system and method based on deep learning to solve the problems of difficult acquisition of data sets and limited application scenarios existing in the existing technical solutions. Summary of the Invention

[0007] In view of the above defects of the prior art, the technical problem to be solved by the present invention is how to overcome the defects such as difficult acquisition of data sets and application being restricted by scenarios in the process of actual application of deep learning network models in the prior art.

[0008] To achieve the above object, the present invention provides a part sorting and recognition system and method based on deep learning to improve the generalization of the part sorting target recognition network, wherein deep learning specifically refers to unsupervised domain adaptation learning.

[0009] A part sorting and recognition system based on deep learning provided by the present invention includes:

[0010] A training module and an application module. A model is established in the training module, and the model is applied to the application module to obtain a recognition result and output the recognition result;

[0011] Wherein, the training module includes:

[0012] A synthetic image making module, which draws a three-dimensional synthetic image of the recognition target, and the three-dimensional synthetic image includes several angles;

[0013] An actual image acquisition module, which acquires a two-dimensional real scene image of the recognition target and uses the two-dimensional real scene image as a part of the training set and the test set;

[0014] A data set generation module, which is respectively connected to the synthetic image making module and the actual image acquisition module. By means of machine learning methods, the three-dimensional synthetic image is labeled with true value boxes and categories, and then the labeled three-dimensional synthetic image is used as the source domain, and the unlabeled real image is used as the target domain, and the training set and the validation set are generated according to a set ratio;

[0015] [[ID=]27]A deep learning module, which is connected to the data set generation module, receives the training set and the validation set, and constructs a deep learning model, including a target recognition network and a domain adaptation network. The target recognition network is used for feature extraction, object classification, and generation of candidate boxes; the domain adaptation network is used for aligning the features of images in the source domain and the target domain.

[0016] Further, the application module also includes:

[0017] The actual image acquisition module and the deep learning module. The actual image acquisition module re-acquires the two-dimensional real-scene image of the recognition target and serves as the input of the model established in the deep learning module, and generates the recognition result in the deep learning module;

[0018] The application module further includes a result output module. The result output module is connected to the deep learning module, obtains the recognition result and the coordinates of the candidate box, converts the coordinates of the candidate box into the world coordinates of the part sorting workbench, and outputs the category and position of the recognition target.

[0019] Further, in the process of constructing the deep learning model, it includes constructing the forward propagation and backward propagation processes of the network: the input training set and validation set are first subjected to feature extraction through the feature extraction layer, and then enter the branches. One part continues the training of the target recognition network, and the other part enters the domain classifier; through the backward propagation of the loss function and the update of the weights by the optimizer, the optimal feature extractor is found so that the effect of the domain classifier is the worst while the prediction result of the self-target recognition of the deep learning model is the best.

[0020] Further, the ratio of generating the training set and the validation set according to the set ratio is 8:2.

[0021] A part sorting and recognition method based on deep learning provided by the present invention, the method includes the following steps:

[0022] Step 1, generate a training data set;

[0023] Step 2, train a deep learning model;

[0024] Step 3, test the deep learning model established in Step 2;

[0025] Step 4, input the real-time image of the recognition object obtained from the actual scene into the deep learning model to obtain the recognition result, convert the pixel-based position information of the recognition object into the world coordinates based on the sorting workbench, and output the final classification result and position information.

[0026] Further, Step 1 includes the following sub-steps:

[0027] Step 1.1, acquire images, including acquiring the three-dimensional model of the recognition object and intercepting two-dimensional single-target images at multiple angles, and capturing no less than one thousand part sorting images through a monocular camera;

[0028] Step 1.2: Image annotation, including performing candidate box annotation on the part sorting image, using the canny operator for contour extraction, and determining the object position with the minimum bounding rectangle, and then outputting it as an xml label file according to the annotation format of the Pascal Voc dataset to obtain the labeled synthetic image;

[0029] Step 1.3: Use the labeled synthetic image as the source domain part and the unlabeled part sorting image as the target domain part. Each is divided into a training set and a validation set according to a ratio of 8:2 and randomly shuffled to generate the training dataset.

[0030] Further, the step 2 includes the following sub-steps:

[0031] Step 2.1: Use the pre-trained ResNet50 weights on imageNet as the initial weights of the deep learning model and set the initial values of the hyperparameters;

[0032] Step 2.2: Construct the overall network architecture, using Faster-RCNN as the target recognition network. Among them, ResNet50 is the backbone network, including a feature extraction part and a classification and candidate box regression part; use the gradient reversal layer based on the adversarial idea as the domain adaptation implementation module, including a domain classifier part;

[0033] Step 2.3: Construct the forward propagation process. Send the training dataset into the deep learning model, obtain a feature map through the feature extractor, send the obtained feature map into the classification and candidate box regression part for target recognition, and at the same time send it into the domain classifier part to determine whether the feature map comes from the source domain part or the target domain part;

[0034] Step 2.4: Construct the backward propagation process, calculate the error loss of the target recognition network, and call the optimizer to update the weight parameters to find an optimal feature extractor and the domain classifier part to minimize the error of the deep learning model. At the same time, calculate the error loss of the domain classifier part and call the optimizer to update the weight parameters to find a domain classifier part with the smallest classification error. At the same time, send the loss through the gradient reversal layer with the opposite sign into the feature extractor to find a feature extractor with the largest error of the domain classifier;

[0035] Step 2.5: Iterate the entire training process no less than 200 times to obtain the recognition result with the minimum loss function and complete the training of the deep learning model.

[0036] Further, the step 3 includes the following sub-steps:

[0037] Step 3.1: Load the weights and various parameters of the trained deep learning model;

[0038] Step 3.2: Use a camera to obtain a real-time image of the part sorting workbench;

[0039] Step 3.3: Preprocess the real-time image so that it meets the network input requirements and send it into the deep learning model for testing;

[0040] Step 3.4: Obtain the recognition result, including the classification result and the position information of the recognition object.

[0041] Further, in the step 1.1, the number of part sorting images captured by the monocular camera is 2000; the part sorting images are single-object or multi-object.

[0042] Further, in the step 2.5, the entire training process is iterated 300 times.

[0043] A part sorting recognition system and method based on deep learning provided by the present invention at least has the following technical effects:

[0044] 1. Existing part sorting technologies based on deep learning all collect data sets for a certain application scenario, train and test in a specific scenario, lacking mobility. Since there are many styles of parts themselves and various sorting requirements, it is unrealistic to specifically make a data set for training for each requirement and scenario. In the technical solution provided by the embodiment of the present invention, training is directly carried out through synthetic images, avoiding the problem that the effects of a network vary in different scenarios;

[0045] 2. The technical solution provided by the embodiment of the present invention is an unsupervised learning method. Only the images taken on-site need to be sent as test images and reference images for domain alignment, without the need for secondary processing, greatly reducing the workload of data set production.

[0046] The concept, specific structure and technical effects generated by the present invention will be further described below in conjunction with the drawings to fully understand the purpose, features and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is the structure diagram of the part sorting recognition system of a preferred embodiment of the present invention;

[0048] Figure 2 is Figure 1 the network training flow chart of the part sorting recognition method in the shown embodiment. DETAILED DESCRIPTION

[0049] The following describes multiple preferred embodiments of the present invention with reference to the accompanying drawings of the specification to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0050] The technical problem to be solved by the embodiments of the present invention is how to overcome the defects such as difficult acquisition of data sets and limited application scenarios in the actual application process of deep learning network models in the prior art.

[0051] A part sorting and recognition system based on deep learning provided by the embodiments of the present invention proposes to use a domain adaptation transfer learning method to align the features of the modeled images of parts or other network synthesized images with the images obtained from the actual sorting workbench, so that the training results on the synthesized images can be applied to the actual scenario. Among them, the synthesized images are all single target objects, which are labeled using machine learning methods, and the actual scenario test images do not need to be labeled. The entire training process is unsupervised learning, which solves the problem of difficult acquisition of data sets, and a trained network model can be easily obtained by inputting test images of different scenarios.

[0052] As Figure 1 shown, a part sorting and recognition system based on deep learning provided by the present invention includes:

[0053] A training module and an application module. A model is established in the training module, and the model is applied to the application module to obtain a recognition result and output the recognition result;

[0054] Among them, the training module includes:

[0055] A synthesized image production module. The synthesized image production module draws three-dimensional synthesized images of the recognition target, and the three-dimensional synthesized images include several angles. Three-dimensional images of the parts to be recognized are drawn using modeling software such as CAD, which are often realized before part production. Two-dimensional synthesized images of each angle of the parts to be recognized are obtained as part of the training data.

[0056] An actual image acquisition module. The actual image acquisition module acquires two-dimensional real-scene images of the recognition target and uses the two-dimensional real-scene images as part of the training set and the test set. This module consists of a monocular camera, which is set at the position of the actual part sorting workbench to acquire two-dimensional real-scene images of the parts as part of the training data and the test data.

[0057] The dataset generation module is connected to the synthetic image production module and the actual image acquisition module respectively. It annotates the ground truth boxes and categories of the 3D synthetic images through machine learning methods, and then uses the annotated 3D synthetic images as the source domain and the unannotated real images as the target domain, and generates the training set and the validation set according to a set ratio. The ratio of generating the training set and the validation set according to the set ratio is 8:2.

[0058] The deep learning module is connected to the dataset generation module, receives the training set and the validation set, and constructs a deep learning model, including an object recognition network and a domain adaptation network. The object recognition network is used for feature extraction, object classification, and generation of candidate boxes; the domain adaptation network is used for aligning the features of the images in the source domain and the target domain.

[0059] Among them, the application module also includes:

[0060] The actual image acquisition module and the deep learning module. The actual image acquisition module re-acquires the two-dimensional real scene images of the recognition target and uses them as the input of the model established in the deep learning module, and generates recognition results in the deep learning module;

[0061] The application module also includes a result output module. The result output module is connected to the deep learning module, obtains the recognition results and the coordinates of the candidate boxes, converts the coordinates of the candidate boxes into the world coordinates of the part sorting workbench, and outputs the category and position of the recognition target.

[0062] Among them, in the process of constructing the deep learning model, it includes constructing the forward propagation and backward propagation processes of the network: the input training set and validation set are first subjected to feature extraction through the feature extraction layer, and then enter the branches. One part continues the training of the object recognition network, and the other part enters the domain classifier; through the backward propagation of the loss function and the update of the weights by the optimizer, the optimal feature extractor is found so that the effect of the domain classifier is the worst while the prediction result of the object recognition of the deep learning model itself is the best.

[0063] As Figure 2 shown, it is the training flow chart of a part sorting and recognition method based on deep learning provided by the present invention. The specific solution is:

[0064] First, obtain the synthetic image and the real image of the part to be recognized and make them into a data set. Subsequently, design and construct the overall network architecture, which consists of an object recognition network and a domain adaptation module. Construct the forward propagation and backward propagation processes of the network. The input data will first be feature-extracted through the feature extraction layer, and then enter the branches. One part continues the training of the object recognition network, and the other part enters the domain classifier. Update the weights through the backward propagation of the loss function and the optimizer. Finally, find an optimal feature extractor so that the effect of the domain classifier is the worst while the prediction result of its own object recognition is the best.

[0065] After the training is completed, test and predict the real-time image obtained from the actual scene. After obtaining the recognition result, convert the pixel position information of the candidate box into the world coordinates of the sorting workbench and output the final classification result and position information.

[0066] Specifically, a part sorting and recognition method based on deep learning provided by the present invention includes the following steps:

[0067] Step 1: Generate a training data set;

[0068] Step 2: Train a deep learning model;

[0069] Step 3: Test the deep learning model established in Step 2;

[0070] Step 4: Input the real-time image of the object to be recognized obtained from the actual scene into the deep learning model to obtain the recognition result. Convert the pixel-based position information of the recognized object into the world coordinates based on the sorting workbench, and output the final classification result and position information.

[0071] Among them, Step 1 includes the following sub-steps:

[0072] Step 1.1: Obtain images, including obtaining the 3D model of the object to be recognized and intercepting 2D single-object images at multiple angles. Capture no less than one thousand part sorting images through a monocular camera. Specifically, the number of part sorting images captured by the monocular camera is 2000. The part sorting images are single-object or multi-object.

[0073] Step 1.2: Image annotation, including performing candidate box annotation on the part sorting images, using the canny operator for contour extraction, and delimiting the object position with the minimum bounding rectangle, and then outputting it as an xml label file according to the annotation format of the Pascal Voc data set to obtain a labeled synthetic image.

[0074] Step 1.3: Take the labeled synthetic image as the source domain part and the unlabeled part sorting images as the target domain part. Divide them into a training set and a validation set according to a ratio of 8:2 respectively and shuffle the order randomly to generate a training data set.

[0075] Step 2 includes the following sub-steps:

[0076] Step 2.1: Use the pre-trained ResNet50 weights on ImageNet as the initial weights of the deep learning model and set the initial values of the hyperparameters;

[0077] Step 2.2: Construct the overall network architecture, with Faster-RCNN as the object recognition network. Among them, ResNet50 is the backbone network, including a feature extraction part and a classification and candidate box regression part; use the gradient reversal layer based on the adversarial idea as the domain adaptation implementation module, including a domain classifier part;

[0078] Step 2.3: Construct the forward propagation process. Send the training dataset into the deep learning model, obtain a feature map through the feature extractor, send the obtained feature map into the classification and candidate box regression part for object recognition, and at the same time send it into the domain classifier part to determine whether the feature map comes from the source domain part or the target domain part;

[0079] Step 2.4: Construct the backward propagation process. Calculate the error loss of the object recognition network, and call the optimizer to update the weight parameters to find an optimal feature extractor and domain classifier part to minimize the error of the deep learning model. At the same time, calculate the error loss of the domain classifier part and call the optimizer to update the weight parameters to find a domain classifier part with the smallest classification error. At the same time, send the loss through the gradient reversal layer with the opposite sign into the feature extractor to find a feature extractor to maximize the error of the domain classifier;

[0080] Step 2.5: Iterate the entire training process no less than 200 times to obtain the recognition result with the smallest loss function and complete the training of the deep learning model. In particular, iterate the entire training process 300 times.

[0081] Step includes the following sub-steps:

[0082] Step 3.1: Load the weights and various parameters of the trained deep learning model;

[0083] Step 3.2: Use the camera to obtain the real-time image of the part sorting workbench;

[0084] Step 3.3: Preprocess the real-time image to make the real-time image meet the network input requirements and send it into the deep learning model for testing;

[0085] Step 3.4: Obtain the recognition result, including the classification result and position information of the recognition object.

[0086] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art in this technical field based on the concept of the present invention through logical analysis, reasoning, or limited experiments on the basis of the prior art should fall within the protection scope determined by the claims.

Claims

1. A part sorting and recognition system based on deep learning, characterized in that, Including: A training module and an application module. A model is established in the training module, and the model is applied to the application module to obtain an identification result and output the identification result. Among them, the training module includes: A synthetic image production module. The synthetic image production module draws a three-dimensional synthetic image of the identification target, and the three-dimensional synthetic image includes several angles. An actual image acquisition module. The actual image acquisition module acquires a two-dimensional real-scene image of the identification target and uses the two-dimensional real-scene image as part of the training set and the test set. A data set generation module. The data set generation module is respectively connected to the synthetic image production module and the actual image acquisition module. By means of machine learning methods, the three-dimensional synthetic image is labeled with a truth box and a category, and then the labeled three-dimensional synthetic image is used as the source domain, and the unlabeled real image is used as the target domain, and a training set and a validation set are generated according to a set ratio. A deep learning module. The deep learning module is connected to the data set generation module, receives the training set and the validation set, and constructs a deep learning model, including an object recognition network and a domain adaptation network. The object recognition network is used for feature extraction, object classification, and generation of candidate boxes. The domain adaptation network is used to align the features of the images in the source domain and the target domain. The application module also includes: The actual image acquisition module and the deep learning module. The actual image acquisition module re-acquires a two-dimensional real-scene image of the identification target and uses it as the input of the model established in the deep learning module, and generates the identification result in the deep learning module. The application module further includes a result output module. The result output module is connected to the deep learning module, obtains the identification result and the coordinates of the candidate box, converts the coordinates of the candidate box into the world coordinates of the part sorting workbench, and outputs the category and position of the identification target. In the process of constructing the deep learning model, it includes the forward propagation and backward propagation processes of the network: the input training set and validation set are first subjected to feature extraction through a feature extraction layer, and then enter branches. One part continues the training of the object recognition network, and the other part enters the domain classifier. Through the backward propagation of the loss function and the update of the weights by the optimizer, the optimal feature extractor is found so that the effect of the domain classifier is the worst while the prediction result of the self-object recognition of the deep learning model is the best.

2. The part sorting and recognition system based on deep learning according to claim 1, wherein, The ratio of generating the training set and the validation set according to the set ratio is 8:

2.

3. A part sorting and recognition method based on deep learning, characterized in that, The method includes the following steps: Step 1, generate a training data set; Step 2, train the deep learning model; Step 3, test the deep learning model established in Step 2; Step 4, input the real-time image of the identification object obtained from the actual scene into the deep learning model to obtain an identification result, convert the position information of the identification object based on pixels into the world coordinates based on the sorting workbench, and output the final classification result and position information; The Step 1 includes the following sub-steps: Step 1.1: Obtain images, including obtaining a 3D model of the recognition object and intercepting 2D single-object images at multiple angles, and capturing no less than one thousand part sorting images through a monocular camera; Step 1.2: Image annotation, including performing candidate box annotation on the part sorting images, using the canny operator for contour extraction, and determining the object position through the minimum bounding rectangle, and then outputting as an xml label file according to the annotation format of the Pascal Voc dataset to obtain a synthetic image with labels; Step 1.3: Use the synthetic image with labels as the source domain part and the unlabeled part sorting images as the target domain part, and divide them into a training set and a validation set according to a ratio of 8:2 respectively and shuffle the order randomly to generate the training dataset; The said Step 2 includes the following sub-steps: Step 2.1: Use the pre-trained ResNet50 weights on imageNet as the initial weights of the deep learning model and set the initial values of the hyperparameters; Step 2.2: Construct the overall network architecture, use Faster-RCNN as the object recognition network, where ResNet50 is the backbone network, including a feature extraction part and a classification and candidate box regression part; Use the gradient reversal layer based on the adversarial idea as the domain adaptation implementation module, including a domain classifier part; Step 2.3: Construct the forward propagation process, send the training dataset into the deep learning model, obtain a feature map through the feature extractor, send the obtained feature map into the classification and candidate box regression part for object recognition, and at the same time send it into the domain classifier part to determine whether the feature map comes from the source domain part or the target domain part; Step 2.4: Construct the backpropagation process, calculate the error loss of the object recognition network, and call the optimizer to update the weight parameters to find an optimal feature extractor and the domain classifier part to minimize the error of the deep learning model. At the same time, calculate the error loss of the domain classifier part and call the optimizer to update the weight parameters to find a domain classifier part to make the classification error of the domain classifier part as small as possible. At the same time, send the loss to the feature extractor after taking the opposite through the gradient reversal layer to find a feature extractor to make the error of the domain classifier as large as possible; Step 2.5: Iterate the entire training process no less than 200 times to obtain the recognition result with the minimum loss function and complete the training of the deep learning model.

4. The part sorting and recognition method based on deep learning according to claim 3, characterized in that, The said Step 3 includes the following sub-steps: Step 3.1: Load the weights and various parameters of the trained deep learning model; Step 3.2: Use the camera to obtain the real-time image of the part sorting workbench; Step 3.3: Preprocess the real-time image to make the real-time image meet the network input requirements and send it into the deep learning model for testing; Step 3.4: Obtain the recognition result, including the classification result and the position information of the recognition object.

5. The method for part sorting and recognition based on deep learning according to claim 3, wherein, In the step 1.1, the number of the part sorting images captured by the monocular camera is 2000; the part sorting images are single-object or multi-object.

6. The part sorting and recognition method based on deep learning according to claim 3, characterized in that In the step 2.5, the entire training process is iterated 300 times.

Images