Intelligent article sorting and packaging system based on depth vision

Abstract

The invention discloses an intelligent article sorting and packaging system based on depth vision. The intelligent article sorting and packaging system mainly comprises a target detection module, a defect detection module and an intelligent packaging module. Wherein the target detection module uses an improved YOLO deep neural network to quickly recognize the poses of articles to be sorted. And performing defect detection on the articles on the production line through the improved CFA defect detection model. And finally, the two tasks of sorting and packaging are completed on one production line at the same time, and the packaging mode is rapidly replaced according to requirements without programming. According to the method, the improved CFA defect detection module is adopted, and the SE attention mechanism module is inserted into the wide residual network WideResNet50-2, so that the model is better helped to learn and adapt to different models of articles and data distribution, more important features are concerned, and the generalization performance is improved. According to the programming-free interface designed by the invention, a user can conveniently replace different packaging modes, package different types of articles and select whether to carry out a defect detection task or not.

Description

Technical Field

[0001] The present invention relates to the fields of visual inspection and parallel robotic arm operation on industrial assembly lines. Specifically, it includes a vision-based target detection module, a defect recognition module, and an intelligent sorting and packaging system design module. Background Art

[0002] Sorting and packaging are important links in the production process of items. Due to the relatively rapid iteration or update of the types of products processed on the production line in recent years, the following problems have emerged in the sorting and packaging link: (1) The complex and variable product processes often lead to various defects in the products. Removing defective products through defect detection is a crucial link for ensuring high-quality products. However, manually removing defective products with the naked eye is prone to fatigue and sorting errors during long-term work or large-scale production, and the cost is relatively high; (2) Currently, when using a robotic arm system for sorting and packaging, it is often "one machine for one use". When the products on the assembly line are updated, it is necessary to re-design the vision detection program and plan the movement route of the robotic arm, which is time-consuming, cumbersome, and difficult to meet the actual needs of "one machine for multiple uses".

[0003] With the rapid development of deep vision detection technology in the field of artificial intelligence, the detection of surface abnormalities / defects of products is expected to be achieved using a vision detection system. By using a convolutional neural network (CNN) that has been extensively pre-trained on datasets such as ImageNet to extract image features, etc., a defect detection model can be constructed. However, since the pre-trained dataset often differs significantly from the real target, it is easy to overestimate the normality of abnormal features, resulting in false detections. Therefore, based on the pre-training of visual data, how to enable the defect detection model to continuously learn and adapt to the defect detection of new types of products is an urgent problem to be solved.

[0004] On the other hand, when using a robotic arm system for sorting and packaging, it is urgent to achieve "one machine for multiple uses", that is, for the production of different types of small packages, the robotic arm operation control model can be conveniently replaced, so that products, packaging methods, etc. on the same production line can be replaced without programming to meet the rapid change requirements of the products processed on the production line. Summary of the Invention

[0005] To address the above problems, the present invention constructs an intelligent sorting and packaging system for items on a production line based on deep vision, which mainly includes three modules: an object detection module, a defect detection module, and an intelligent packaging module. Among them, the object detection module uses an improved YOLO deep neural network to quickly identify the pose of the items to be sorted. Then, the improved CFA defect detection model (CFA: Coupled-Hypersphere-Based Feature Adaptation) is used to detect the defects of the items on the production line. If the item is defect-free, the parallel manipulator will pick up and place the item into the packaging box according to the requirements of the current packaging task; if the item is defective, the parallel manipulator will pick up the defective item and place it at a designated position on the other side. Finally, the two tasks of sorting and packaging are completed simultaneously on a production line, and the packaging method can be quickly changed without programming according to the requirements.

[0006] The specific steps are as follows:

[0007] In the present invention, a Basler acA2500-14gc industrial camera is used to collect visual images of different models of products. The collected images are divided into two categories: normal images (positive samples) and images with defects (negative samples). During the production of the dataset, one-fourth of the positive samples and the negative samples are jointly used as the samples to be tested, and the remaining three-fourths of the positive samples are used as the samples to be trained. The training of the defect detection model only requires the use of positive samples, while object detection requires the application of both positive samples and negative samples.

[0008] In the present invention, an improved YOLO deep neural network is used to identify small packaged foods in the collected pictures. Since during packaging and sorting, it is necessary to accurately identify the pose of the items to be packaged so that the manipulator can quickly grasp them. The present invention realizes the efficient detection of the object size and rotation angle by introducing the rotation angle loss θ and combining the dynamic weight adjustment strategy. The improved network dynamically adjusts the weights of the positioning loss, size loss, and angle loss to ensure that when the model detects objects of different sizes and angles, it can adaptively adjust the focus of attention to ensure the detection accuracy.

[0009] In the present invention, the defect detection module uses an improved CFA algorithm to effectively detect defects in small packaged foods, such as food fragmentation. The CFA algorithm was proposed by Lee et al. from Inha University in South Korea in 2022. It uses a coupled hypersphere-based feature adaptation method to achieve complex anomaly localization by using features adapted to the target dataset. CFA includes a learnable patch descriptor for learning and embedding object-oriented features and an extensible memory bank independent of the size of the target dataset. The algorithm adopts transfer learning technology to increase the density of normal features by applying patch description and memory bank to the feature extractor CNN, so as to clearly distinguish abnormal features.

[0010] To improve the effectiveness of feature extraction, the CFA algorithm is improved in the present invention. In the algorithm, the WideResNet50-2 network is utilized to effectively reduce the risk of overfitting by doubling the number of channels in each convolutional layer. Then, an SE attention mechanism module is inserted into it, enabling the model to better adapt to the changes of different input samples without significantly increasing the model's parameters. The improved network is used to extract the features of the object surface, and during the training process, the features of different models of items can be effectively extracted and aggregated together. In the testing phase, the patch description features obtained from any sample in the test set are matched with the nearest neighbor memory features searched in the memory bank, and a heat map representing the degree of abnormality is generated. Finally, a scoring map for abnormal localization in the heat map is calculated through a specific scoring function to achieve the defect detection of the item. Through experimental verification, the defect detection ability of the improved CFA algorithm is significantly better than that of the original CFA algorithm.

[0011] The third module of the present invention is the intelligent packaging module. The core function of this module is to allow users to set the type of item, the size of the small package, and the number of layers, columns, and rows of the required packaging, etc.; and set these parameters as variables to avoid the need for the robotic arm to reprogram and plan the grasping for different species and packaging tasks. This module enables the system to intelligently plan the placement position of each item and the grasping motion trajectory of the robotic arm according to the packaging task.

[0012] Overall, the system constructed in the present invention can be divided into a vision module and a motion module of the parallel robotic arm. The vision module includes a target detection and a defect detection module. Each module is encapsulated and integrated into a system to achieve non-programming. For different packaging tasks, only the relevant parameters need to be modified, and the parallel robotic arm can autonomously plan the motion path to sort and package the small bagged foods after defect detection.

[0013] Due to the adoption of the above technical solutions, the present invention has the following advantages:

[0014] The improved YOLO deep neural network can quickly identify the pose and category of the item to be packaged, facilitating the packaging task. In the present invention, an improved CFA defect detection module is adopted. By inserting the SE attention mechanism module into the wide residual network WideResNet50-2, it can better help the model learn and adapt to different models of items and data distributions, focus on more important features, and improve the generalization performance. Applying the patch description and the memory bank to the improved CNN can alleviate the bias of CNN pre-training. The improved CFA not only has adaptability to different data sets but also can effectively learn the key features of different data sets, reducing the complexity of training and the work of replacing the model.

[0015] In addition, the non-programmable interface designed by the present invention enables users to conveniently change different packaging methods, package different types of items, and select whether to perform defect detection tasks. The entire system can directly achieve the packaging and sorting of different items through the operation interface only by modifying relevant parameters. Brief Description of the Drawings

[0016] Figure 1 is a flowchart of an intelligent sorting and packaging system based on machine vision;

[0017] Figure 2 is a schematic diagram of a positive sample collected by Basler acA2500-14gc;

[0018] Figure 3 is a schematic diagram of a negative sample collected by Basler acA2500-14gc;

[0019] Figure 4 is a schematic diagram of the SE module;

[0020] Figure 5 is a schematic diagram of an improved CFA module;

[0021] Figure 6 is a partial detection comparison diagram of the CFA before and after improvement in the MVTec dataset;

[0022] Figure 7 is a partial detection diagram of the improved CFA algorithm for negative samples;

[0023] Figure 8 is a non-programmable operation interface diagram of the entire system. Detailed Embodiment

[0024] The following further describes in detail the specific embodiments of the present invention in conjunction with the drawings;

[0025] Att Figure 1 is a system flowchart of an intelligent sorting and packaging system based on machine vision. The system inputs the types, packaging methods, and tasks of items through a non-programmable module. After starting the system through the non-programmable interface, the items are placed on the conveyor belt, and the Basler camera collects images of the items on the conveyor belt in real time. The improved YOLO network is used to identify the items on the conveyor belt, and the improved CFA defect detection module is used for defect detection. If the item has a defect, the robot classifies it into the defect handling area; if there is no defect, the robot performs sorting and packaging according to the preset packaging task. The system combines advanced visual recognition technology and robot operation to achieve efficient and accurate item defect detection and classification packaging. The detailed steps for realizing the above functions are as follows:

[0026] First, use an industrial Basler acA2500-gc camera to collect several packaged foods with different shapes, sizes or colors, with a resolution of 1900 pixels * 1900 pixels. Since objects in motion are collected on the industrial production line for detection, blurring may occur. Therefore, during the collection process, the white balance function of the Basler camera is used to ensure that the objects in the captured images are clearer.

[0027] Appendix Figure 2 and Appendix Figure 3 are positive and negative sample examples collected using the camera, respectively, for visual module training.

[0028] In this invention, an improved YOLO deep neural network is used to identify small packaged foods to be detected on the production line. The loss function of the YOLO deep neural network consists of the following parts: localization loss, classification loss, and confidence loss. The localization loss includes the center coordinates (x, y), width (w), and height (h) of the bounding box, and adds the overall form of the angle (θ) loss function and dynamic weight adjustment strategy as shown in Equation (1) L = αL loc + βL CSC + γL conf (1) where: L loc is the localization loss, L CSC is the classification loss, and L conf is the confidence loss; the weights α, β, and γ are adaptively adjusted according to the actual detection object, as shown in Equations (1) and (2). Where C 1 , C 2 are constants, and the weights are adaptively adjusted according to different detection objects, so as to achieve more accurate detection. γ = C 2 ·|θ| (3) The weight α can improve the localization accuracy of the model when detecting objects of different sizes, and the weight γ ensures more accurate confidence prediction of the target pose when detecting the object pose. The localization loss consists of the following: coordinate loss L coord for predicting the center coordinates (x, y) of the bounding box; size loss L size for predicting the width and height of the bounding box; angle loss L angle for predicting the angle of the bounding box. The three together constitute the localization loss, and the overall form is as shown in Equation (4). L loc = L coord + L size + L angle (4)

[0029] AppendixFigure 4 This is the schematic diagram of the SE module. The input X undergoes an arbitrary transformation F tr to become U, whose dimension is H×W×C. Assuming that the output U is not optimal and the importance of each channel is different, global average pooling F is performed on each output channel sq (·) to generate a 1×1×C vector, as shown in Equation (5):

[0030] Then it passes through F ex (2, W). This process includes two fully connected layers and two activation functions, and the weights of each channel will be obtained, as shown in Equation (6): s = F ex (z, W) = σ(g, (z, W)) = σ(W 2 δ(W 1 z)) (6) Two fully connected layers W 1 , W 2 Process the vector z obtained in the previous step to get the channel weight value s. Different values in s represent the weight information of different channels, and different weights are assigned to the channels.

[0031] Finally, multiply U and the generated feature vector s(1×1×C) to get the final output, as shown in Equation (7): X′ = F scale (u c , s c ) = s c u c (7) That is, each of the H*W values in each channel of the feature map U is multiplied by the weight value of the corresponding channel in s. Integrate the SE attention mechanism module into the network after WideResNet50-2 as a feature extractor. Attached Figure 5Schematic diagram of an improved CFA defect detection module. Inserting the SE module into the Bottleneck of WideResNet50-2 includes: first, performing three convolutional and batch normalization operations, then compressing the feature map in the spatial dimension through global average pooling, passing through two fully connected layers and ReLU and Sigmoid activation functions to generate channel weights, finally adjusting the original feature map through scaling, and performing residual connection to output the final feature map. The improved CNN is used as a feature extractor, and patch description and memory bank are applied to it. In the training stage, contrastive supervised learning is carried out based on the superimposed hypersphere created centered on the memory features, that is, the coupled hypersphere; in the testing stage, the patch description features obtained from any sample in the test set are matched with the nearest neighbor memory features searched in the memory bank to achieve defect detection. The improved defect detection module can adaptively adjust each channel, improve the generalization performance of the network and the learning performance on different data distributions, reduce the complexity of training and the work of frequently replacing models.

[0032] Appendix Figure 6 Partial comparison diagrams of the CFA module before and after improvement in MVTec detection. The improved CFA module can better adapt to different item inputs, capture important features better, and thus reduce the probability of problems such as false detection during defect detection. Appendix Figure 7 Detection diagram of the improved CFA module for the collected negative samples.

[0033] In the present invention, a parallel robotic arm is used to perform the sorting and packaging tasks. Appendix Figure 8 Diagram of the programming-free operation interface of the entire system. Select the type of item to be packaged, as well as the number of layers, columns, and rows of packaging according to requirements. Set the number of layers of packaging as L, the number of columns as C, the number of rows as R, and the size of the small package as W×H×D (length×width×height). In addition, set the initial position of the item as (x 0 , y 0 , z 0 ), the placement position as (x 1 , y 1 , z 1 ), and i, j, k respectively represent the row, column, and layer where the packaging position is currently located, all initially 1. Then there are the following relationships: x 1 = x 0 - (j - 1)×H y 1 = y 0 + (i - 1)×W z 1 = z 0 + (k - 1)×D (8) If there are defects, they will be sorted out; otherwise, they will be packed and placed according to Equation (8).

[0034] The entire system encapsulates the perception results of the vision module and the motion planning route of the parallel manipulator, realizing non-programming. The entire system can achieve sorting out defective parts and packing intact objects according to requirements only by inputting relevant parameters for different packaging tasks. Attached Figure 8 is the non-programming operation interface diagram of the entire system. Different items can be selected through the operation interface, and the corresponding rows, columns, and layers can be input according to different packaging tasks. The operation interface can select packaging tasks or defect detection tasks. If the packaging task is selected, small items will be directly packed; if defect detection is selected, defective small items will be sorted out, and intact items will be packed according to the input requirements.

Claims

1. The intelligent sorting and packaging system for items based on deep vision is characterized by: It contains three modules: target detection module, defect detection module and intelligent packaging module; The target detection module uses an improved YOLO deep neural network to quickly identify the position and posture of the items to be sorted. Then, the defect detection module performs defect detection on items on the production line through the improved CFA defect detection model and feeds back the defect detection results to the intelligent packaging module; At the end of the intelligent packaging module, if the item is not defective, the parallel robot arm will pick up and place the item in the packaging box according to the requirements of the current packaging task; if the item is defective, the parallel robot arm picks up the defective item and places it at the designated position on the other side; ultimately, the two tasks of sorting and packaging are completed on one production line at the same time, and the packaging method can be quickly changed without programming according to requirements.

2. The intelligent item sorting and packaging system based on deep vision according to claim 1 is characterized in that: Use Basler acA2500-14gc industrial camera to collect visual images of different models of products; the collected images are divided into two categories: normal images and images with defects; During the data set creation process, one quarter of the positive samples and negative samples are used as test samples, and the remaining three quarters of the positive samples are used as training samples; the training of the defect detection model only requires the use of positive samples, while target detection requires the use of both positive and negative samples.

3. The intelligent item sorting and packaging system based on deep vision according to claim 1 is characterized in that: Improved YOLO deep neural network to identify small packaged foods in collected images; By introducing the rotation angle loss θ and combining it with a dynamic weight adjustment strategy, efficient detection of object size and rotation angle can be achieved; the improved YOLO deep neural network dynamically adjusts the weights of positioning loss, size loss and angle loss to ensure that the model can adaptively adjust its focus when detecting objects of different sizes and angles to ensure detection accuracy.

4. The intelligent item sorting and packaging system based on deep vision according to claim 1 is characterized in that: The defect detection module uses an improved CFA algorithm to effectively detect defects in small bagged foods. The improved CFA algorithm includes a learnable patch descriptor for learning and embedding target-oriented features and an expandable memory library independent of the target dataset size. It uses transfer learning technology to increase the density of normal features by applying patch descriptions and memory libraries to the feature extractor CNN, thereby clearly distinguishing abnormal features.

5. The object intelligent sorting and packaging system based on deep vision according to claim 4 is characterized in that: The improved CFA algorithm uses the WideResNet50-2 network to effectively reduce the risk of overfitting by doubling the number of channels in each convolutional layer, and then inserts the SE attention mechanism module into it, so that the model can better adapt to the changes of different input samples without significantly increasing the parameters of the model. The improved network is used to extract the surface features of objects, and the features of objects of different models can be effectively extracted and aggregated during the training process; in the test phase, the patch description features obtained from any sample in the test set are matched with the nearest neighbor memory features searched in the memory library, and a heat map representing the degree of abnormality is generated; finally, the scoring function is used to calculate the score map of the abnormality location in the heat map to achieve defect detection of objects.

6. The intelligent item sorting and packaging system based on deep vision according to claim 1 is characterized in that: The core of the intelligent packaging module is to allow users to set parameters, including the type of items, small package size, and the number of layers, columns, and rows required for packaging; the intelligent packaging module enables the system to intelligently plan the placement of each item and the gripping motion trajectory of the robot according to the packaging task.

7. The intelligent item sorting and packaging system based on deep vision according to claim 1 is characterized in that: The intelligent sorting and packaging system for items is divided into a visual module and a motion module of a parallel robot. The visual module includes target detection and defect detection modules. Each module is packaged and integrated to achieve programming-free operation. By re-modifying relevant parameters for different packaging tasks, the parallel robot can autonomously plan the motion path and sort and package small bagged foods after defect detection. The type of item, packaging method and task are input through the programming-free module; after starting the system through the programming-free interface, the items are placed on the conveyor belt, and the Basler camera collects images of the items on the conveyor belt in real time; the items on the conveyor belt are identified through the improved YOLO network, and defect detection is performed using the improved CFA defect detection module; if the item has defects, the parallel robot arm classifies it into the defect handling area; If there are no defects, the parallel robot will sort and pack according to the preset packaging tasks; The perception results of the visual module and the route of the parallel robot motion planning are encapsulated to achieve programming-free. The entire system can realize different packaging tasks by only inputting relevant parameters to sort out defective items and pack intact items as required; the programming-free operation interface of the entire system, select different items through the operation interface, and input corresponding rows, columns, and layers according to different packaging tasks; the operation interface can select packaging tasks or defect detection tasks. If the packaging task is selected, small items will be directly packaged; If defect detection is selected, small defective items will be sorted out and intact items will be packaged according to the input requirements.

8. The object intelligent sorting and packaging system based on depth vision according to claim 7 is characterized in that: First, an industrial Basler acA2500-gc camera was used to capture images of several packaged foods of different shapes, sizes, or colors, with a resolution of 1900 pixels * 1900 pixels. The white balance function of the Basler camera was used during the acquisition process to ensure that the objects in the images captured by the camera were clearer. Use the positive and negative samples collected by the camera for visual module training; The improved YOLO deep neural network is used to identify small packaged foods to be detected on the production line. The loss function of the YOLO deep neural network consists of the following parts: positioning loss, classification loss, and confidence loss. The positioning loss includes the center coordinates (x, y), width w, and height h of the bounding box. The overall form of the added angle θ loss function and the dynamic weight adjustment strategy is as shown in formula (1): L=αL loc +βL CSC +γL conf (1) Where: L loc is the positioning loss, L CSC is the classification loss, L conf is the confidence loss; each weight α, β, γ is adaptively adjusted according to the actual detection object, as shown in equations (1) and (2); C1 and C2 are constants, and the weights are adaptively adjusted according to different detection objects to achieve accurate detection; γ=C2·|θ| (3) The weight α can improve the positioning accuracy of the model when detecting objects of different sizes, and the weight γ ensures that the confidence prediction of the target posture is more accurate when detecting the posture of the object; the positioning loss is composed of the following: coordinate loss L coord Used to predict the center coordinates (x, y) of the bounding box; size loss L size Used to predict the width and height of the bounding box; angle loss L angle The angle used to predict the bounding box, the three together constitute the positioning loss, the overall form is as follows: L loc =L coord +L size +L angle (4) In the SE module, the input X undergoes an arbitrary transformation F tr It becomes U, whose dimension is H×W×C. Assuming that the output U is not optimal, the importance of each channel is different, and a global average pooling F is performed on each output channel. sq (·), generating a 1×1×C vector, as shown in formula (5): Then through F ex (·,W), this process includes two fully connected layers and two activation functions, and the weight of each channel is obtained as shown in formula (6): s=F ex (z,W)=σ(g,(z,W))=σ(W2δ(W1z)) (6) The two fully connected layers W1 and W2 process the vector z obtained in the previous step to obtain the channel weight value s. Different values ​​in s represent the weight information of different channels, giving different weights to the channels. Multiply U and the generated feature vector s(1×1×C) to get the final output, as shown in formula (7): X′=F scale (u c ,s c )=s c u c (7) That is, the H*W values ​​of each channel in the feature map U are multiplied by the weight of the corresponding channel in s; the SE attention mechanism module is integrated into the network after WideResNet50-2 as a feature extractor.

9. The intelligent item sorting and packaging system based on deep vision according to claim 7 is characterized in that: Figure 1. Schematic diagram of the improved CFA defect detection module. Inserting the SE module into the Bottleneck of WideResNet50-2 includes: first, performing three convolutions and batch normalization operations, then compressing the feature map in the spatial dimension through global average pooling, and then generating channel weights through two fully connected layers and ReLU and Sigmoid activation functions. Finally, the original feature map is adjusted by scaling and residual connection is performed to output the final feature map. The improved CNN is used as a feature extractor, and patch description and memory library are applied to it. In the training phase, comparative supervised learning is performed based on the superimposed hypersphere created with the memory feature as the center, namely the coupled hypersphere. In the testing phase, the patch description features obtained from any sample in the test set are matched with the nearest neighbor memory features searched in the memory library to achieve defect detection. The improved defect detection module can adaptively adjust each channel.

10. The intelligent item sorting and packaging system based on depth vision according to claim 7, characterized in that: Use parallel robots to carry out sorting and packaging tasks; Select the type of items to be packaged as required, as well as the number of layers, columns, and rows of the packaging; set the number of layers to L, the number of columns to C, the number of rows to R, and the size of the small package to W×H×D; set the initial position of the item to (x0, y0, z0), the placement position to (x1, y1, z1), i, j, k represent the current row, column, and layer of the packaging position, and are all initially 1; then the following relationship exists: x1=x0-(j-1)×H y1=y0+(i-1)×W z1=z0+(k-1)×D (8) If there are defects, they will be sorted out. If there are no defects, they will be packaged and placed according to formula (8).

Images