Intelligent sorting and packaging system for items based on depth vision

Abstract

This invention discloses an intelligent sorting and packaging system for items based on depth vision, mainly comprising three modules: an object detection module, a defect detection module, and an intelligent packaging module. The object detection module utilizes an improved YOLO deep neural network to quickly identify the pose of items to be sorted. Then, an improved CFA defect detection model is used to detect defects in items on the production line. Ultimately, it achieves simultaneous sorting and packaging tasks on a single production line, with packaging methods that can be quickly changed without programming as required. This invention employs an improved CFA defect detection module by inserting an SE attention mechanism module into the WideResNet50-2 residual network, thereby better helping the model learn and adapt to different item types and data distributions, focusing on more important features, and improving generalization performance. The no-programming interface designed in this invention allows users to easily change different packaging methods, package different types of items, and select whether to perform defect detection.

Description

Technical Field

[0001] This invention relates to the field of visual inspection and parallel robotic arm operation on industrial production lines. Specifically, it includes a vision-based target detection module, a defect recognition module, and an intelligent sorting and packaging system design module. Background Technology

[0002] Sorting and packaging is an important part of the production process. Due to the rapid iteration and updating of product types on the production line in recent years, the sorting and packaging process has encountered the following problems: (1) Complex and varied product processes often lead to various defects in products. Removing defective products is a crucial step in ensuring high-quality products. However, removing defective products by human eyes can easily lead to fatigue during long hours of work or large-scale production, resulting in sorting errors, and the cost is also high; (2) Currently, the use of robotic arm systems for sorting and packaging is often "one machine for one purpose". When products on the production line are updated, it is necessary to redesign the vision inspection program and plan the movement route of the robotic arm, which is time-consuming, cumbersome, and difficult to meet the actual needs of "one machine for multiple purposes".

[0003] With the rapid development of deep vision inspection technology in the field of artificial intelligence, the detection of surface anomalies / defects in products is expected to be achieved using visual inspection systems. Defect detection models can be built by pre-training convolutional neural networks (CNNs) on datasets such as ImageNet to extract image features. However, because the pre-trained datasets often differ significantly from the real targets, the normality of anomaly features is easily overestimated, leading to false detections. Therefore, how to enable defect detection models to continuously learn and adapt to the detection of defects in new types of products, based on visual data pre-training, is a problem that urgently needs to be solved.

[0004] On the other hand, when using robotic arm systems for sorting and packaging, it is urgent to achieve "one machine for multiple uses", that is, the robot's arm operation control model can be easily changed for different types of small packaging production, so that products and packaging methods on the same production line can be changed without programming, in order to adapt to the rapid change of products processed on the production line. Summary of the Invention

[0005] To address the above challenges, this invention constructs an intelligent sorting and packaging system for production lines based on depth vision, mainly comprising three modules: a target detection module, a defect detection module, and an intelligent packaging module. The target detection module utilizes an improved YOLO deep neural network to quickly identify the pose of items to be sorted. Then, an improved CFA (Coupled-Hypersphere-Based Feature Adaptation) defect detection model is used to detect defects in items on the production line. If the item is defect-free, a parallel robotic arm will pick it up and place it into a packaging box according to the requirements of the current packaging task; if the item is defective, the parallel robotic arm will pick up the defective item and place it at a designated location on the other side. Ultimately, this achieves simultaneous completion of sorting and packaging tasks on a single production line, with packaging methods that can be quickly changed without programming as required.

[0006] The specific steps are as follows:

[0007] This invention uses a Basler acA2500-14gc industrial camera to acquire visual images of different product models. The acquired images are divided into two categories: normal images (positive samples) and images with defects (negative samples). During dataset creation, one-quarter of the positive samples and negative samples are used together as test samples, while the remaining three-quarters of the positive samples are used as training samples. Training the defect detection model only requires positive samples, while target detection requires both positive and negative samples.

[0008] This invention utilizes an improved YOLO deep neural network to identify small packaged food items in captured images. During packaging and sorting, accurate identification of the pose of the items to be packaged is crucial for rapid gripping by a robotic arm. This invention achieves efficient detection of object size and rotation angle by introducing a rotation angle loss θ and combining it with a dynamic weight adjustment strategy. The improved network dynamically adjusts the weights of the localization loss, size loss, and angle loss to ensure that the model adaptively adjusts its focus when detecting objects of different sizes and angles, thus guaranteeing detection accuracy.

[0009] In this invention, the defect detection module utilizes an improved CFA algorithm to effectively detect defects in small-packaged food, such as food breakage. The CFA algorithm, proposed in 2022 by Lee et al. from Inha University, South Korea, employs a coupled hypersphere feature adaptation method to achieve complex anomaly localization by using features adapted to the target dataset. CFA includes a learnable patch descriptor for learning and embedding target-oriented features and a scalable memory library independent of the target dataset size. The algorithm employs transfer learning techniques, applying the patch descriptor and memory library to the feature extractor (CNN) to increase the density of normal features, thereby clearly distinguishing anomalous features.

[0010] To improve the effectiveness of feature extraction, this invention improves the CFA algorithm. The algorithm utilizes the WideResNet50-2 network, effectively mitigating overfitting risk by doubling the number of channels in each convolutional layer. Then, an SE attention mechanism module is inserted, allowing the model to better adapt to variations in different input samples without significantly increasing the model's parameters. The improved network is used to extract object surface features, effectively extracting and aggregating features from different item models during training. In the testing phase, patch description features obtained from any sample in the test set are matched with nearest-neighbor features searched in the memory database, generating a heatmap representing the degree of anomaly. Finally, a scoring map of anomaly localization in the heatmap is calculated using a specific scoring function, achieving defect detection for the item. Experiments verify that the improved CFA algorithm significantly outperforms the original CFA algorithm in defect detection.

[0011] The third module of this invention is the intelligent packaging module. The core function of this module is to allow users to set the type of item, the small package size, and the required number of layers, columns, and rows; and to set these parameters as variables to avoid the need for the robotic arm to be reprogrammed and replanned for different items and packaging tasks. This module enables the system to intelligently plan the placement of each item and the robotic arm's grasping trajectory according to the packaging task.

[0012] Overall, the system constructed in this invention can be divided into a vision module and a motion module for a parallel robotic arm. The vision module further includes target detection and defect detection modules. All modules are encapsulated and integrated into a single system, achieving programmability without the need for programming. For different packaging tasks, only the relevant parameters need to be modified, and the parallel robotic arm can autonomously plan its motion path to sort and package small-bagged food items after defect detection.

[0013] By adopting the above technical solutions, the present invention has the following advantages:

[0014] The improved YOLO deep neural network can quickly identify the pose and category of items to be packaged, facilitating packaging tasks. In this invention, an improved CFA defect detection module is employed. By inserting the SE attention mechanism module into the WideResNet50-2 residual network, the model can better learn and adapt to different item types and data distributions, focusing on more important features and improving generalization performance. Applying patch descriptions and memory libraries to the improved CNN can alleviate the bias of CNN pre-training. The improved CFA is not only adaptable to different datasets but also effectively learns key features from different datasets, reducing training complexity and the need for model replacement.

[0015] Furthermore, the programming-free interface designed in this invention allows users to easily change different packaging methods, package different types of items, and select whether to perform defect detection. The entire system only requires modification of relevant parameters to directly achieve packaging and sorting of different items through the user interface. Attached Figure Description

[0016] Figure 1 This is a flowchart of an intelligent sorting and packaging system based on machine vision;

[0017] Figure 2 This is a schematic diagram of positive samples collected using a Basler acA2500-14gc.

[0018] Figure 3 This is a schematic diagram of negative samples collected using a Basler acA2500-14gc.

[0019] Figure 4 This is the schematic diagram of the SE module;

[0020] Figure 5 This is a schematic diagram of the improved CFA module;

[0021] Figure 6 This is a comparison of some detections of CFA before and after the improvement in the MVTEC dataset;

[0022] Figure 7 It is a partial detection map of negative samples from an improved CFA algorithm;

[0023] Figure 8 This is a diagram of the entire system's no-programming operation interface. Detailed Implementation

[0024] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings;

[0025] Appendix Figure 1 This is a system flowchart of an intelligent sorting and packaging system based on machine vision. The system inputs the type of item, packaging method, and task through a no-programming module. After starting the system via the no-programming interface, the items are placed on a conveyor belt, and Basler cameras capture images of the items on the conveyor belt in real time. An improved YOLO network identifies the items on the conveyor belt, and an improved CFA defect detection module performs defect detection. If an item has a defect, the robot classifies it to the defect handling area; if no defect is found, the robot sorts and packages it according to the preset packaging task. This system combines advanced visual recognition technology and robot operation to achieve efficient and accurate defect detection and sorting / packaging of items. The detailed steps for implementing the above functions are as follows:

[0026] First, an industrial Basler acA2500-gc camera was used to capture images of packaged food items of varying shapes, sizes, and colors at a resolution of 1900 pixels x 1900 pixels. Since the images captured on an industrial production line are of moving objects, blurriness can occur. Therefore, the Basler camera's white balance function was used during the acquisition process to ensure that the objects in the captured images were sharper.

[0027] Appendix Figure 2 and attached Figure 3 These are positive and negative samples collected using a camera, used for training the vision module.

[0028] This invention utilizes an improved YOLO deep neural network to identify small bagged food items to be inspected on an assembly line. The loss function of the YOLO deep neural network consists of the following parts: localization loss, classification loss, and confidence loss. The localization loss includes the bounding box center coordinates (x, y), width (w), and height (h). An angle (θ) loss function and a dynamic weight adjustment strategy are added. The overall form is shown in equation (1).

[0029] L=αL loc +βL CSC +γL conf (1)

[0030] Where: L loc It is the location loss, L CSC It is classification loss, L conf This is the confidence loss; the weights α, β, and γ are adaptively adjusted according to the actual detection object, as shown in equations (1) and (2). Among them, C1 and C2 are constants, and the weights will be adaptively adjusted according to different detection objects to achieve more accurate detection.

[0031]

[0032] γ=C2·|θ| (3)

[0033] Weight α improves the model's localization accuracy when detecting objects of different sizes, while weight γ ensures more accurate prediction of target pose confidence when detecting object pose. The localization loss consists of the following components: coordinate loss L... coord Used to predict the center coordinates (x, y) of the bounding box; size loss L size Used to predict the width and height of the bounding box; angular loss L angle The angles used to predict the bounding box, together with the other two, constitute the localization loss, which is in the form of Equation (4).

[0034] L loc =L coord +L size +Langle (4)

[0035] Appendix Figure 4 This is the schematic diagram of the SE module. The input X undergoes arbitrary transformation F. tr The output is transformed into U, with dimensions H×W×C. Assuming that the output U is not optimal and varies in importance for each channel, global average pooling F is performed on each output channel. sq (·), generate a 1×1×C vector, as shown in equation (5):

[0036]

[0037] Then through F ex (2,W), this process involves two fully connected layers and two activation functions, which will yield the weights for each channel, as shown in equation (6):

[0038] s = F ex (z,W)=σ(g,(z,W))=σ(W2δ(W1z)) (6)

[0039] Two fully connected layers, W1 and W2, process the vector z obtained in the previous step to obtain the channel weight values ​​s. Different values ​​in s represent the weight information of different channels, and different weights are assigned to the channels.

[0040] Finally, U is multiplied by the generated feature vector s (1×1×C) to obtain the final output, as shown in equation (7):

[0041] X′=F scale (u c s c ) = s c u c (7)

[0042] That is, the H*W values ​​of each channel in the feature map U are multiplied by the weights of the corresponding channels in s. The SE attention mechanism module is integrated into the network after WideResNet50-2 as a feature extractor. (Appendix) Figure 5This is a schematic diagram of the improved CFA defect detection module. Inserting the SE module into the WideResNet50-2 bottleneck involves: first, performing three convolutions and batch normalization operations; then, compressing the feature map spatially using global average pooling; followed by two fully connected layers and ReLU and Sigmoid activation functions to generate channel weights; finally, scaling the original feature map and performing residual connections to output the final feature map. The improved CNN acts as a feature extractor, applying patch descriptions and a memory library. During training, contrastive supervised learning is performed based on a superimposed hypersphere (coupled hypersphere) created centered on the memory features. During testing, patch description features obtained from any sample in the test set are matched with the nearest neighbor memory features searched in the memory library to achieve defect detection. The improved defect detection module can adaptively adjust each channel, improving the network's generalization performance and learning performance on different data distributions, reducing training complexity and the need for frequent model changes.

[0043] Appendix Figure 6 This image shows a comparison of the CFA module before and after the improvement in MVTEC inspection. The improved CFA module is better able to adapt to different item inputs and captures important features more effectively, thus reducing the probability of false detections during defect detection. (Attached) Figure 7 This is a detection map of the acquired negative samples by the improved CFA module.

[0044] This invention employs parallel robotic arms to perform sorting and packaging tasks. (See attached image) Figure 8 This is a diagram of the no-programming interface for the entire system. Select the types of items to be packaged, as well as the number of layers, columns, and rows. Define the number of layers as L, the number of columns as C, the number of rows as R, and the dimensions of the small package as W×H×D (length×width×height). Furthermore, set the initial position of the item to (x0, y0, z0), the placement position to (x1, y1, z1), and i, j, and k to represent the current row, column, and layer of the package, respectively, all initially set to 1. The following relationships exist:

[0045] x1 = x0 - (j-1) × H

[0046] y1=y0+(i-1)×W

[0047] z1=z0+(k-1)×D (8) If there are defects, they will be sorted out; otherwise, they will be packaged and placed according to formula (8).

[0048] The entire system encapsulates the perception results of the vision module and the motion planning path of the parallel robotic arm, achieving programming-free operation. The system can handle different packaging tasks by simply inputting relevant parameters, enabling it to sort out defective items and package intact items according to requirements. (Attached) Figure 8 This is a diagram of the no-programming operation interface for the entire system. Through the interface, different items can be selected, and the corresponding rows, columns, and layers can be input according to different packaging tasks. The interface allows selection of either a packaging task or a defect detection task. If a packaging task is selected, small items will be packaged directly; if a defect detection task is selected, defective small items will be sorted out, and intact items will be packaged according to the input requirements.

Claims

1. A depth vision based intelligent sorting and packing system for items, characterized in that, It comprises three modules: target detection module, defect detection module and intelligent packaging module; The target detection module uses an improved YOLO deep neural network to quickly identify the pose of the items to be sorted; Then, the defect detection module detects defects in the items on the production line through an improved CFA defect detection model, and feeds back the defect detection results to the intelligent packaging module; in the intelligent packaging module, if the items are defect-free, the parallel robot hand will pick up and place the items into the packaging box according to the requirements of the current packaging task; if the items have defects, the parallel robot hand will pick up the defective items and place them at another designated position; finally, the sorting and packaging tasks are completed simultaneously on one production line, and the packaging method is quickly changed according to the requirements without programming; The intelligent sorting and packaging system for items is divided into a vision module and a motion module of a parallel robot hand, wherein the vision module comprises a target detection module and a defect detection module; each module is packaged and integrated to achieve programming-free; for different packaging tasks, relevant parameters are modified, and the parallel robot hand can plan the motion path autonomously to sort and package the small bagged food after defect detection; The type, packaging method and task of the items are input through the programming-free module; after starting the system through the programming-free interface, the items are placed on the conveyor belt, and the Basler camera captures the images of the items on the conveyor belt in real time; the improved YOLO network identifies the items on the conveyor belt, and the improved CFA defect detection module is used for defect detection; if the items have defects, the parallel robot hand will classify them to the defect processing area; If there is no defect, the parallel robot hand will sort and package according to the preset packaging task; The perception results of the vision module and the route planned by the parallel robot hand are packaged to achieve programming-free, and the entire system can achieve the sorting of defective parts and the packaging of intact parts according to the requirements by only inputting relevant parameters for different packaging tasks; the programming-free operation interface of the entire system, by selecting different items through the operation interface, inputs the corresponding row, column and layer according to different packaging tasks; the operation interface can select a packaging task or a defect detection task, if the packaging task is selected, the small items will be directly packaged; If the defect detection task is selected, the small items with defects will be sorted out, and the intact items will be packaged according to the input requirements; First, an industrial Basler acA2500-gc camera is used to collect a number of small bagged food with different shapes, sizes or colors, with a resolution of 1900 pixels*1900 pixels; during the collection process, the white balance function of the Basler camera is used to ensure that the items in the captured images are clearer; The positive and negative sample examples collected by the camera are used for vision module training; The improved YOLO deep neural network is used to identify small bagged food to be detected on a pipeline; the loss function of the YOLO deep neural network is composed of the following parts: positioning loss, classification loss and confidence loss; the positioning loss includes the center coordinates (x, y) of the boundary box, the width w and the height h, and the overall form of the angle θ loss function and the dynamic weight adjustment strategy is as formula (1): （1）； Wherein: is a positioning loss, is a classification loss, is a confidence loss; each weight , , is adaptively adjusted according to the actual detection object, such as formula (1), (2); wherein is a constant, and the weight is adaptively adjusted according to different detection objects, so as to realize accurate detection; （2）； （3）； weight The positioning accuracy of the model can be improved when detecting different sizes of objects, weight When detecting the posture of an object, the confidence prediction of the target posture is more accurate; the positioning loss is composed of the following: coordinate loss for predicting the center coordinates (x, y) of the bounding box; size loss for predicting the width and height of the bounding box; angle loss for predicting the angle of the bounding box, and the three together constitute the positioning loss, and the overall form is as formula (4); （4）； In the SE module, the input X is arbitrarily transformed into U, which has dimensions H x W x C. Assuming that the output U is not optimal, global average pooling is performed on each output channel, with different importance levels for each channel to generate a 1 x 1 x C vector, as shown in equation (5): （5）； Then, the process is This process contains two fully connected layers and two activation functions, and the weight of each channel is obtained, as shown in equation (6): （6）； Two fully connected layers , The vector z obtained in the previous step is processed to obtain a channel weight value s. Different values in s represent weight information of different channels, and different weights are given to different channels. The U and the generated feature vector s (1*1*C) are multiplied to obtain the final output, as shown in formula (7): （7）； That is, the H*W values of each channel in the feature map U are multiplied by the weight of the corresponding channel in s; the SE attention mechanism module is integrated into the network after WideResNet50-2 as a feature extractor; The improved CFA defect detection module schematic diagram; the SE module is inserted into the bottleneck of WideResNet50-2, including: first, three convolution and batch normalization operations are performed, then the feature map is compressed in the spatial dimension through global average pooling, and then two fully connected layers and ReLU, Sigmoid activation function are used to generate channel weights, finally the original feature map is scaled and adjusted, and residual connection is performed to output the final feature map; the improved CNN is used as a feature extractor, and patch description and memory library are applied thereto; in the training stage, the contrastive supervised learning is carried out based on the superimposed hyper-sphere, i.e. coupled hyper-sphere, created with the memory features as the center; in the test stage, the patch description features obtained from any sample in the test set are matched with the nearest neighbor memory features searched in the memory library, so as to realize defect detection; the improved defect detection module can adaptively adjust each channel; The parallel machine hand is used to carry out the task of sorting and packaging; the kind of article to be packaged is selected according to the requirement, and the layer number, column number and row number of the package are selected; the layer number of the package is set as L, the column number is set as C, the row number is set as R and the size of the small package is set as WxHxD; the initial position of the article is set as , the placement position is set as , i, , k respectively represent the row, column and layer of the current package position, and are all initially 1; the following relationship exists: ； ； （8）； If there is a defect, it will be sorted out, and if there is no defect, it will be packaged and placed according to formula (8).

2. The deep vision-based smart sorting and packing system of claim 1, wherein, Basler acA2500-14gc industrial camera is used to collect visual images of different types of products; the collected images are divided into two categories: normal images and images with defects; In the data set production process, one fourth of the positive samples and the negative samples are used as test samples together, and the remaining three fourths of the positive samples are used as training samples; the training of the defect detection model only needs to use positive samples, and the target detection needs to apply positive samples and negative samples.

3. The deep vision-based smart sorting and packing system of claim 1, wherein, The improved YOLO deep neural network is used to identify small bagged food in the collected pictures; By introducing a rotation angle loss And combined with the dynamic weight adjustment strategy to realize the efficient detection of the size and rotation angle of the object; the improved YOLO deep neural network adjusts the weights of the positioning loss, size loss and angle loss dynamically, ensures that the model can adaptively adjust the focus when detecting objects of different sizes and angles, and ensures the detection accuracy.

4. The deep vision-based smart sorting and packing system of claim 1, wherein, The defect detection module uses the improved CFA algorithm to effectively detect defects in small bagged food, and the improved CFA algorithm includes a learnable patch descriptor for learning and embedding target-oriented features and a scalable memory library independent of the size of the target data set; the migration learning technology is adopted, the patch description and memory library are applied to the feature extractor CNN to increase the density of normal features, so as to clearly distinguish abnormal features.

5. The deep vision-based smart sorting and packing system of claim 4, wherein, The improved CFA algorithm uses a WideResNet50-2 network, effectively reduces the risk of overfitting by expanding the number of channels in each convolutional layer by 2, and then inserts an SE attention mechanism module, so that the model better adapts to the changes of different input samples, and does not significantly increase the parameter quantity of the model. The improved network is used to extract the surface features of the articles. In the training process, the features of different types of articles can be effectively extracted and aggregated together. In the test stage, the patch description features obtained from any sample in the test set are matched with the nearest neighbor memory features searched in the memory library, and a heat map representing the degree of abnormality is generated. Finally, the scoring function is used to calculate the scoring map of the abnormal positioning in the heat map, and the defect detection of the article is realized.

6. The deep vision-based smart sorting and packing system of claim 1, wherein, The core of the intelligent packaging module is to allow the user to set parameters, including the type of the article, the size of the small package, and the required number of layers, columns and rows of the package. The intelligent packaging module enables the system to intelligently plan the placement position of each article and the grabbing motion trajectory of the robot hand according to the packaging task.

Images