Material sorting method and device based on deep learning target detection and multi-label classification
Through deep learning object detection and multi-label classification methods, combined with YOLO v5 and GPU platform, the problems of background changes, adhesion recognition, CPU delay and multi-manufacturer requirements in material sorting are solved, and efficient and accurate material sorting and multi-manufacturer are achieved. Tag identification to adapt to different customer needs.
Patent Information
- Application Number
- CN202410926210.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-01
AI Technical Summary
The existing material sorting technology has problems of low efficiency and insufficient accuracy in background and light changes, sticky material recognition, CPU processor delay and multiple manufacturer demand differences.
The target detection and multi-label classification method based on deep learning are adopted, and the YOLO v5 target detection system and multi-label classification model are combined with the GPU platform to realize accurate material identification and multi-label sorting, and the weight coefficient is set to meet different customer needs.
It improves the accuracy and efficiency of material sorting, reduces the workload of repeated labeling, adapts to the sorting requirements of different manufacturers, and accelerates the calculation speed and reduces the power through GPU.
Smart Images

Figure CN118887464A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of material sorting and detection, and more specifically, to a material sorting method and device based on deep learning target detection and multi-label classification. Background Art
[0002] For the sorting method of material quality inspection, such as the sorting of fruits and vegetables, the original method mainly relies on manual inspection and sorting methods. However, this method is usually inefficient due to its lack of intelligence and automation. It also introduces human subjective judgment. The standards vary from person to person, and the quality of the sorted fruits and vegetables is not uniform.
[0003] At present, with the continuous improvement of automation applications, when sorting materials, the method based on material image color recognition is used for detection. However, when using this method for material identification, the following problems will arise:
[0004] (1) It is greatly affected by the background and light. When the foreground and background of the material are very different, the target can be segmented by image color recognition. However, when the on-site environment changes, such as the light source weakens, the accuracy of target segmentation by this method will decrease;
[0005] (2) When materials are stuck together, traditional image recognition methods cannot accurately identify and distinguish the stuck materials;
[0006] (3) The traditional image recognition method is generally run on a CPU processor, but other software is usually run on the CPU processor, which will cause the detection process to be time-consuming and unstable, affecting the material sorting efficiency;
[0007] (4) At present, when classifying materials, different manufacturers may have different sorting requirements for the same material. For example, for a small yellow apple, some manufacturers want to sort it into the small fruit recycling bin, while other manufacturers may sort it into the yellow fruit recycling bin. The requirements of different manufacturers may be different. Some manufacturers even sort it into the small fruit recycling bin for a period of time, and into the yellow fruit recycling bin for another period of time. For materials with a fixed category label, such as yellow fruit, one manufacturer may consider lighter yellow to be yellow fruit, while other manufacturers may consider darker yellow to be yellow fruit. For these, traditional image recognition methods cannot make judgments. Summary of the invention
[0008] The present invention provides a material sorting method and device based on deep learning target detection and multi-label classification, which can detect material defects more accurately and enable one material to correspond to multiple labels for multi-label material sorting.
[0009] In order to achieve the above object, the technical solution provided by the present invention is:
[0010] A material sorting method based on deep learning target detection and multi-label classification, the method comprising:
[0011] Get the original image of the material;
[0012] Label the original image to obtain a deep learning target detection dataset;
[0013] Using the deep learning target detection data set, deep learning model training is performed and target materials are extracted;
[0014] Establish a multi-label classification model, input the multi-label classification model into the image of the target material to obtain multiple labels for an image [a 1 ,a 2 ,a 3 ,...a i ], sort materials according to labels;
[0015] a is the corresponding value of different label outputs, and i is the number of labels.
[0016] Furthermore, in the process of annotating the original image to obtain a deep learning target detection dataset, when annotating the original image, each image is annotated with a label category.
[0017] Furthermore, the YOLO v5 target detection system is used to train the deep learning model; the YOLOv5 target detection system includes a conv layer, a C3 layer and an SPPF layer; the deep learning target detection dataset passes through two conv layers and then the C3 layer; after the C3 layer, it passes through another conv and C3 layer to reach the SPPF layer; after the SPPF module, it enters the detect module and then outputs.
[0018] Furthermore, the corresponding values of multiple label outputs [a 1, a 2, a 3, ...a i ] Assign a value by comparing the set threshold with the probability value;
[0019] The multi-label classification model outputs different probability values for different labels.
[0020] When the probability value is greater than the set threshold, a i is 1;
[0021] When the probability value is not greater than the set threshold, a i is 0;
[0022] Materials are sorted according to the assignment results.
[0023] Furthermore, the corresponding values of different label outputs [a 1, a 2, a 3, ...a i ] and then set different weight coefficients accordingly, and sort the materials according to the weight coefficients.
[0024] Furthermore, the multi-label classification model includes a conv layer, a Block network module, an attention mechanism, an average pooling layer, and a fully connected layer;
[0025] The image of the extracted target material passes through the conv layer, is sent to the Block network module and the attention mechanism, then passes through the convolution layer and a Block network module and the attention mechanism, and then passes through the average pooling layer. After being sent to the fully connected layer, the probability value is obtained through the Sigmoid function and then assigned to the output.
[0026] Furthermore, the probability value is obtained in the following manner:
[0027]
[0028] S(x) is the probability value, ranging from 0 to 1, and x is the weight.
[0029] Furthermore, the weight coefficient ranges from 0.01 to 0.99.
[0030] The present invention also provides a material sorting device based on deep learning target detection and multi-label classification.
[0031] A shooting component is used to obtain the original image of the material;
[0032] Deep learning model and multi-label classification model deployment platform, used to deploy trained deep learning models and multi-label classification models;
[0033] The host computer outputs the sorting results according to the labels.
[0034] Furthermore, the photographing component is an area array camera, which can collect color images of materials in real time;
[0035] The deep learning model and multi-label classification model deployment platform is the NVIDIA ORIN NX GPU platform.
[0036] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:
[0037] (1) The material sorting method based on deep learning target detection and multi-label classification of the present invention establishes a deep learning model to extract target materials, including adhered targets, and performs target recognition through deep learning detection methods with higher accuracy, thereby improving the material sorting accuracy.
[0038] A multi-label classification model is established to set multiple label categories for a material. During sorting, customers can set a material of a certain label category as the final sorting category according to their needs. When the required label category changes, there is no need to re-label the sample material, and the material of another label category can be changed to the final sorting category, which greatly reduces the sorting workload.
[0039] (2) The material sorting method based on deep learning target detection and multi-label classification of the present invention establishes a multi-label classification model that can output multiple label categories for an image as needed to perform multi-label material sorting.
[0040] (3) The material sorting method based on deep learning target detection and multi-label classification of the present invention, the simplified YOLO v5 target detection system can perform target detection, including adhered targets, with higher recognition accuracy and faster computing speed.
[0041] (4) The material sorting method based on deep learning target detection and multi-label classification of the present invention sets weight coefficients for label categories. Each defect label category has a corresponding weight coefficient. Customers can set different weight coefficients according to their own judgment criteria to achieve the purpose of sorting. The more serious the defect, the greater the weight coefficient. The degree of defect is identified according to the weight coefficient, so that defect screening is performed according to the customer's requirements for defects. In addition, the amount of screening can be set according to the weight coefficient. The higher the weight coefficient is set, the more serious the defect is, and the less material is screened out. Conversely, the lower the weight coefficient is set, the lighter the defect is, and the more material is screened out.
[0042] (5) The material sorting device based on deep learning target detection and multi-label classification of the present invention, the deep learning model and multi-label classification model deployment platform is the GPU platform of NVIDIA ORIN NX, which can take advantage of the high computing power of the graphics card to speed up the calculation speed; NVIDIA ORIN NX also has the advantages of low power consumption, fast speed, stable system, and small structure. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0044] Figure 1 A schematic diagram of a process for a material sorting method based on deep learning target detection and multi-label classification provided in an embodiment;
[0045] Figure 2YOLO v5 target detection system architecture diagram provided for the embodiment;
[0046] Figure 3 A multi-label classification model architecture diagram provided for an embodiment;
[0047] Figure 4 A schematic diagram of the device framework of a material sorting device based on deep learning target detection and multi-label classification provided in an embodiment. DETAILED DESCRIPTION
[0048] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0049] For the convenience of explanation, this embodiment takes fruit and vegetable sorting as an example. When sorting fruits and vegetables, image recognition is used for sorting. When the foreground and background are segmented, if the environment changes, the segmentation accuracy will be reduced. In addition, fruits and vegetables often stick together during transmission, and it is difficult to segment the objects that stick together during image segmentation.
[0050] Another common practical situation is that when sorting target materials, a target material may be marked with multiple defects. For example, a pistachio may have three label categories: small fruit, deformed fruit, and yellow fruit. Different customers may set it to a certain label category for sorting, or even a customer may set it to a certain label category for sorting in different time periods, and use another label category for sorting in another time period. At present, the market generally follows customer needs and labels it as the customer-specified category when labeling data. For example, if a customer thinks it is a yellow fruit, then the target image of this category needs to be classified as a yellow fruit. If another customer thinks it is a small fruit, then the target image of this category needs to be classified as a small fruit. And so on. This method will greatly increase the workload of labeling.
[0051] In some cases, for a target material that has been marked with a certain label, such as a cherry tomato with a defect category of small fruit, different customers have different definitions of small fruit. Some customers believe that a fruit with a diameter less than 1 cm is a small fruit, while others believe that a fruit with a diameter less than 1.2 cm is a small fruit. In this case, there is no unified sample for classification. The current solution is to label samples according to customer needs, which will greatly increase the labeling workload and increase labeling repeatability.
[0052] like Figure 1As shown, this embodiment provides a material sorting method based on deep learning target detection and multi-label classification, which specifically includes the following steps:
[0053] S10, obtaining the original image of the material;
[0054] S20, annotating the original image to obtain a deep learning target detection data set;
[0055] S30, performing deep learning model training and extracting target materials through the deep learning target detection data set;
[0056] S40, establish a multi-label classification model, input the multi-label classification model into the image of the extracted target material, and obtain multiple labels for an image [a 1 ,a 2 ,a 3 ,...a i ], sort materials according to labels;
[0057] a is the corresponding value output by different labels, and i is the number of labels. The label is the sorting category of a target material A. For example, a pistachio has six sorting categories: good fruit, yellow fruit, skin fruit, closed fruit, deformed fruit, and small fruit. The six sorting categories output corresponding values respectively. The label of a pistachio is [a 1 ,a 2 ,a 3 ,a 4 ,a 5 ,a 6 ].
[0058] In this embodiment, by marking a target material with multiple labels, customers can set a certain label category to sort materials according to their needs. In this way, the sample material is marked once, and a certain label category is selected for material sorting according to customer needs. When the sorting requirements change, there is no need to re-mark, which greatly reduces the workload of material sorting and improves sorting efficiency.
[0059] In order to better understand the material sorting method based on deep learning target detection and multi-label classification provided in this embodiment, each step will be described in detail.
[0060] S10, obtaining the original image of the material;
[0061] During the material transmission process, an area array camera is used to collect data in real time. For example, a Hikvision area array camera is used to collect the original image of the material in real time. In this embodiment, the original image data size of the material is a 640*640 RGB color image.
[0062] In other embodiments, other cameras may be used to obtain the original image of the material, as long as a color image of the material can be obtained, and the method of obtaining the original image of the material is not limited.
[0063] S20, annotating the original image to obtain a deep learning target detection data set;
[0064] The original images of the materials are labeled. During labeling, only one label category is labeled for the material in each image, so as to obtain a deep learning target detection dataset for deep learning model training and extracting target materials.
[0065] In order to improve the efficiency of labeling the original image of the material, the deep learning labeling tool labelimg is used to label the image in this embodiment. Labelimg is an open source data labeling tool that can label multiple image formats, making the labeled images suitable for multiple deep learning models.
[0066] S30, performing deep learning model training and extracting target materials through the deep learning target detection data set;
[0067] In this embodiment, the YOLO v5 target detection system is used for deep learning model training.
[0068] The YOLO series of deep learning models use a grid approach to divide each image into multiple grids. Each grid is responsible for predicting one or more objects. In simple terms, each grid can generate a prediction box. The process of generating a prediction box in a grid is that the grid memory contains several (usually three) prediction box templates, that is, "anchors". Each anchor has a preset width, height, coordinates, and confidence. The confidence indicates the probability of an object existing in the grid. During the training process, the confidence of the anchor in the grid where the midpoint of the true annotation box falls is set to 1, indicating that there is an object in the grid, indicating that the grid where the anchor is located has an object, and the confidence of other anchors without prediction boxes is 0. If the difference in width and height and the difference in coordinates between the anchor and the true box are regarded as losses, and the binary cross entropy is regarded as the loss of confidence, then the object detection problem will be greatly simplified to a simple regression prediction and classification problem. At present, the YOLO series algorithms range from YOLOv1 to YOLOv8. In this embodiment, the YOLOv5 target detection system with higher accuracy, stronger generalization ability and faster speed is used to train the deep learning model and perform defect recognition and judgment.
[0069] The current YOLO v5 network structure is divided into three parts: backbone network, neck structure, and head structure. However, the existing network structure is time-consuming. The network is now simplified to make the simplified network faster.
[0070] like Figure 2 As shown in the figure, the simplified YOLO v5 target detection system includes a conv layer, a C3 layer and an SPPF layer; the deep learning target detection data set passes through two conv layers and then the C3 layer; after the C3 layer, it passes through another conv and C3 layer to reach the SPPF layer; after the SPPF module, it enters the detect module and is output.
[0071] In this embodiment, the original image data size of the material is 640*640, the input image size of the YOLO v5 target detection system is 640*640, and after passing through the conv layer, C3 layer and SPPF layer, 1 image is output. Each image has 3 prediction boxes, and each box has 6 parameters, which are XY coordinates, width and height, whether the target is a material, and the probability of being a material.
[0072] The original material image obtained in this embodiment has a high resolution and the target foreground image is large. There is no need to perform 16 or 32 times downsampling. Only 8 times downsampling is needed to extract the target object, including the adhered target, with fast running speed and high efficiency.
[0073] The target material, including the adhered target material, is extracted through the YOLO v5 target detection system to obtain an image of the extracted target material.
[0074] S40, establish a multi-label classification model, input the multi-label classification model into the image of the extracted target material, and obtain multiple labels for an image [a 1 ,a 2 ,a 3 ,...a i ], sort materials according to labels.
[0075] In existing multi-category classification network models, a softmax function is usually used to obtain categories after target feature extraction. This method is not suitable for obtaining label categories in this embodiment.
[0076] like Figure 3 As shown, in this embodiment, a multi-label classification model is established to obtain label categories.
[0077] The multi-label classification model includes a conv layer, a Block network module, an attention mechanism, an average pooling layer and a fully connected layer; the image of the extracted target material passes through the conv layer, is sent to the Block network module and the attention mechanism, then passes through the conv layer and a Block network module and the attention mechanism, then passes through the average pooling layer, is sent to the fully connected layer, and the probability value is obtained through the Sigmoid function and then assigned and output.
[0078] In this embodiment, through the multi-label classification model, the corresponding value of each target material output by different labels [a 1 a 2 a 3 ...a i ]The value is assigned by comparing the set threshold with the probability value.
[0079] The multi-label classification model outputs different probability values for different labels. When the probability value is greater than the set threshold, a i is 1; when the probability value is not greater than the set threshold, a i is 0; sort materials according to the assignment results.
[0080] Specifically, in this embodiment, the original image of 640*640 passes through the conv layer in the multi-label classification model, the image size becomes 320*320, the number of channels is adjusted to 64, and the image with parameters of 320*320*64 is sent to the Block network module and the attention mechanism, the image parameters become 160*160*256, and then passes through the conv layer, the image parameters become 80*80*512, and then passes through a Block network module and the attention mechanism, the image parameters become 40*40*1024, and then passes through an adaptive average pooling layer, and then is sent to the fully connected layer; after the probability value is obtained by the Sigmoid function, the output is assigned.
[0081] The calculation formula for the probability value obtained by the Sigmoid function is:
[0082]
[0083] S(x) is the probability value, ranging from 0 to 1; x is the weight, obtained through multi-label classification model operation; e is a natural constant.
[0084] Still taking pistachios as an example, a pistachio is marked with six label categories, namely good fruit, yellow fruit, skin fruit, closed fruit, deformed fruit, and small fruit. Any pistachio image is set with these six label categories.
[0085] A pistachio image is passed through the Sigmoid function of the multi-label classification model to obtain 6 probability values. In actual use, a threshold is set first. The threshold is set based on experience, and the threshold and probability value are compared to determine whether the first label is a good fruit. If it is, the fruit is considered to be a good fruit; if not, all subsequent probability values are determined to be greater than the threshold. If greater, it is set to 1, otherwise it is 0. For example, if the first image is a good fruit, the label output is [1,0,0,0,0,0], and the second image is a deformed fruit and a small fruit, the label output is [0,0,0,0,1,1]. In some cases, a material image will have multiple 1 labels, and customers can later determine which category the category is finally judged to be based on their own judgment criteria.
[0086] When materials need to be sorted by combining labels with weight coefficients, the corresponding values of different label outputs [a 1, a 2, a 3, ...a i ] and then set different weight coefficients accordingly, and sort the materials according to the weight coefficients.
[0087] Specifically, when a pistachio image is output with 6 label category probability values by the multi-label classification model, 6 weight coefficients need to be output, which represent the severity of a defect. For example, if a pistachio defect is a small fruit, the larger the weight coefficient, the smaller the target fruit diameter. Customers can sort according to the weight coefficient of the small fruit defect later. Taking pistachios as an example, if the image defects are deformed fruits and small fruits, the labels are [0,0,0,0,1,1; 0,0,0,0,0.58,0.62], and 6 labels become 12. The first 6 are label categories, and the latter are corresponding label weight coefficients. If the label is 0, the weight coefficient is 0. If the label is 1, the weight coefficient is set according to the severity of the defect. The weight coefficient here is different from the weight meaning in the above. The weight coefficient here is manually assigned according to experience. The principle of weight coefficient assignment setting is: the more serious the defect, the larger the weight coefficient. Finally, sorting is performed according to the weight coefficient, and the weight coefficient range is 0.01 to 0.99.
[0088] According to the multi-label classification model with weight coefficients, each defect label category has a corresponding weight coefficient. Customers can set different weight coefficients according to their own judgment criteria to achieve the purpose of sorting, reduce the work of repeated sample screening, and be more conducive to practical use.
[0089] like Figure 4As shown, the present invention also provides a device for a multi-label material sorting method based on deep learning target detection, including a shooting component, a deep learning model and a multi-label classification model deployment platform and a host computer, wherein the shooting component is used to obtain the original image of the material; the deep learning model and the multi-label classification model deployment platform are used to deploy the trained deep learning model and the multi-label classification model; the host computer is used to output the sorting results according to the labels.
[0090] In a specific implementation scenario, the shooting component is an array camera that can capture color images of materials in real time. The deep learning model and multi-label classification model deployment platform is the GPU platform of NVIDIA ORIN NX. Taking advantage of the high computing power of the NVIDIA ORIN NX platform graphics card, the deep learning model and multi-label classification model are deployed on the NVIDIA ORIN NX platform to speed up the reasoning speed; and NVIDIA ORIN NX also has the advantages of low power consumption, fast speed, stable system, and small structure. The host computer is used to execute the host computer software. The host computer can include but is not limited to: computers, servers. In addition, the host computer can also include mobile devices such as laptops and tablets, which are not limited here.
[0091] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and this document is not limited here. The specific implementation methods described do not constitute limitations on the scope of protection of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.
Claims
1. A material sorting method based on deep learning target detection and multi-label classification, characterized by: The method comprises: Get the original image of the material; Label the original image to obtain a deep learning target detection dataset; Using the deep learning target detection data set, deep learning model training is performed and target materials are extracted; Establish a multi-label classification model, input the multi-label classification model into the image of the target material to obtain multiple labels [a1, a2, a3, ... a i ], sort materials according to labels; a is the corresponding value of different label outputs, and i is the number of labels.
2. The method according to claim 1, characterized in that: The original image is annotated to obtain a deep learning target detection data set. When the original image is annotated, each image is annotated with a label category.
3. The method according to claim 2, characterized in that: The YOLO v5 target detection system is used to train the deep learning model; the YOLO v5 target detection system includes a conv layer, a C3 layer and an SPPF layer; the deep learning target detection data set passes through two conv layers and then the C3 layer; after the C3 layer, it passes through another conv and C3 layer to reach the SPPF layer; after the SPPF module, it enters the detect module and then outputs.
4. The method according to claim 1, characterized in that: The corresponding values of multiple label outputs [a 1, a 2, a 3, ...a i ] Assign a value by comparing the set threshold with the probability value; The multi-label classification model outputs different probability values for different labels. When the probability value is greater than the set threshold, a i is 1; When the probability value is not greater than the set threshold, a i is 0; Materials are sorted according to the assignment results.
5. The method according to claim 4, characterized in that: The corresponding values of different label outputs [a 1, a 2, a 3, ...a i ] and then set different weight coefficients accordingly, and sort the materials according to the weight coefficients.
6. The method according to claim 4 or 5, characterized in that: The multi-label classification model includes a conv layer, a Block network module, an attention mechanism, an average pooling layer, and a fully connected layer; The image of the extracted target material passes through the conv layer, is sent to the Block network module and the attention mechanism, then passes through the convolution layer and a Block network module and the attention mechanism, and then passes through the average pooling layer. After being sent to the fully connected layer, the probability value is obtained through the Sigmoid function and then assigned to the output.
7. The method according to claim 6, characterized in that: The probability value is obtained as follows: S(x) is the probability value, ranging from 0 to 1, and x is the weight.
8. The method according to claim 5, characterized in that: The weight coefficient ranges from 0.01 to 0.
99.
9. A material sorting device based on deep learning target detection and multi-label classification, characterized in that: include, A shooting component is used to obtain the original image of the material; Deep learning model and multi-label classification model deployment platform, used to deploy trained deep learning models and multi-label classification models; The host computer outputs the sorting results according to the labels.
10. The device according to claim 9, characterized in that: The shooting component is an area array camera, which can collect color images of materials in real time; The deep learning model and multi-label classification model deployment platform is the NVIDIA ORIN NX GPU platform.
