Quality control and sorting system that can perform classification with parallel processing and deep learning on multi-channel conveyor belt

Abstract

The invention consists of a mechanical system developed to be used in the sorting operations carried out on the conveyor belt after many quality checks in the industry, especially in food sorting, and an algorithm that has parallel processing with this mechanical system.

Description

[0001] QUALITY CONTROL AND SORTING SYSTEM THAT CAN PERFORM CLASSIFICATION WITH PARALLEL PROCESSING AND DEEP LEARNING ON MULTI-CHANNEL CONVEYOR BELT

[0002] TECHNICAL FIELD

[0003] The invention consists of a mechanical system developed to be used in the sorting operations carried out on the conveyor belt after many quality checks in the industry, especially in food sorting, and an algorithm that has parallel processing specialty with this mechanical system.

[0004] BACKGROUND

[0005] Quality control and sorting systems are widely used in many areas, especially in food, plastic industry, mining industry and automotive spare parts. In the food industry, many processes are carried out before packaging the products. In the last stage, there are quality control procedures. Quality control operations are carried out on horizontal sorting machines, vertical sorting machines or on the conveyor belt. The quality control process performed on the conveyor belt is the final control process; This process is usually done with human workers. The biggest reason why the final check is done with the human eye is that the sorting machines used before the final check are generally image processing based sorting machines and it is difficult to achieve close to 100% accuracy with image processing technology. Another reason is that these horizontal and vertical sorting machines operate at very high speeds; 100% separation can never be achieved due to the fact that objects are seen at very different angles as they move through the air, sometimes they come close to each other, and sometimes due to problems in blowing. For these reasons, a final inspection with the human eye is always needed before the products are packaged. These final checks are usually made on the conveyor belt. However, as a result of the developments in deep learning and artificial intelligence, the final quality control operations on the conveyor belt are now carried out by robotic automation or half robot and half human power. Since the system will be more stable on the conveyor belt, the final quality control and sorting process can be more stable and with higher accuracy than horizontal and vertical sorting machines. 2-axis or 3-axis robot arms are generally used in image processing / deep learning supported robotic sorting operations on the conveyor belt. However, since a single robot arm will have to reach all relevant points of a conveyor belt of X cm width, the amount of sorting these robots can do per unit time may be limited. For these reasons, sometimes conveyor belts are operated slowly and sometimes more than one robot arm is used instead of a single robot arm. Also sometimes, robot arms may miss some faulty objects if they cannot reach them in terms of time. At this point, thanks to the system subject to the invention, the conveyor belt, which is X cm long horizontally, is divided into vertical partitions of equal length Y cm. Thus, X / Y parallelism is achieved.

[0006] Area scan cameras or line scan cameras are used for sorting on the conveyor belt. In order for area scanning cameras to be synchronized with the conveyor belt, encoder systems that measure the instantaneous speed of the belt are used. In this way, the fps (frame per second) value of the field scanning camera is instantly changed according to the instantaneous speed change of the conveyor belt. Again, in line scan cameras, the line scan speed of the camera is changed dynamically by using encoder systems. In both camera systems, objects may touch the above or below edges in some images, and sometimes half of the object (or a part of the object) is formed in frame number n and the other half (or part of the object) is formed in frame number n+1 (Figure 1b). This brings with it some problems in classification.

[0007] When a classification is desired with traditional deep learning techniques, data sets are created from images of the products to be classified. If a deep learning-based quality control application in the food sector or other sectors is needed, the data sets of defective and defect-free products are first created. For example, in a hazelnut sorting application, while a whole hazelnut without shell is a defect-free product, a hazelnut shell or cracked hazelnut may be a defective product. After the data sets of these products are created, convolutional neural network structures are trained with these data sets and the test accuracy is obtained. In order to increase the test accuracy, the data in the data sets can be increased or the network parameters can be fine-tuned. After these processes, the network structure working with high accuracy is run in real time on the system. When this network structure is run in real time on the conveyor belt, a situation can occur in Figure 1 b when traditional imaging methods are used. In the example in Figure 1 b, the defect-free data is selected as a square image and the defective data is selected as a triangle image. The first image that occurs when the system starts working at t=0 will be as in Figure 1 b. In Figure 1b, 10 full object images and 2 half or partial object images are formed. When the objects in the image are wanted to be classified, these 12 object images will be sent to the network structure and it will be predicted which class these 12 images belong to. However, it is expected that the classification of 2 half or partial images will be classified incorrectly. This incorrect classification process, which is explained in detail in Figures 1a and 1b, is encountered in many applications in the industry in a similar way.

[0008] Another problem of current systems is the classification and sorting problems that arise due to the products being adjacent to each other when performing image processing / deep learning based sorting on the conveyor belt. Even if the classification of adjacent objects is done with YOLO architectures, the fact that adjacent objects are pulled by the robot arm or the vacuum attached to the end of the robot arm can cause errors in the application. The fact that the products are separated from each other while the products moving on the conveyor belt ensures that both CNN-based classification and sorting with the robot arm or the vacuum attached to the robot arm are with high accuracy.

[0009] LIST OF FIGURES

[0010] Figure 1a: Representation of buffering of line images from a line scan camera according to traditional methods

[0011] Figure 1b: Representation of incorrect segmentations resulting from buffering of line images from a line scan camera according to traditional methods Figure 2: Single channel mechanical drawing

[0012] Figure 3: N channel Mechanical Drawing

[0013] Figure 4: Working principle of the system

[0014] Figure 5: Quality control and sorting process on a conveyor belt wide enough to carry a single row of products

[0015] Figure 6: Explanation of the quality control and sorting process on a time graph on a conveyor belt wide enough to carry a single row of products.

[0016] Figure 7: Single Channel Fine Sorting Algorithm

[0017] Figure 8: Quality control and sorting process on a conveyor belt wide enough to carry multiple rows of products

[0018] Figure 9: Working principle of quality control and sorting process on a time diagram on a conveyor belt wide enough to carry multiple rows of products Figure 10: An algorithm developed for quality control and sorting on a conveyor belt wide enough to carry multiple rows of products.

[0019] Correspondences of the numbers given in the figures:

[0020] 1 . Hopper

[0021] 2. First Groove

[0022] 3. Second Groove

[0023] 4. Defective Product

[0024] 5. Defective Product

[0025] 6. Defect-free Product

[0026] 7. Camera

[0027] 8. Processor

[0028] 9. Automation

[0029] 10. Vacuum Unit

[0030] 11 . Conveyor Belt

[0031] DETAILED DESCRIPTION OF THE INVENTION

[0032] The invention consists of a conveyor belt (11 ) system, a vacuum unit (10) and an electronic device that can detect defective / defect-free products and determine the vacuuming time of the products to be vacuumed as a result of this detection. In this context, the invention is a system that includes a hopper (1 ), a first groove (2) and a second groove (3); in order to sort and vacuum the defective (4, 5) and defect-free products (6) coming from the hopper (1 ), through the first groove (2) and the second groove (3); It is designed to include a conveyor belt (11 ) fed from at least two grooves and at least one camera (7) located on this conveyor belt (11 ), a processor (8) that has the capacity to process the images coming from the cameras (7) in parallel in case there is more than one camera (7), software on which the developed algorithm can work and that can communicate bidirectionally with the automation (9) system included in the invention, an automation system (9) that can establish communication between this processor (8) and the machine and operate the correct signals at the correct times and a vacuuming unit (10) that can sort out faulty objects according to the signals coming from this automation (9) system. There is a software within the processor (8) that can perform intelligent and precise sorting. Thanks to the automation (9) that communicates bidirectionally with the processor (8), the products detected as defective products (4, 5) are vacuumed by the vacuum unit (10) and sorted precisely. The technical effect of the invention is that the complete images of the products coming from the conveyor belt (11 ) are created in a single frame with dynamic buffering, and thus the defective / defect-free classification process is carried out with high accuracy, and the developed algorithm can be run in parallel if there is one or more cameras (7) in the system.

[0033] In the conveyor belt (11 ) system, at least two grooves (2, 3) are used to separate the objects to be sorted. In the system developed, the speed of the objects in the first groove (2) is Vol, and the speed of the objects in the second groove (3) is Vo2. The speed of the objects falling from the grooves (2, 3) to the conveyor belt (11 ) is always adjusted as Vo2> Vol. The speed of the objects on the conveyor belt that falling from the grooves (2, 3) will be Vkand Vkis always adjusted as Vk> Vo2. In this way, the separation of the objects from each other will be in optimum level and the fact that they do not touch each other on the vertical channel while moving on the conveyor belt are ensured.

[0034] The conveyor belt (11 ) can be divided into N equal channels (at least two) according to the application and the camera (7) to be used (See Figure 3). The functions of the processor (8) included in the invention, according to different configurations (single section conveyor belt, multi-section conveyor belt, product differences, etc.), are explained in detail and with sample events below.

[0035] The processor (8), which is one of the hardware structure elements defined above, includes a software that can perform the necessary calculation operations for the separation of defective products (4, 5) and defect-free products (6) and for the vacuuming unit (10) to vacuum the defective (4, 5) or defect-free products at the right time after this separation. The technical effect of the processor (8) is that the process is performed only when the defective product (4, 5) or defect-free product (6) is present in the image taken by the camera (7), and thus the classification of full-size objects in CNN (convolutional neural network structures) network structures (e.g. AlexNet, lnceptionV3, ResNet ...) is more accurate. According to the user request, the defective product (4, 5) or the defect-free product (6) can be vacuumed by the vacuuming unit (10). Below, a scenario is defined in which the defective product (4, 5) is vacuumed. The same scenario can be applied for the defect-free product (6). The difference is that the software defines which product will be vacuumed. In order to eliminate the problems of not being able to identify objects in the previous technique, in an embodiment of the invention, a single channel fed mechanical mechanism as shown in Figure 2 was developed. An example of how the mechanism in Figure 2 works in the field is given in Figure 5. With a line scan camera (7) placed on the single-channel conveyor belt (11), images of objects on the conveyor belt (11 ) can be obtained. However, within traditional methods, undesirable situations such as those in Figures 1 a and 1 b occur as a result of buffering the line images coming from the line scan camera (7) (“Buffer”: Buffering the images for processing). In order to eliminate these undesirable situations, within our invention, the line images coming from the camera (7) are analyzed by the software in the processor (8) and processed with a dynamic buffering method included in this software. The image in Figure 5 is the image of the time t = 0, and when the conveyor belt is running, the operating logic of the smart buffering system at different t times is carried out as in Figure 6.

[0036] Line scan cameras can generally have pixels in multiples of 1024 (2k(2048), 4k(4096), 8k(8192) and 16k(16384)). Again, the scanning speeds of the camera can be from 1 kHz to 100KHz. The method of the invention is suitable for all cameras with different pixels and different scanning speeds, and for a better understanding of Figure 5 and Figure 6, it is visualized through an example in which the camera used has a scanning speed of 1 kHz. In other words, an example where a line image occurs every 1 ms is given.

[0037] Within the system subject to the invention, it was previously stated that the images taken from the camera (7) are processed by a software in the processor (8). Thanks to the algorithm in the software in this processor (8), first the RGB image coming from the camera (7) is converted to a “monochrome (gray format)” image. In industrial applications, the conveyor belt (11 ) is generally selected as white in food sorting systems. With a threshold value applied to the monochrome image, the entire image is converted to a binary image. In this way, the conveyor belt (11 ) (i.e. the background) is converted to a black (8-bit value of 0), and the objects on the conveyor belt (11 ) are converted to a white (8-bit value of 255). Thanks to this transformation, the line images coming from the camera (7) are first converted to monochrome and then to binary images by the processor (8); if any of the pixels belonging to the object (at least 10 pixels side by side to ignore noise) are white as a result of the transformation, it is interpreted as there are pixels belonging to the object on this line image. If Figure 6, which shows the progression of Figure 5 in time, is examined in detail with this information; at t=0, the camera (7) and the conveyor belt (11 ) start working. When the line image coming from the camera (7) is analyzed at T=0 (when it is converted from RGB to monochrome and then from monochrome to binary), it is understood that all pixels belong to the background (i.e. all pixels are black (8-bit 0)). When the line pixels coming from the camera (7) are examined after T=0, it is understood that all pixels belong to the background, therefore pixels do not carry any information about the object. For this reason, the first line pixels coming from the camera (7) are not buffered in the dynamic buffer on the processor (8). When we look at Figure 6, this process continues until t=100ms. At the moment T= 100ms, the first pixels belonging to the object arrive. These first pixels are buffered in the previously created dynamic buffer. This buffering process ends at t=200ms. When the first line image after t=200ms is analyzed (that is, if the scanning speed of the camera (7) used is 1 Khz, at t=201 ms), the buffering process is terminated because there are no pixels belonging to the object. The image written to the dynamic buffer between t=100ms and t=200ms is given as an input image to the convolutional neural network, which was previously loaded into the software running within the processor (8), to be classified. In this example, the square image is considered as an defect-free image, and the images outside the square are considered as defective images. The image written to the buffer between t=100ms and t=200ms is sent to the convolutional neural network to be classified and this image is classified as a defect-free image. Since the image written to the buffer between t=100ms and t=200ms is classified as defect-free in the convolutional neural network, no communication process is performed between the processor (8) and the automation (9). When the line images coming from the camera (7) are analyzed between t=200ms and t=350ms, since there is no information about the object (or because all pixel values are 0 (zero) when the line images are converted to binary images), no line images coming in this time period are written to the dynamic buffer. At T=350ms, the first pixels belonging to the triangle image have arrived. When the image arriving at T=350ms is analyzed (converted from RGB to monochrome and then from monochrome to binary image), there will be pixels with a value of 255, so the line image arriving at t=350ms is buffered. This buffering process continues until t=500ms. When the first line image after t=500ms (that is, if the scanning speed of the camera used is 1 Khz, at t=501 ms) was analyzed, the buffering process was terminated because there would be no pixels belonging to the object. The buffered image from t=350ms to t=500ms is sent to the convolutional neural network in the software in the processor (8) to be classified. When this image is classified in the convolutional neural network, it is classified as a defective object. The detection of defective products (4, 5) and defect-free products (6) is determined by classifying the images coming from the camera (7) with the previously trained convolutional neural network structures in the software in the processor (8). If the reference images previously defined as defect-free match the images coming from the camera (7), it is classified as a defect-free product (6), and if they do not match, it is classified as a defective product (4, 5).

[0038] After the processor (8) detects the defective product (4, 5) and the defect-free product (6), in line with the requirements of the area where the system in question is used, a time calculation must be made for any of these products (faulty products or defect-free products may be required to be sorted) to be pulled by the vacuuming unit

[0039] (10). In the examples in Figure 5 and Figure 6, it is assumed that the conveyor belt

[0040] (11) moves 1 mm every 1 ms. For the vacuuming process to be carried out precisely, the most ideal time is calculated for the object to be vacuumed when its center point coincides with the center of the vacuum. The buffering time of the object is 500ms- 350ms = 150ms, i.e., a time of 75ms is needed to reach the center point of the object. Since the first image of the object to be vacuumed is formed at t=350ms, the image of its center point is formed at t=425ms. According to Figure 6, the distance between the line seen by the camera (7) and the exact center of the vacuum unit (10) is 1000 mm, that is, if the conveyor belt (11 ) advances 1 mm every ms, it becomes 1000 ms. According to all this calculation, the vacuuming process should occur in 425ms + 1000ms = 1425 ms. All these calculations were made by calculating that the speed of the conveyor belt (11 ) was always constant. However, in practice, the speed of the conveyor belt (11) may change instantaneously. Deviations in cases where there are speed changes can be calculated and integrated into the software.

[0041] In another embodiment of the invention, there is an encoder. This encoder contains the location information of the conveyor belt (11 ). The mentioned location information is included in the automation (9) content. In short, automation (9) can know where the position of the conveyor belt (11 ) is with the signal it receives from the encoder in each movement. In this case, there is no need to make time calculations to produce the signal required for the vacuum unit (10) to operate after the defective / defect-free product separation as given in the previous section. The position of a point of the conveyor belt (11 ) where it passes under the camera (7) and the position where it will pass under the equipment in the next stage (vacuuming unit (10) in the current system) can be determined via the encoder. Because the distance between the two units is fixed according to how many steps of the encoder it corresponds to. Therefore, the automation (9) waits for the encoder to take a step equal to this distance and generates a signal at the time of the step when the next equipment should work. Thus, the next equipment (vacuum unit (10) in the current system) is activated.

[0042] The automation (9) has as many outputs as the number of channels on the conveyor belt (11 ). In the case of more than one camera (7) in the system, thanks to the processor (8) being able to work in parallel, all channels of the conveyor belt (11 ) and the outputs included in the automation (9) are matched. Therefore, the channel through which the product flows is determined by the processor (8) and the signal containing this channel information is sent to the automation (9) at the output. The vacuuming unit (10) contains as many vacuum units as the number of channels on the conveyor belt (11 ). There are outputs on the automation (9) equal to the number of these vacuum units. All these outputs are matched with the vacuum units. Therefore, if the faulty / errorless information signal of which channel and the midpoint information of the object are transferred to the automation (9) via the processor (8), thanks to this information and signal coming to the automation (9), the vacuum of which channel should be opened after how many steps (information read from the encoder). It sends the signal to the output matching the relevant channel, thus the relevant vacuum in the relevant channel is activated for a period of time predetermined by the automation (9); in this way, the faulty product in the relevant channel is sucked by the vacuum unit.

[0043] Within the scope of the invention, instead of the equipment defined as the vacuuming unit (10), any industrial system vacuuming or collection can be integrated. The important part here that provides the technical effect is that the processor (8) can perform parallel processing as many as the number of channels and each thread works independently of each other and produces a signal for the conveyor belt (11 ) channel where the detection is made. This signal can be included in an output with the desired function.

[0044] When the system, whose operating logic is explained above on a single-channel conveyor belt (11 ), is operated on a multi-channel conveyor belt (11 ) and when the belt is divided into Kn equal parts, the image coming from the camera (7) is first divided into Kn equal parts by a software running on the embedded system (8). In other words, if the resolution of camera (7) is 1024*K, each line image coming from camera (7) is divided into pieces of length 1024*K / Kn. In practice, the K value is generally 1 ,2,4,8,16. For an exact division, it would be more accurate to choose the Kn values from the 2An (2,4,8,16,32,64) space. The process of dividing the line image coming from the camera (7) into Kn equal parts, as described above, can be performed with any hardware capable of parallel processing. As explained in Figure 10, after the line image coming from the camera (7) is divided into Kn equal parts, each part is processed with the single channel precision sorting algorithm explained in detail in Figure 7. In this way, all operations performed in a single channel can be performed simultaneously by Kn number of threads. This means that the system works in parallel in Kn numbers. In summary, the above-described method is operated by the processor (8) for each segmented image. Therefore, the same process is applied for each segmented image.

Claims

CLAIMS1 . A quality control and sorting system characterized by comprising a conveyor belt (11 ), at least one camera (7) located on the conveyor belt (11 ), an automation (9) and an electronic device further comprises a software that processes the images taken from the camera / cameras (7) with the dynamic buffering method within the processor (8) and a vacuuming unit (10) sorting and vacuuming the defective (4, 5) and defect-free products (6) coming from the hopper (1 ).

2. The quality control and sorting system according to Claim 1 characterized by comprising a single channel conveyor belt (11 ).

3. The quality control and sorting system according to Claim 1 characterized by comprising at least two channel conveyor belts (11 ).

4. The quality control and sorting system according to Claim 2 or 3 characterized by comprising a first groove (2) and a second groove (3) and the conveyor belt (11 ) fed from at least two grooves (1 , 2) whose speed is adjusted so that the objects' progress speed is V01in the first groove (2), and the objects' progress speed is Vo2in the second groove (3), and Vo2> V01at all times, and conveyor belt speed is Vkand whose Vkspeed is adjusted as Vk> Vo2at all times.

5. The quality control and sorting system according to any of the above claims characterized by comprising the processor (8) further comprises the software containing a convolutional neural network, where the image written to the dynamic buffer is sent to be classified as the input image.

6. The quality control and sorting system according to claim 3 characterized by comprising the processor (8) further comprises the software that performs the segmentation of the image coming from the camera (7) into the number of channels.

7. The quality control and sorting system according to any of the above claims characterized comprising the processor (8) that can convert the RGB format line image coming from the camera (7) into a "monochrome (gray format)" image and apply a threshold value, and detect the pixels belonging to the object as a result of the conversion, and is configured to send these line images to the dynamic buffer as long as the pixels are detected belongs to object.

8. The quality control and sorting system according to any of the above claims characterized by comprising the processor (8) further comprising the software containing convolutional neural network structures trained with defective (4, 5) or defect-free (6) products to be used as a reference during sorting.

9. The quality control and sorting system according to Claim 8 characterized by comprising the processor (8) further comprising the software that can detect defective products (4, 5) and defect-free products (6) as a result of classifying the images coming from the camera (7) with previously defined shapes with a convolutional neural network structure that has been previously trained with defective products (4, 5) and defect-free products (6).

10. The quality control and sorting system according to Claim 3 characterized by comprising the automation system (9) with outputs equal to the number of channels on the conveyor belt (11 ) and a vacuuming unit (10) containing as many vacuum units as the number of channels on the conveyor belt (11 ).

11. The quality control and sorting system according to Claim 10 characterized by comprising the processor (8) further comprises the software that matches each conveyor belt (10) channel with each output of the automation (9) in order to send a signal to the automation (9) output in the channel through which the product flows.

12. The quality control and sorting system according to Claim 10 or 11 characterized comprising the processor (8) further comprises the software that adds half the buffering time to the fixed distance between the camera (7) and the vacuum unit (10) at exactly half of the buffering time of the object, and at the end of this period, sends a signal to the automation (9) output in the channel through which the product flows13. The quality control and sorting system according to Claim 10 or 11 characterized by comprising an encoder further comprises the position information of the conveyor belt (11 ).

14. The quality control and sorting system according to Claim 13 characterized by comprising the automation (9) calculating the number of steps of the encoder based on the signal and size information coming from the processor (8) in order to calculate the number of steps from the moment the product is under the camera (7) to the vacuum unit (10).

15. The quality control and sorting system according to claim 1 characterized by comprising the processor (8) further comprises the software that can generate signals for an external collection or sorting system when it is integrated into the system.

16. A method for running a quality control and sorting system characterized by comprising the steps below;- Converting RGB images taken by a camera (7) to a “monochrome (gray format)” image by a software within a processor (8),- Converting the entire image to a binary image with a threshold value applied to the monochrome image,- Interpreting as there are pixels belonging to the object on this line image, if any of the pixels belonging to the object are white as a result of the conversion,- Transferring the received image to the dynamic buffer if a pixel belonging to the object is detected,- Stopping the transfer of the image to the dynamic buffer as long as the pixel belonging to the object is not detected,- Classification of the image transferred to the dynamic buffer on the convolutional neural network previously defined in the system,- Adding the fixed distance of the vacuum unit in case the image is classified as a faulty product on the convolutional neural network, from the moment the exact midpoint of the imaged object is under the camera (7) until the remaining half of the object passes under the camera, when the object reaches the full vacuum unit (10),- Sending a signal to the automation (9) connected to the vacuum unit for the vacuum unit (10) to operate.

Images