Inspection system and inspection method for inspecting low density foreign substance
The foreign substance inspection system uses a MWIR camera and deep learning model to detect low-density foreign substances through packaging, addressing the limitations of existing systems by accurately identifying and segregating defective products from normal ones.
Patent Information
- Application Number
- PCT/KR2024/006249
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-11-09
- Filing Date
- 2024-05-09
- Publication Date
- 2024-11-14
AI Technical Summary
Existing foreign substance inspection systems struggle to accurately detect low-density foreign substances within products, particularly those made of materials like vinyl and plastic, and often fail to prevent defective products from being mixed with normal ones, as they cannot penetrate packaging materials and lack effective differentiation.
A system combining a conveyor belt, product detection sensor, PLC, camera device with a 3-5 μm MWIR camera, and deep learning model that processes images to distinguish between normal and defective products, allowing for the detection of low-density foreign substances through packaging materials and preventing defective products from being mixed with normal ones.
The system effectively detects low-density foreign substances by penetrating packaging materials and prevents defective products from being mixed with normal ones, enhancing the accuracy of foreign substance identification and ensuring proper product sorting.
Smart Images

Figure KR2024006249_14112024_PF_FP_ABST
Abstract
Description
Inspection system and method for inspecting low-density foreign substances
[0001] The present disclosure relates to an inspection system and an inspection method for inspecting low-density foreign substances, and more particularly, to an inspection system and an inspection method capable of distinguishing between normal products and foreign substances with high accuracy by combining thermal imaging and machine vision.
[0002]
[0003] The material described in this section merely provides background information for the present disclosure and does not constitute prior art.
[0004] Recently, with the increase in single-person households and the growing preference for convenience foods, the production of easy-to-cook foods is increasing.
[0005] Despite thorough hygiene management, various foreign substances can be introduced into ready-to-eat foods during the manufacturing process. Consumers are more averse to the introduction of foreign substances into food than other product categories, necessitating thorough inspection of food products before shipment. Various technologies have been proposed to detect the presence of foreign substances.
[0006] For example, according to KR 10-2507591 B1, products can be inspected for foreign matter using X-ray imaging. When using X-ray imaging, the product being inspected and the foreign matter are displayed differently depending on their density.
[0007] However, X-ray images present a challenge in identifying low-density foreign substances within the product being inspected. These low-density foreign substances include vinyl, plastic, and other materials.
[0008] Meanwhile, cameras that utilize the wavelength band of SWIR (Short Wavelength InfraRed) or LWIR (Long Wavelength InfraRed) are also used to detect low-density foreign substances, but there is a problem that waves in these wavelength bands cannot penetrate the packaging material.
[0009] In addition, conventional foreign substance inspection systems have the problem that even if they detect defective products containing foreign substances, they are difficult to detect if the defective products are mixed in with normal products.
[0010]
[0011] Accordingly, the present disclosure aims to provide a foreign substance inspection system and inspection method capable of identifying low-density foreign substances and photographing them through packaging materials.
[0012] In addition, the present disclosure aims to provide a foreign substance inspection system and inspection method that can prevent the problem of defective products being mixed among normal products.
[0013] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.
[0014]
[0015] According to one embodiment of the present disclosure, a foreign matter inspection system is provided, including: a conveyor belt configured to transport a product to be inspected; a product detection sensor configured to detect the introduction of the product to be inspected onto the conveyor belt; a PLC configured to generate a camera device trigger signal and transmit the signal to the camera device when the introduction of the product to be inspected is detected by the product detection sensor; the camera device configured to photograph the product to be inspected and process the photographed image; and a PC configured to receive a deep learning model learned by a server and input the image processed by the camera device into the deep learning model, thereby outputting a prediction of whether or not a foreign matter is included; wherein the camera device comprises an MWIR camera that photographs the product to be inspected using a wavelength of 3 ㎛ to 5 ㎛; and an image processing unit configured to process a raw image photographed by the MWIR camera.
[0016] In addition, preferably, the server of the foreign substance inspection system according to one embodiment of the present disclosure includes: a thermal image input module configured to receive a raw image captured by a MWIR camera; a foreign substance inclusion input module configured to receive information on whether a product to be inspected contains a foreign substance; an image processing unit configured to process the input raw image to generate a processed image; a learning information storage unit configured to generate and store a learning information dataset using the processed image and the foreign substance inclusion; and a learning unit configured to learn a deep learning model using the processed image as input data and the foreign substance inclusion as output data.
[0017] In addition, preferably, the image processing unit of the foreign substance inspection system according to one embodiment of the present disclosure generates a processed image by binarizing the raw image and converting the binarized image to a preset size, and when the PC receives the image processed by the image processing unit, it outputs a predicted foreign substance inclusion of 0 or 1.
[0018] In addition, preferably, in the foreign substance inspection system according to one embodiment of the present disclosure, if the predicted foreign substance inclusion is 0, the PC displays the raw image or the processed image through the output unit, and stores the raw image or the processed image of the product to be inspected, the metadata of the raw image, and the value 0 as a dataset in a database.
[0019] In addition, preferably, in the foreign substance inspection system according to one embodiment of the present disclosure, if the predicted foreign substance inclusion is 1, the PC transmits 1 to the PLC, the PLC waits for a predetermined time, generates a lighting signal and transmits the lighting signal to a warning light, the PLC sets a counter 1 in a memory, generates a rejector operation signal and transmits the rejector operation signal to a rejector, and when the rejector receives the rejector operation signal, moves the inspection target product corresponding to the predicted foreign substance inclusion to a defective product loading space.
[0020] In addition, preferably, the defective product loading space of the foreign substance inspection system according to one embodiment of the present disclosure is provided with a defective product sensor configured to detect the introduction of the moved inspection target product, and when the PLC receives a detection signal of the defective product sensor within a preset time from the time when counter 1 is set in the memory, the PLC clears the memory, and when the PLC does not receive a detection signal of the defective product sensor within a preset time from the time when the memory counter 1 is set, the PLC generates a conveyor belt stop signal and transmits the conveyor belt stop signal to the conveyor belt.
[0021] In addition, according to one embodiment of the present disclosure, a method for inspecting foreign substances is provided, including: (a1) a step in which a deep learning model is trained by a server, the step of using a processed image of a product to be inspected as input data and the presence or absence of a foreign substance as output data; (a2) a step in which the deep learning model is transmitted to a PC; (b1) a step in which the introduction of a product to be inspected is detected by a product detection sensor, and the product detection sensor generates a product introduction signal and transmits the signal to a PLC; (b2) a step in which the PLC receives the product introduction signal, waits for a predetermined time, and generates a trigger signal and transmits the signal to a camera device; (b3) a step in which the camera device captures a raw image of the product to be inspected using an MWIR camera, and the camera device processes the image using an image processing unit; (b4) a step in which the processed image is transmitted to the PC; and (c) a step in which, when the processed image is input to the deep learning model, a predicted presence or absence of a foreign substance is output; wherein the MWIR camera captures the product to be inspected using a wavelength of 3 μm to 5 μm.
[0022] In addition, preferably, the step (a1) of the inspection method according to one embodiment of the present disclosure includes: (a10) a step in which the thermal image input module of the server receives a raw image captured using an MWIR camera; (a11) a step in which the foreign substance inclusion input module of the server receives a foreign substance inclusion corresponding to the raw image captured in the step (a10); (a12) a step in which the image processing unit of the server processes the raw image input in the step (a10) to generate a processed image; and (a13) a step in which the learning unit of the server trains a deep learning model using the processed image generated in the step (a12) as input data and the foreign substance inclusion input in the step (a11) as output data.
[0023] In addition, preferably, the step (c) of the inspection method according to one embodiment of the present disclosure includes: (c10) a step of outputting the predicted presence or absence of foreign matter as 0; (c11) a step of the PC displaying the raw image or the processed image through an output unit; and (c12) a step of the PC transmitting and storing the raw image or the processed image of the product to be inspected, metadata of the raw image, and a value of 0 as a dataset in a database.
[0024] In addition, preferably, the step (c) of the inspection method according to one embodiment of the present disclosure further includes: (c20) a step of outputting whether the predicted foreign matter is included as 1; (c21) a step of the PC transmitting 1 to the PLC; (c22) a step of the PLC waiting for a predetermined time and then generating a lighting signal and transmitting the lighting signal to a warning light; (c23) a step of the PLC setting a counter 1 in a memory and generating a rejector operation signal to generate the rejector operation signal; and (c24) a step of the rejector moving the inspection target product corresponding to the predicted foreign matter inclusion to a rejector loading space when the rejector receives the rejector operation signal.
[0025] In addition, preferably, the defective product loading space of the inspection method according to one embodiment of the present disclosure is provided with a defective product sensor configured to detect the introduction of the moved inspection target product, and the step (c) further includes: (c25) when the PLC receives a detection signal of the defective product sensor within a preset time from a point at which counter 1 is set in the memory, the PLC clears the memory; and (c26) when the PLC does not receive a detection signal of the defective product sensor within a preset time from a point at which the memory counter 1 is set, the PLC generates a conveyor belt stop signal and transmits the conveyor belt stop signal to the conveyor belt.
[0026]
[0027] As described above, the foreign substance inspection system and inspection method according to one embodiment of the present disclosure photograph the product to be inspected by irradiating a wavelength of an energy band capable of detecting low-density foreign substances while being able to penetrate packaging materials, thereby enabling detection of foreign substances that have entered the product to be inspected after packaging has been completed, thereby increasing the probability of detecting foreign substances.
[0028] In addition, the foreign substance inspection system according to one embodiment of the present disclosure has the effect of properly notifying the user of the need to remove defective products by stopping all operations if defective products are not properly removed.
[0029]
[0030] FIG. 1 is a block diagram of a foreign substance inspection system according to one embodiment of the present disclosure.
[0031] FIG. 2 is a block diagram of a PC according to one embodiment of the present disclosure.
[0032] Figure 3 is a flowchart of a foreign substance inspection method according to one embodiment of the present disclosure.
[0033] Figure 4 is a flowchart of a foreign substance inspection method according to one embodiment of the present disclosure.
[0034] FIG. 5 is a block diagram of a server according to one embodiment of the present disclosure.
[0035] Figure 6 is a flowchart of a learning method of a deep learning model according to one embodiment of the present disclosure.
[0036] FIG. 7 compares a foreign substance inspection system according to one embodiment of the present disclosure with a conventional inspection system.
[0037]
[0038] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given identical reference numerals, even if they appear in different drawings. Furthermore, in describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.
[0039] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.
[0040] In this disclosure, the term "product to be inspected" refers to a product inspected by a foreign matter inspection system, and may preferably refer to packaging materials and food within the packaging materials. In this disclosure, the product to be inspected is described as packaging materials and food within the packaging materials, but is not necessarily limited thereto.
[0041]
[0042] 1. Description of the foreign substance inspection system
[0043] FIG. 1 is a block diagram of a foreign substance inspection system according to one embodiment of the present disclosure. FIG. 2 is a block diagram of a PC according to one embodiment of the present disclosure.
[0044] Referring to FIGS. 1 and 2, a foreign substance inspection system (1) according to one embodiment of the present disclosure will be described. The foreign substance inspection system (1) is configured to detect foreign substances introduced into a product to be inspected and to distinguish between normal and defective products. To this end, the foreign substance inspection system (1) includes all or part of a PLC (10), a defective product sensor (11), a rejector (12), a conveyor belt (13), a warning light (14), a camera device (15), a product detection sensor (16), a PC (17), a server (18), and a database (19).
[0045] A PLC (programmable logic controller, 10) is configured to control the system (1) as a whole by communicating with other components included in the foreign substance inspection system (1) via wired or wireless communication.
[0046] The defective product sensor (11) is configured to sense at least a portion of the inspection target product placed on the conveyor belt (13), specifically, the inspection target product determined to be defective.
[0047] A conveyor belt (13) is configured to transport products to be inspected. Products to be inspected are fed into one end of the conveyor belt (13), and products to be inspected are discharged from the other end. At this time, a driving unit is provided on the conveyor belt (13), so that the conveyor belt (13) can transport products to be inspected in one direction.
[0048] The inspection target product transported on the conveyor belt (13) is determined to be a normal product or a defective product. At this time, the inspection target product determined to be a defective product can be removed from the conveyor belt (13) by a rejector (12). For this purpose, the rejector (12) may be, for example, a take-out robot (not shown), but is not necessarily limited thereto.
[0049] The warning light (14) lights up for a preset period of time when a defective product removed by the rejector (12) is sensed by the defective product sensor (11).
[0050] The camera device (15) is configured to photograph the product to be inspected being transported by the conveyor belt (13) and to process the photographed image. To this end, the camera device (15) includes an MWIR camera (150) and an image processing unit (152).
[0051] The MWIR camera (150) is configured to photograph the product to be inspected. The MWIR camera (150) photographs the product to be inspected using mid-wave infrared (MWIR). Here, MWIR refers to a wavelength of 3 to 5 ㎛. While conventional SWIR band cameras generally cannot penetrate packaging materials formed of materials including NY, PE, and LDPE, the MWIR camera (150) using the wavelength of the MWIR band can penetrate packaging materials formed of materials including NY, PE, and LDPE, thereby enabling detection of foreign substances introduced into the packaging material after packaging is completed. As a result, foreign substances introduced during packaging can be detected, thereby increasing the probability of detecting foreign substances. In other words, the MWIR camera (150) can detect radiant heat of the product to be inspected.
[0052] The image processing unit (152) is configured to process a raw image captured by the MWIR camera (150).
[0053] Specifically, the image processing unit (152) generates a black-and-white image by binarizing a raw image and converting it to gray scale. Thereafter, the image processing unit (152) generates a processed image by converting the generated black-and-white image into an image of a preset size. Here, the preset size may be, for example, 223 pixels.
[0054] The image processed by the image processing unit (152) is transmitted to the PC (17).
[0055] The product detection sensor (16) is positioned at one end of the conveyor belt (13), i.e., at the location where the product to be inspected is introduced, and is configured to detect the introduction of the product. When the product detection sensor (16) detects a product, it can generate a detection signal and transmit it to the PLC (10).
[0056] The PC (17) receives a deep learning model learned by the server (18) and is configured to input a processed image transmitted by the camera device (15) into the deep learning model. The PC (17) is configured to output whether or not a foreign substance is included corresponding to the input processed image.
[0057] To this end, the PC (17) includes all or part of an input unit (170), an output unit (171), a control unit (172), a processing unit (173), and a communication unit (174) (see FIG. 2). Meanwhile, the components of the PC (17) described in the present disclosure only describe the configurations necessary for inspection according to the present disclosure, and descriptions of additional components typically included in the PC (17) are omitted.
[0058] The input unit (170) is configured to receive commands or data from the outside.
[0059] Commands or data input through the input unit (170) can be transmitted to other components of the PC (17). In the present disclosure, an image processed by the image processing unit (152) can be input through the input unit (170).
[0060] The output unit (171) is configured to output commands or data processed or generated by the PC (17). In the present disclosure, the output unit (171) may include a display and a speaker.
[0061] The control unit (172) is configured to control one or more other components of the PC (17). Here, the one or more other components are a concept that includes both hardware and software. That is, the control unit (172) can control the input unit (170), output unit (171), processing unit (173), and communication unit (174) of the PC (17).
[0062] The processing unit (173) can perform a data processing or calculation function, and as part of the data processing or calculation function, is configured to store commands or data received from other components in volatile memory, process the commands or data stored in the volatile memory, and store the results in non-volatile memory.
[0063] The processing unit (173) can input the processed image input through the input unit (170) into a pre-downloaded deep learning model to calculate whether or not a foreign substance is present. Meanwhile, a detailed description of the deep learning model is provided in FIGS. 5 and 6.
[0064] The communication unit (174) is configured for communication with external devices, and through the communication unit (174), the PC (17) can send and receive commands and data with the PLC (10), camera device (15), server (18), and database (19).
[0065] PC (17) classifies the result value as "1" when the product to be inspected contains foreign substances, i.e., when it is judged to be a defective product. In addition, PC (17) classifies the result value as "0" when the product to be inspected does not contain foreign substances, i.e., when it is a normal product.
[0066] If it is judged as 1, the PC (17) transmits a predetermined signal to the PLC (10). At this time, the PC (17) and the PLC (10) can transmit and receive signals using the SSR or Ethernet / IP Interface.
[0067] If judged as 0, the PC (17) can provide the user with an image captured or processed by the camera device (15) and an OK display through the output unit (171). In addition, the PC (17) can store the image of the inspection target product judged as 0, the metadata of the image, and the 0 value in the database (19).
[0068] The server (18) is configured to process, store, and transmit / receive data. The server (18) is configured to train a deep learning model. This will be described in detail in FIGS. 5 and 6 . In the present disclosure, the server (18) may be a separately provided device, such as an IDC, but is not necessarily limited thereto, and may be a cloud.
[0069] The database (19) is configured to temporarily or permanently store data received from the PC (17). The database (19) may be a separate storage device provided outside the PC (17), but is not necessarily limited thereto, and may be a storage device provided in the PC (17) or PLC (10).
[0070]
[0071] 2. Description of foreign substance inspection method
[0072] 2.1. Description of the method for determining whether foreign substances are present
[0073] Figure 3 is a flowchart of a foreign substance inspection method according to one embodiment of the present disclosure.
[0074] Referring to Figure 3, the process from the step of introducing the product to be inspected into the foreign substance inspection system (1) to the step of determining whether or not it contains foreign substances is described.
[0075] When a product to be inspected is introduced into the foreign substance inspection system (1), the introduction of the product to be inspected is detected by the product detection sensor (16). The product detection sensor (16) generates a product introduction signal and transmits it to the PLC (10).
[0076] When the PLC (10) receives a product input signal, it waits for a first delay time, then generates a trigger signal for shooting and transmits it to the camera device (15). Here, the first delay time is preferably the same time as the time it takes for the input inspection target product to be transported to the location where the camera device (15) is placed.
[0077] When the camera device (15) receives a trigger signal, it can detect the radiant heat of the product to be inspected by taking a picture of the product to be inspected using the MWIR camera (150). The raw image taken by the MWIR camera (150) is processed by the image processing unit (152) of the camera device (15).
[0078] The image processing unit (152) binarizes the raw image and converts it to gray scale to create a black and white image. Thereafter, the image processing unit (152) converts the generated black and white image into an image of a preset size, thereby creating a processed image. Here, the preset size may be, for example, 223 pixels.
[0079] The image processed through the above process is transmitted to the PC (17).
[0080] At this time, the PC (17) can download the deep learning model from the server (18) in advance before receiving the processed image and store it in internal memory (not shown).
[0081] PC (17) can input the processed image into the aforementioned deep learning model and output whether or not a foreign substance is included. At this time, PC (17) can output a result value of 1 if a foreign substance is included and output a result value of 0 if a foreign substance is not included.
[0082]
[0083] 2.2. Description of the method after determining whether foreign substances are present
[0084] Figure 4 is a flowchart of a foreign substance inspection method according to one embodiment of the present disclosure.
[0085] Referring to Fig. 4, the steps after the presence or absence of foreign substances is determined by the foreign substance inspection system (1) are described.
[0086] If it is determined by the PC (17) that there is no foreign substance, i.e., if the result value is 0, the PC (17) outputs a thermal image and an OK mark together to the output unit (171). Here, the thermal image may be an image captured by the camera device (15) or a processed image.
[0087] Additionally, the PC (17) can store the image of the inspection target product judged as 0, the image's metadata, and the 0 value in the database (19). At this time, the data stored in the database (19) can be used for learning or updating a deep learning model in the future.
[0088] Meanwhile, if the PC (17) determines that a foreign substance is contained, i.e., if the result value is 1, the PC (17) transmits the result value to the PLC (10). At this time, the PC (17) and the PLC (10) can transmit and receive signals or data using the SSR or Ethernet / IP Interface.
[0089] When the PLC (10) receives the result value 1 from the PC (17), it waits for a second delay time. Here, the second delay time may be the same as the time it takes for the product to be inspected, which is judged to be defective, to be transported from the location where the camera device (15) is located to the location where the rejector (12) is located.
[0090] The PLC (10) generates a lighting signal and transmits it to the warning light (14). When the warning light (14) receives the lighting signal, it lights up for a preset period of time. This allows the user to know that there are defective products among the products being inspected.
[0091] Simultaneously with the generation of the lighting signal, or sequentially, the PLC (10) sets counter 1 in the memory (not shown), generates a rejector operation signal, and transmits it to the rejector (12).
[0092] The rejector (12) operates when it receives a rejector operation signal. The rejector (12) can move a product to be inspected that has been determined to be defective from the conveyor belt (13) to a space provided separately. For this purpose, the rejector (12) may be a removal robot, but is not necessarily limited thereto.
[0093] A defective product sensor (11) is provided in a separately provided space. At this time, when a product to be inspected is detected by the defective product sensor (11), the defective product sensor (11) generates a product detection signal and transmits it to the PLC (10).
[0094] PLC (10) checks whether a product detection signal is received within a preset time from the time counter 1 is set in the memory.
[0095] At this time, if a product detection signal is received within a preset time, the PLC (10) clears the memory. That is, if a defective product is detected and the defective product is detected by the defective product sensor (11) within a predetermined time, the PLC (10) determines that the defective product has been removed from the conveyor belt (13), and can continue performing the existing work.
[0096] Meanwhile, if a product detection signal is not received within a preset time, the PLC (10) generates a conveyor belt stop signal and transmits it to the conveyor belt (13).
[0097] When the conveyor belt (13) receives a conveyor belt stop signal, it stops operating. That is, when a defective product is detected and the defective product is not detected by the defective product sensor (11) within a certain period of time, the PLC (10) determines that the defective product has not been properly removed from the conveyor belt (13), and stops all operations, thereby notifying the user of the need to remove the defective product.
[0098]
[0099] 3. Description of the deep learning model's training method
[0100] FIG. 5 is a block diagram of a server according to one embodiment of the present disclosure.
[0101] Referring to FIG. 5, the server (18) includes all or part of an input unit (180), an image processing unit (183), a learning information storage unit (184), and a learning unit (185).
[0102] The input unit (180) is configured to receive data for learning. The input unit (180) includes a thermal image input module (181) and a foreign substance inclusion input module (182).
[0103] The thermal image input module (181) is configured to receive a raw image as a thermal image captured by the MWIR camera (150).
[0104] The input module (182) for foreign substance inclusion is configured to receive input on whether a foreign substance is included inside the product to be inspected, which is photographed by the MWIR camera (150).
[0105] The image processing unit (183) is configured to preprocess an image input by the thermal image input module (181). Specifically, the image processing unit (183) binarizes a raw image and converts it to gray scale, thereby generating a black and white image.
[0106] Thereafter, the image processing unit (183) generates a processed image by converting the generated black and white image into an image of a preset size. Here, the preset size may be, for example, 223 pixels.
[0107] Through the image processing process described above, information used for learning can be transformed into a shape and size appropriate for learning.
[0108] The learning information storage unit (184) stores the processed image and whether or not the processed image contains foreign substances as a dataset. At this time, when the learning information storage unit (184) creates a dataset, the RAY or NUMPY library can be used to create a dataset by associating the processed image and whether or not the image contains foreign substances with X and Y appropriate for TENSORFLOW KERAS learning.
[0109] The learning unit (185) trains a deep learning model that uses the processed image as input data and the presence or absence of foreign substances as output data.
[0110] At this time, the learning unit (185) classifies the learning information dataset into training, validation, and test datasets. Preferably, the ratio of training: validation: test is randomly classified to be 0.65: 0.2: 0.15. RandomSplitter can be used for random classification.
[0111] At this time, train_test_split can be used when the training dataset and test dataset are adjusted, and validation_pct can be used when the validation dataset is adjusted.
[0112] The learning unit (185) can train a deep learning model by compiling layers of a deep learning model after augmenting a training dataset and a validation dataset. Preferably, the learning unit (185) can use a data augmentation technique during data augmentation to expand the diversity of the training data and prevent overfitting. Data augmentation can include image enlargement, cropping, rotation, random flipping, random resizing, and brightness adjustment.
[0113] The learning unit (185) can transfer learn a deep learning model with the network structure of Resnet and Mobilenet using the Pytorch-based Fastai library. This allows the deep learning model to be configured and compiled into advanced layers.
[0114] The learning unit (185) can evaluate the model using a test data set, and save the deep learning model in pkl format and the evaluation values (accuracy, precision, recall, f-score, support) in csv format.
[0115] The deep learning model learned by the learning unit (185) can be transmitted to the PC (17) when there is a request from the PC (17).
[0116]
[0117] Figure 6 is a flowchart of a learning method of a deep learning model according to one embodiment of the present disclosure.
[0118] Referring to Fig. 6, when an image captured by an MWIR camera (150) is input by an input unit (180), it is processed by an image processing unit (183) provided in a server (18) (S600).
[0119] The processed image and the corresponding presence or absence of foreign substances are created and stored as a learning information dataset (S610).
[0120] Using the stored learning information dataset, the learning unit (185) trains a deep learning model (S620). At this time, the deep learning model can use processed images as input data and the presence or absence of foreign substances as output data. The deep learning model is trained on a server (18) and downloaded to a PC (17).
[0121] When an image captured and processed by a camera device (15) is transmitted to a PC (17), the PC (17) inputs the received processed image into a stored deep learning model, thereby outputting whether a predicted foreign substance is included (S630).
[0122] The PC (17) can provide the user with the predicted presence of foreign substances through the output unit (171), and the foreign substance inspection system (1) can control all or part of the components of the foreign substance inspection system (1) using the predicted presence of foreign substances.
[0123]
[0124] Figure 7 compares a foreign substance inspection system according to one embodiment of the present disclosure with a conventional inspection system. Meanwhile, Figure 7 photographs an inspection target product packaged in a packaging material comprising NY, PE, and LDPE.
[0125] FIG. 7(a) shows a comparison of the results of photographing the same inspection target product using a camera device (15) according to one embodiment of the present disclosure, i.e., an MWIR camera (150), and the results of photographing the same product using a conventional camera device, specifically, a SWIR and X-ray camera.
[0126] Referring to Fig. 7(a), it can be confirmed that the MWIR camera (150) clearly distinguishes between food and low-density foreign substances within the product to be inspected. On the other hand, the results of the photographing using the SWIR camera show that it barely penetrates the packaging material. Furthermore, it can be seen that in some areas where it penetrates, no image that could be considered a low-density foreign substance is distinguished. In addition, the results of the photographing using the X-ray camera show that no part that could be judged as a low-density foreign substance is visible at all.
[0127] Through these results, it can be seen that the foreign substance inspection system (1) according to one embodiment of the present disclosure has a superior ability to detect low-density foreign substances compared to conventional methods.
[0128]
[0129] FIG. 7(b) shows a comparison of the results of photographing the same inspection target product using a camera device (15) according to one embodiment of the present disclosure, i.e., an MWIR camera (150), and the results of photographing the same product using a conventional camera device, specifically, an LWIR and X-ray camera.
[0130] Referring to Fig. 7(b), it can be seen that the MWIR camera (150) can clearly and precisely capture images of internal materials by penetrating the packaging material. On the other hand, the results of the images taken using the LWIR camera show that it can barely penetrate the packaging material. In addition, the results of the images taken using the X-ray camera show that although the packaging material penetration ability is excellent, it is difficult to distinguish between low-density foreign substances and food.
[0131] Through these results, it can be seen that the foreign substance inspection system (1) according to one embodiment of the present disclosure has excellent packaging material penetration ability and at the same time, excellent low-density foreign substance detection ability.
[0132]
[0133] The above description is merely an example of the technical idea of the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.
[0134]
[0135] [Explanation of symbols]
[0136] 1: Foreign substance inspection system
[0137] 10: PLC
[0138] 11: Defective sensor
[0139] 12: Rejector
[0140] 13: Conveyor belt
[0141] 14: Warning lights
[0142] 15: Camera device
[0143] 16: Product detection sensor
[0144] 17: PC
[0145] 18: Server
[0146] 19: Database
Claims
1. A conveyor belt configured to transport the product to be inspected; A product detection sensor that detects the introduction of a product to be inspected onto the conveyor belt; A PLC that generates a camera device trigger signal and transmits it to the camera device when the product to be inspected is detected by the above product detection sensor; The camera device configured to photograph the product to be inspected and process the photographed image; and A PC configured to receive a deep learning model learned by a server and output whether a predicted foreign substance is included by inputting an image processed by the camera device into the deep learning model; The above camera device, A MWIR camera that photographs the product to be inspected using a wavelength of 3 ㎛ to 5 ㎛; and An image processing unit configured to process a raw image captured by the MWIR camera; Foreign substance inspection system.
2. In paragraph 1, The above server, A thermal image input module configured to receive raw images captured by a MWIR camera; A foreign substance inclusion input module configured to receive input on whether a product to be inspected contains foreign substances; An image processing unit configured to process the input raw image and generate a processed image; A learning information storage unit that creates and stores a learning information dataset using the processed image and whether or not the foreign matter is included; and A learning unit configured to learn a deep learning model using the processed image as input data and whether or not the foreign substance is included as output data; Foreign substance inspection system.
3. In paragraph 2, The image processing unit generates a processed image by binarizing the raw image and converting the binarized image to a preset size, When the PC receives an image processed by the image processing unit, it outputs whether the image contains a predicted foreign substance of 0 or 1. Foreign substance inspection system.
4. In paragraph 3, If the predicted foreign substance inclusion is 0, The above PC displays the raw image or the processed image through the output unit, The raw image or processed image of the product to be inspected, the metadata of the raw image, and the 0 value are stored in a database as a dataset. Foreign substance inspection system.
5. In paragraph 3, If the above predicted foreign substance inclusion is 1, The above PC sends 1 to the above PLC, The above PLC waits for a predetermined amount of time, then generates a lighting signal and transmits the lighting signal to the warning light. The above PLC sets counter 1 in the memory, generates a rejector operation signal, and transmits the rejector operation signal to the rejector. When the above rejector receives the rejector operation signal, the product to be inspected corresponding to the presence or absence of the predicted foreign substance is moved to the defective product loading space. Foreign substance inspection system.
6. In paragraph 5, The above defective product loading space is equipped with a defective product sensor configured to detect the introduction of the moved inspection target product, When the PLC receives a detection signal from the defective product sensor within a preset time from the time when counter 1 is set in the memory, the PLC clears the memory, If the PLC does not receive a detection signal from the defective product sensor within a preset time from the time at which the memory counter 1 is set, the PLC generates a conveyor belt stop signal and transmits the conveyor belt stop signal to the conveyor belt. Foreign substance inspection system. 7.(a1) A step in which a deep learning model is trained by a server using the processed image of the product to be inspected as input data and the presence or absence of foreign substances as output data; (a2) A step in which the above deep learning model is transmitted to a PC; (b1) A step in which the introduction of a product to be inspected is detected by a product detection sensor, and the product detection sensor generates a product introduction signal and transmits it to a PLC; (b2) a step in which the PLC receives the product input signal, waits for a predetermined time, generates a trigger signal, and transmits the trigger signal to the camera device; (b3) A step in which the camera device captures a raw image of the product to be inspected using an MWIR camera, and the camera device processes the image using an image processing unit; (b4) a step of transmitting the processed image to the PC; and (c) a step of outputting whether the processed image contains a predicted foreign substance when the processed image is input to the deep learning model; including, The above MWIR camera takes pictures of the product to be inspected using a wavelength of 3 ㎛ to 5 ㎛. Foreign substance inspection method.
8. In paragraph 7, The above step (a1) is: (a10) A step in which the thermal image input module of the above server receives a raw image captured using an MWIR camera; (a11) A step in which the input module for determining whether the server contains foreign substances receives input of whether the raw image captured in the step (a10) contains foreign substances; (a12) a step of generating a processed image by processing the raw image input in step (a10) by the image processing unit of the server; and (a13) A step of training a deep learning model using the processed image generated in step (a12) as input data and the presence or absence of foreign substances input in step (a11) as output data, including the learning unit of the server; Foreign substance inspection method.
9. In paragraph 8, Step (c) above, (c10) A step in which the above predicted foreign substance inclusion status is output as 0; (c11) a step of the PC displaying the raw image or the processed image through the output unit; and (c12) a step of transmitting and storing the raw image or processed image of the product to be inspected, the metadata of the raw image, and the 0 value as a dataset in a database; including; Foreign substance inspection method.
10. In paragraph 8, Step (c) above, (c20) A step in which the presence or absence of the predicted foreign substance is output as 1; (c21) A step in which the PC transmits 1 to the PLC; (c22) The PLC waits for a predetermined amount of time, then generates a lighting signal and transmits the lighting signal to the warning light; (c23) The PLC sets counter 1 in the memory and generates a rejector operation signal to generate the rejector operation signal; and (c24) further comprising a step of moving the inspection target product corresponding to whether the predicted foreign substance is included in the defective product loading space when the rejector receives the rejector operation signal; Foreign substance inspection method.
11. In paragraph 10, The above defective product loading space is equipped with a defective product sensor configured to detect the introduction of the moved inspection target product, Step (c) above, (c25) When the PLC receives a detection signal from the defective product sensor within a preset time from the ten points where the counter 1 is set in the memory, the PLC clears the memory; and (c26) If the PLC does not receive a detection signal from the defective product sensor within a preset time from the time when the counter 1 is set in the memory, the PLC generates a conveyor belt stop signal and transmits the conveyor belt stop signal to the conveyor belt; further comprising; Foreign substance inspection method.
Citation Information
Patent Citations
Method and system for determining package integrity
JP2020509364A
Inspection device and inspection method
JP2023051527A
Inspection apparatus using terahertz wave
KR102045079B1
A device for the fault detection and supplement of chopsticks in the automatic packaging machine
KR200160426Y1
KR20220164124A
