Discrimination device, discrimination method, and program

Abstract

This invention provides a discrimination device, discrimination method, and program that can improve the accuracy of distinguishing between different color types of bottles. [Solution] The discrimination device 10 comprises a removal unit 11B and a color type discrimination unit 11C. The removal unit 11B removes from a first image, which is an image of a bottle, a region that satisfies a predetermined condition for at least one of the brightness or saturation in the first image. The color type discrimination unit 11C determines the color type of the bottle based on a second image, which is the image after the region has been removed.

Description

Technical Field

[0001] The present disclosure relates to a discrimination device, a discrimination method, and a program.

Background Art

[0002] At waste treatment plants, a large amount of waste flows on a belt conveyor every day, and the waste is separated into recyclable waste and non-recyclable waste. At the site where the waste is processed, the waste separation work is carried out manually by people. Although the waste separation work is a simple task, since the burden on the workers who separate the waste is large, a system that automatically separates the waste (hereinafter sometimes referred to as a waste separation system) has been developed.

[0003] In the waste separation system, each waste flowing on the belt conveyor is recognized as a bottle to be sorted based on an image captured by an imaging device, and based on the recognition result, a desired sorting target is taken out from the group of bottles flowing on the belt conveyor using a robot hand or a suction pad.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Among the bottles to be sorted in the waste separation system, there may be bottles with labels (hereinafter sometimes referred to as labeled bottles).

[0006] Furthermore, since both the label and the bottle itself are often made of highly glossy materials, both the image with and without the label tends to appear shiny. As a result, the accuracy of identifying the color type of the bottle (hereinafter sometimes referred to as color type identification) tends to decrease when this shine occurs.

[0007] Therefore, the purpose of this disclosure is to provide a discrimination device, discrimination method, and program that can improve the accuracy of distinguishing between different color types of bottles. [Means for solving the problem]

[0008] A discrimination device according to one aspect of the present disclosure comprises a removal unit and a discrimination unit. The removal unit removes from a first image, which is an image of a bottle, a region that satisfies a predetermined condition for at least one of the brightness or saturation in the first image. The discrimination unit determines the type of color of the bottle based on a second image, which is an image after the region has been removed. [Effects of the Invention]

[0009] The disclosure discrimination device, discrimination method, and program can improve the accuracy of distinguishing between different bottle colors. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows an example of the configuration of the sorting system according to Example 1. [Figure 2] Figure 2 shows an example of the hardware configuration of the discrimination device according to Example 1. [Figure 3] Figure 3 shows an example of the configuration of the functional blocks of the sorting system according to Example 1. [Figure 4] Figure 4 is a flowchart showing an example of the operation flow of the discrimination device according to Example 1. [Figure 5] Figure 5 is a diagram illustrating the instance segmentation used in the discrimination device according to Example 1. [Figure 6]Figure 6 is a diagram illustrating the object detection used in the discrimination device according to Example 1. [Figure 7] Figure 7 shows an example of the operation of the image processing unit of the discrimination device according to Example 1. [Figure 8] Figure 8 shows an example of the operation of the image processing unit of the discrimination device according to Example 1. [Figure 9] Figure 9 is a flowchart showing an example of the operation flow of the discrimination device according to Example 2. [Figure 10] Figure 10 shows an example of the operation of the image processing unit of the discrimination device according to Example 2. [Modes for carrying out the invention]

[0011] Embodiments of the discrimination device, discrimination method, and program disclosed herein will be described in detail below with reference to the drawings. Furthermore, the art of this disclosure is not limited by the following description, and the components described herein include those easily conceivable to those skilled in the art, substantially identical components, and so-called equivalent components. Moreover, various omissions, substitutions, modifications, and combinations of components are possible without departing from the spirit of the following embodiments.

[0012] [Example 1] (Configuration of the sorting system) Figure 1 shows an example of the configuration of the sorting system according to Example 1. The configuration of the sorting system 1 according to this embodiment will be described with reference to Figure 1.

[0013] As shown in Figure 1, the sorting system 1 includes a discrimination device 10, a camera 20, a sorting device 30, and a belt conveyor 40. The discrimination device 10, the camera 20, and the sorting device 30 are capable of data communication with each other.

[0014] Hereinafter, as an example, the case where the sorting system 1 shown in FIG. 1 is installed in a waste treatment plant where a group of bottles flows on the belt conveyor 40 will be described. That is, hereinafter, the case where the object to be sorted by the sorting system 1 is a bottle will be described as an example. Here, the bottle is described as a concept including not only glass bottles but also plastic bottles, containers, PET bottles, etc.

[0015] In the discrimination of the color type of bottles in a conventional waste sorting system, when using a learned model for the image of a bottle (hereinafter sometimes referred to as a bottle image), the accuracy of the discrimination of the color type of labeled bottles tends to be low. Hereinafter, the part where a label is attached to a labeled bottle may be referred to as the "label part". Also, the image of the label part may be referred to as the "label part image". The shape of a normal bottle often has a narrow mouth part, a neck part connected to the mouth part, a body part wider than the neck part, and a shoulder part connecting the neck part and the body part. In a labeled bottle, the label part often exists on the body part. Also, a label is an attached member on which information about a product is written or a pattern is printed, and is formed of, for example, a film-based material or paper as a material. Also, the label may be directly attached to the surface of the bottle or wound (wrapped) around the bottle. In this embodiment, the configuration and operation of the sorting system 1 for improving the accuracy of bottle color type discrimination will be described in detail.

[0016] Also, hereinafter, as an example, the case where each bottle flowing on the belt conveyor 40 is sorted into three types of color type bottles: a transparent bottle (hereinafter sometimes referred to as a transparent bottle), a brown bottle (hereinafter sometimes referred to as a brown bottle), and a bottle having a color other than brown (hereinafter sometimes referred to as other color bottle) will be described.

[0017] The camera 20 is arranged above the belt conveyor 40 on which the bottle group is conveyed, has a predetermined imaging angle, and is an imaging device that images a predetermined area on the upper surface of the belt conveyor 40 from above the belt conveyor 40 at a constant frame rate. Therefore, the captured image captured by the camera 20 becomes an image of the bottle group (hereinafter sometimes referred to as a bottle group image). The bottle group image is transmitted from the camera 20 to the discrimination device 10.

[0018] The discrimination device 10 is a device that discriminates the color types of each bottle flowing on the belt conveyor 40 based on the bottle group image captured by the camera 20 for the bottle group flowing on the belt conveyor 40. Further, the discrimination device 10 controls the operation of the sorting device 30 according to the discrimination result of each bottle.

[0019] The sorting device 30 is a device that sorts the bottle group conveyed in the conveying direction CD by the belt conveyor 40 into transparent bottles, brown bottles, and other-colored bottles under the control of the discrimination device 10.

[0020] The belt conveyor 40 is a conveyor device that conveys the bottle group placed on the belt conveyor 40 in the conveying direction CD. That is, the belt conveyor 40 forms a conveyance path for conveying the bottle group in the conveying direction CD.

[0021] (Hardware configuration of the discrimination device) FIG. 2 is a diagram showing an example of the hardware configuration of the discrimination device according to the first embodiment. With reference to FIG. 2, the hardware configuration of the discrimination device 10 according to the present embodiment will be described.

[0022] As shown in FIG. 2, the discrimination device 10 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an auxiliary storage device 505, a network I / F 506, an external device I / F 507, a display 508, and an input device 509.

[0023] The CPU 501 is an arithmetic unit that controls the operation of the entire discrimination device 10. The ROM 502 is a non-volatile memory device that stores programs such as the IPL (Initial Program Loader) that are first executed by the CPU 501. The RAM 503 is a volatile memory device used as the work area of ​​the CPU 501.

[0024] The auxiliary storage device 505 is a non-volatile storage device that stores various data such as various setting information and programs. The auxiliary storage device 505 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0025] Network I / F506 is an interface circuit for data communication using a network. Network I / F506 is, for example, a NIC (Network Interface Card) that enables communication using the TCP (Transmission Control Protocol) / IP (Internet Protocol) protocol. Network I / F506 may also be a communication interface with wireless communication functionality based on standards such as Wi-Fi (registered trademark).

[0026] The external device I / F 507 is an interface circuit for data communication with external devices such as the camera 20 and the sorting device 30. The external device I / F 507 is an interface circuit for serial communication standards such as USB (Universal Serial Bus) communication, Bluetooth (registered trademark), or fieldbus. In the example shown in Figure 2, the camera 20 and the sorting device 30 are shown connected to a common external device I / F 507, but this is not limited to this, and they may be connected to different interface circuits. In addition, at least one of the camera 20 and the sorting device 30 may communicate data via the network I / F 506.

[0027] The display 508 is a display device such as a liquid crystal or OLED (Organic Light Emitting Diode) that displays various screens and the like.

[0028] The input device 509 is a device such as a mouse, keyboard, or touch panel that allows user input.

[0029] The CPU 501, ROM 502, RAM 503, auxiliary storage device 505, network interface 506, external device interface 507, display 508, and input device 509 described above are connected to each other via a bus 510, such as an address bus and a data bus, enabling communication between them.

[0030] Note that the hardware configuration of the discrimination device 10 shown in Figure 2 is just one example, and it is not necessary to have all of the components, nor is it possible to have other components.

[0031] (Configuration and operation of the sorting system's functional blocks) Figure 3 shows an example of the configuration of the functional blocks of the sorting system according to Embodiment 1. The configuration and operation of the functional blocks of the sorting system 1 according to this embodiment will be described with reference to Figure 3.

[0032] As shown in Figure 3, the discrimination device 10 includes an image processing unit 11, a storage unit 12, and a sorting device control unit 13.

[0033] The image processing unit 11 is a functional unit that receives images of a group of bottles being transported on a belt conveyor 40, captured by the camera 20, via an external device I / F 507, and performs various processes to determine the color type of each bottle from the image of the group of bottles. As shown in Figure 3, the image processing unit 11 includes an image recognition unit 11A, a removal unit 11B, and a color type determination unit 11C. The image processing unit 11 is realized, for example, by a program executed by the CPU 501 shown in Figure 2.

[0034] The image recognition unit 11A is a functional unit that recognizes the bottle image of each bottle from the bottle group image captured by the camera 20. Next, the image recognition unit 11A recognizes the outline of each bottle (hereinafter sometimes referred to as the bottle outline) from the bottle image of each bottle using a trained model for detecting the outline of each bottle based on instance segmentation. This trained model is stored, for example, in the memory unit 12. Here, instance segmentation is a method for identifying the region of an object in an image and recognizing the object by dividing the region for each object. Then, the image recognition unit 11A recognizes the rectangle that circumscribes the recognized bottle outline with the smallest area (hereinafter sometimes referred to as the bottle circumscribed rectangle).

[0035] The trained model may be one that detects the bounding box of each bottle using object detection rather than instance segmentation. Here, a bounding box is a rectangle that circumscribes an object contained in an image, and has sides parallel to the X direction (horizontal direction of the image) and the Y direction (vertical direction of the image). In this case, the image recognition unit 11A recognizes the bounding box for each bottle in the bottle group image using a trained model based on object detection, and recognizes the bottle contour of the bottle contained in each bounding box. Then, the image recognition unit 11A recognizes the bounding rectangle of the bottle that circumscribes the recognized bottle contour with the smallest area.

[0036] Furthermore, while instance segmentation and object detection were used as examples to recognize bottle contours as described above, the method is not limited to these algorithms, and other known algorithms may also be used.

[0037] The removal unit 11B is a functional unit that extracts a rectangular image enclosed by the bounding rectangle of the bottle (hereinafter sometimes referred to as the bounding rectangle image) from the bottle group image and removes areas where glare exists and the label portion image based on binarization processing of brightness and saturation. Here, glare on a bottle refers to the occurrence of areas where brightness is high due to the reflection of light shining on the bottle.

[0038] The color type discrimination unit 11C is a functional unit that discriminates the color type of the bottle in the bounding rectangle image. In this case, the color type discrimination unit 11C discriminates the color type of the bottle using a plurality of trained models stored in the memory unit 12. The color type discrimination unit 11C corresponds to the "discrimination unit" of the present invention.

[0039] The memory unit 12 is a functional unit that stores multiple trained models used in the color type discrimination unit 11C. Specifically, as shown in Figure 3, the memory unit 12 stores the first discrimination model M1, the second discrimination model M2, the third discrimination model M3, the fourth discrimination model M4, the fifth discrimination model M5, and the sixth discrimination model M6 as multiple trained models. The memory unit 12 is implemented by the auxiliary storage device 505 shown in Figure 2.

[0040] The first discriminant model M1 is a pre-trained model specifically designed to distinguish between brown bottles and bottles of other bright colors. The second discriminant model M2 is a pre-trained model specifically designed to distinguish between clear bottles and bottles of other bright colors. The third discriminant model M3 is a pre-trained model specifically designed to distinguish between brown bottles and bottles of other light colors. The fourth discriminant model M4 is a pre-trained model specifically designed to distinguish between clear bottles and bottles of other light colors. The fifth discriminant model M5 is a pre-trained model specifically designed to distinguish between brown bottles and bottles of other dark colors. The sixth discriminant model M6 is a pre-trained model specifically designed to distinguish between clear bottles and bottles of other dark colors.

[0041] Furthermore, the first to sixth discrimination models M1 to M6 described above are not limited to being stored in the memory unit 12. At least one of these trained models may be stored in an external server device, and the color type discrimination unit 11C may access and use the said server device.

[0042] The sorting device control unit 13 is a functional unit that communicates with the sorting device 30 via the external device I / F 507 and controls the operation of the sorting device 30. The sorting device control unit 13 is realized, for example, by a program executed by the CPU 501 shown in Figure 6.

[0043] As shown in Figure 3, the sorting device 30 has a sorting unit 31. The sorting unit 31 is implemented by, for example, a robot hand or a suction pad.

[0044] The sorting device 30, in accordance with the control from the sorting device control unit 13, causes the sorting unit 31 to remove clear bottles, brown bottles, and bottles of other colors from the group of bottles being transported on the belt conveyor 40. The sorting unit 31 is movable in both horizontal and vertical directions. For example, it transports the clear bottles removed from the group of bottles to a first box located to the side of the belt conveyor 40, the brown bottles removed from the group of bottles to a second box located to the side of the belt conveyor 40, and the bottles of other colors removed from the group of bottles to a third box located to the side of the belt conveyor 40. As a result, the group of bottles is separated into clear bottles placed in the first box, brown bottles placed in the second box, and bottles of other colors placed in the third box.

[0045] Furthermore, at least some of the functional units of the image processing unit 11 and the sorting device control unit 13 shown in Figure 3 may be implemented using hardware such as an FPGA (Field Programmable Gate Array) or an integrated circuit such as an ASIC (Application Specific Integrated Circuit), or they may be implemented using a combination of software and hardware.

[0046] Furthermore, the image recognition unit 11A, removal unit 11B, color type discrimination unit 11C, and sorting device control unit 13 shown in Figure 3 are conceptual representations of their functions and are not limited to this configuration. For example, multiple functional units shown as independent functional units in the discrimination device 10 in Figure 3 may be configured as a single functional unit. Alternatively, the functions of one functional unit in the discrimination device 10 shown in Figure 3 may be divided into multiple functions and configured as multiple functional units. Moreover, each functional unit of the discrimination device 10 does not need to be configured as a clear software module as a block as shown in Figure 3; the functions of each functional unit can be realized as a whole when a program is executed in the discrimination device 10.

[0047] (Operation flow of the discrimination device) Figure 4 is a flowchart illustrating an example of the operation flow of the discrimination device according to Embodiment 1. Figure 5 is a diagram illustrating instance segmentation used in the discrimination device according to Embodiment 1. Figure 6 is a diagram illustrating object detection used in the discrimination device according to Embodiment 1. The operation flow of the discrimination device 10 according to this embodiment will be explained with reference to Figures 4 to 6. Note that the processing procedure shown in Figure 4 starts when the bottle group image captured by the camera 20 is input to the image processing unit 11.

[0048] <Step S100> First, the image recognition unit 11A resets the counter value n to "0". Then, it proceeds to step S105.

[0049] <Step S105> Next, the image recognition unit 11A uses the trained model stored in the memory unit 12 to recognize each bottle image included in the bottle group image. Then, the process proceeds to step S110.

[0050] <Step S110> Next, the image recognition unit 11A sets the number of bottle images detected in the image recognition in step S105 to "N". Then, it proceeds to step S115.

[0051] <Step S115> Next, the image recognition unit 11A increments the counter value n by "1". From this point onward, the processing in steps S120 to S185 is performed for each bottle image detected in step S105.

[0052] <Step S120> Next, the image recognition unit 11A uses the trained model stored in the memory unit 12 to recognize the bottle outline (bottle contour) from the bottle image based on instance segmentation. Here, the bottle contour recognition operation by instance segmentation will be explained with reference to Figure 5. Assume that the image shown in Figure 5(a) includes a bottle image B1, and the image recognition unit 11A uses the trained model for detecting bottle contours based on instance segmentation to recognize the bottle contour CO1 shown in Figure 5(b) from the bottle image B1.

[0053] As mentioned above, the trained model may be a trained model that detects the bounding box of each bottle using object detection rather than instance segmentation. Here, Figure 6 illustrates the bottle contour recognition operation using object detection. Assume that the image shown in Figure 6(a) includes a bottle image B1, and the image recognition unit 11A recognizes the bounding box BB1 for the bottle image B1 from the bottle image B1 using a trained model based on object detection, as shown in Figure 6(b). Then, as shown in Figure 6(c), the image recognition unit 11A recognizes the bottle contour CO1 of the bottle included in the bounding box BB1.

[0054] Furthermore, the image recognition unit 11A calculates the coordinates of the area centroid of the region enclosed by the recognized bottle contour (hereinafter sometimes referred to as contour centroid coordinates) and outputs the calculated contour centroid coordinates to the sorting device control unit 13. Then, the process proceeds to step S125.

[0055] <Step S125> Next, the image recognition unit 11A recognizes the bounding rectangle around the bottle contour recognized in step S120. Then, it proceeds to step S130.

[0056] <Step S130> Next, the removal unit 11B extracts the bounding rectangle image enclosed by the bounding rectangle of the bottles from the bottle group image. The bounding rectangle image includes the image enclosed by the bottle contour (hereinafter sometimes referred to as the bottle contour image). The removal unit 11B also replaces the pixels that make up the bounding rectangle image, excluding the pixels that make up the bottle contour image, with pixels with a grayscale value of "0" (i.e., black pixels). The bottle image and bounding rectangle image described above are examples of the "first image" of the present invention. Then, the process proceeds to step S135.

[0057] <Step S135> Next, the removal unit 11B generates an image from the bounding rectangle image after performing a luminance binarization process. Specifically, the removal unit 11B binarizes the luminance of all pixels forming the bottle contour image of the bounding rectangle image into high-luminance and low-luminance components, and generates an image (hereinafter sometimes referred to as the luminance binarized image) in which all pixels forming the bottle contour image are converted into either high-luminance pixels (hereinafter sometimes referred to as high-luminance pixels) or low-luminance pixels (hereinafter sometimes referred to as low-luminance pixels). In this binarization process, the removal unit 11B first converts the RGB pixel data, which is displayed using the three primary colors of red, green, and blue, into YUV pixel data, in which color is represented by a combination of a luminance signal (Y), the difference between the luminance signal and the blue component (U), and the difference between the luminance signal and the red component (V). Then, the removal unit 11B performs the binarization process using the Y component of the YUV pixel data. Furthermore, the removal unit 11B generates a luminance-binarized image by performing a binarization process, for example, using "Otsu's binarization," to convert all pixels forming the bottle contour image into either high-luminance pixels or low-luminance pixels. Alternatively, the removal unit 11B generates a luminance-binarized image by performing a binarization process using a predetermined threshold TH1, converting pixels with a luminance equal to or greater than the threshold TH1 into high-luminance pixels, and converting pixels with a luminance less than the threshold TH1 into low-luminance pixels. The luminance-binarized image is generated separately from the bounding rectangle image and the bottle contour image. Then, the process proceeds to step S140.

[0058] <Step S140> Next, the removal unit 11B identifies the regions where high-luminance pixels exist in the luminance binarized image as regions where glare exists. Then, the removal unit 11B removes glare from the bottle contour image of the circumscribed rectangular image by converting the pixels in the regions corresponding to the regions where high-luminance pixels identified in the luminance binarized image exist, among all the pixels forming the circumscribed rectangular image processed in step S130, into black pixels (hereinafter sometimes referred to as black pixels). For example, if each pixel forming the bottle contour image is represented using 256 gradations from 0 to 255, the removal unit 11B converts the pixels corresponding to the high-luminance pixels in the luminance binarized image among the pixels forming the circumscribed rectangular image into black pixels with a gradation of 0. This removes the glare, which is the part other than the base color of the bottle, and reduces the error with the color characteristics of the bottle that should be distinguished. Hereinafter, the circumscribed rectangular image after glare removal may be referred to as the glare-removed image. As described above, in the bounding rectangle image, the region corresponding to the area where high-luminance pixels identified in the luminance binarized image exist is an example of the "region that satisfies the predetermined conditions" of the present invention. Then, proceed to step S145.

[0059] <Step S145> Next, the removal unit 11B generates an image from the glare-removed image after performing a saturation binarization process. Here, the removal unit 11B excludes pixels from the saturation binarization process that were replaced with black pixels in step S130 and pixels converted to black pixels in step S140 (hereinafter sometimes referred to as first black pixels) from all the pixels forming the glare-removed image. In other words, the removal unit 11B binarizes the saturation of pixels other than the first black pixels (hereinafter sometimes referred to as first effective pixels) from all the pixels forming the glare-removed image into high saturation and low saturation, and generates an image (hereinafter sometimes referred to as a saturation binarized image) in which all first effective pixels included in the glare-removed image are converted into either pixels with high saturation (hereinafter sometimes referred to as high saturation pixels) or pixels with low saturation (hereinafter sometimes referred to as low saturation pixels). In this binarization process, the removal unit 11B first converts the RGB pixel data, which is displayed using the three primary colors Red, Green, and Blue, into HSV pixel data consisting of three components: Hue, Saturation, and Value. Then, the removal unit 11B performs binarization using the S component of the HSV pixel data. The removal unit 11B also performs binarization using, for example, "Otsu's Binarization" to convert all first effective pixels into either high-saturation pixels or low-saturation pixels, thereby generating a saturation-binarized image. Alternatively, the removal unit 11B performs binarization using a predetermined threshold TH2 to convert pixels with a saturation of TH2 or higher into high-saturation pixels and pixels with a saturation of less than TH2 into low-saturation pixels, thereby generating a saturation-binarized image. The saturation-binarized image is generated separately from the glare-removed image. Then, the process proceeds to step S150.

[0060] <Step S150> Next, the removal unit 11B identifies the region where high-saturation pixels exist in the saturation-binarized image as the label portion image. Then, the removal unit 11B removes the label portion image from the gloss-removed image by converting the pixels in the region corresponding to the region where the high-saturation pixels identified in the saturation-binarized image exist, out of all the first effective pixels included in the gloss-removed image, into black pixels. For example, if each pixel forming the gloss-removed image is represented using 256 gradations from 0 to 255, the removal unit 11B converts the pixels corresponding to the high-saturation pixels in the saturation-binarized image, out of all the first effective pixels in the gloss-removed image, into black pixels with a 0 gradation. This makes it possible to remove the label portion image, which is the part other than the background color of the bottle, and reduces the error with the color characteristics of the bottle that should be identified. Hereinafter, the gloss-removed image after the label portion image has been removed may be referred to as the label-removed image. As described above, in the image after gloss removal, the region corresponding to the region where high-saturation pixels identified in the saturation binarized image exist is an example of the "region that satisfies the predetermined conditions" of the present invention. Also, the image after label removal is an example of the "second image" of the present invention. Then, proceed to step S155.

[0061] <Step S155> Next, the color type discrimination unit 11C obtains feature quantities (hereinafter sometimes referred to as image features) from the label-removed image. At this time, the color type discrimination unit 11C excludes the first black pixels and the pixels converted to black pixels in step S150 (hereinafter sometimes referred to as second black pixels) from the acquisition of image features. In other words, the color type discrimination unit 11C obtains image features based on pixels other than the first black pixels and the second black pixels (hereinafter sometimes referred to as second effective pixels) from all pixels that make up the label-removed image. For example, the color type discrimination unit 11C obtains the average value of the saturation of all second effective pixels (hereinafter sometimes referred to as the average saturation) and the average value of the luminance of all second effective pixels (hereinafter sometimes referred to as the average luminance) as image features. Then, the process proceeds to step S160.

[0062] <Step S160> Next, the color type discrimination unit 11C determines whether the average saturation value is greater than or equal to the threshold THA. If the average saturation value is greater than or equal to the threshold THA (step S160: Yes), the process proceeds to step S165. If the average saturation value is less than the threshold THA (step S160: No), the process proceeds to step S170.

[0063] <Step S165> If the average saturation value is greater than or equal to the threshold THA (step S160: Yes), the color type discrimination unit 11C selects a first discrimination model M1 and a second discrimination model M2 from among the multiple trained models stored in the memory unit 12 as trained models to be used to discriminate the type (color type) of bottle based on the label-removed image. Then, based on the label-removed image, the color type discrimination unit 11C uses the two trained models, the first discrimination model M1 and the second discrimination model M2, to determine whether the type (color type) of bottle in the bounding rectangle image extracted in step S130 is a transparent bottle, a brown bottle, or a bottle of another color. The process then proceeds to step S185.

[0064] <Step S170> The color type discrimination unit 11C determines whether the average luminance value is greater than or equal to the threshold THB. If the average luminance value is greater than or equal to the threshold THB (step S170: Yes), the process proceeds to step S175. If the average luminance value is less than the threshold THB (step S170: No), the process proceeds to step S180.

[0065] <Step S175> If the average saturation value is less than the threshold THA and the average luminance value is greater than or equal to the threshold THB (step S160: No, step S170: Yes), the color type discrimination unit 11C selects the third discrimination model M3 and the fourth discrimination model M4 from among the multiple trained models stored in the memory unit 12 as trained models to be used to discriminate the type (color type) of bottle based on the label-removed image. Then, based on the label-removed image, the color type discrimination unit 11C uses the two trained models, the third discrimination model M3 and the fourth discrimination model M4, to determine whether the type (color type) of bottle in the bounding rectangle image extracted in step S130 is a transparent bottle, a brown bottle, or a bottle of another color. The process then proceeds to step S185.

[0066] <Step S180> If the average saturation value is less than the threshold THA and the average luminance value is less than the threshold THB (step S160: No, step S170: No), the color type discrimination unit 11C selects the fifth discrimination model M5 and the sixth discrimination model M6 from among the multiple trained models stored in the memory unit 12 as trained models to be used to discriminate the type (color type) of bottle based on the label-removed image. Then, based on the label-removed image, the color type discrimination unit 11C uses the two trained models, the fifth discrimination model M5 and the sixth discrimination model M6, to determine whether the type (color type) of bottle in the bounding rectangle image extracted in step S130 is a transparent bottle, a brown bottle, or a bottle of another color. Then, the process proceeds to step S185.

[0067] As described in steps S160 and S170 above, the color type discrimination unit 11C determines a trained model to be used for color type discrimination based on the average saturation value and the average luminance value. Here, when discriminating bottles into three types: clear bottles, brown bottles, and other colored bottles, other colored bottles include a wide variety of colors such as green, black, blue, and red. If these are treated as the same class by a machine learning classification algorithm (for example, SVM (Support Vector Machine)), the variation in color features may be large, and the discrimination accuracy may decrease. Therefore, as described above, by switching to a trained model specialized for discrimination accuracy according to color features and performing inference, the discrimination accuracy can be further improved. Alternatively, the color type discrimination unit 11C may determine the trained model to be used for color type discrimination based on the mode of saturation and the mode of luminance instead of the average saturation value and the average luminance value.

[0068] <Step S185> The image recognition unit 11A determines whether the counter value n has reached the number of detections N. If the counter value n has reached the number of detections N (step S185: Yes), the process proceeds to step S190. If the counter value n has not reached the number of detections N (step S185: No), the process returns to step S115.

[0069] <Step S190> The sorting device control unit 13 sets the centroid coordinates of the contour of each bottle image calculated in step S120 to the coordinates indicating the extraction point when the sorting unit 31 extracts the bottles (hereinafter sometimes referred to as extraction point coordinates). Here, the extraction point coordinates set by the sorting device control unit 13 are coordinates in the coordinate system of the bottle group image, that is, the coordinate system of the camera 20. Therefore, the sorting device control unit 13 converts the set extraction point coordinates in the camera 20 coordinate system to the extraction point coordinates in the coordinate system of the sorting device 30. The sorting device control unit 13 outputs a control signal including the converted extraction point coordinates to the sorting device 30. Then, the process proceeds to step S200.

[0070] <Step S200> The sorting device 30 then sequentially moves the sorting unit 31 directly above each of the pickup point coordinates indicated in the control signal. The sorting unit 31 uses each of the pickup point coordinates as the pickup point to pick up clear bottles, brown bottles, and bottles of other colors from the group of bottles being transported on the belt conveyor 40, transporting the clear bottles to the first box, the brown bottles to the second box, and the bottles of other colors to the third box. If the sorting unit 31 is formed using a robot hand, the sorting unit 31 sets the pickup point to the gripping position of the robot hand. If the sorting unit 31 is formed using a suction pad, the sorting unit 31 sets the pickup point to the suction position of the suction pad.

[0071] (Example of image processing unit operation) Figure 7 shows an example of the operation of the image processing unit of the discrimination device according to Embodiment 1. Figure 8 shows an example of the operation of the image processing unit of the discrimination device according to Embodiment 1. An example of the operation of the image processing unit 11 of the discrimination device 10 according to this embodiment will be explained with reference to Figures 7 and 8.

[0072] For example, when a group image of bottles W11, as shown in Figure 7, is captured by the camera 20, the group image of bottles W11 includes a bottle image B11, which is an image of a brown bottle. Also, as shown in Figure 7, the bottle image B11 is an image of a bottle with a label, and includes a label portion image LB11 that covers the entire body of the bottle.

[0073] As shown in Figure 7, first, the image recognition unit 11A recognizes the bottle contour CO11 of the bottle image B11 and calculates the contour centroid coordinates CG11 of the bottle contour CO11 (step S120).

[0074] Next, the image recognition unit 11A recognizes the bounding rectangle RE11 of the bottle with respect to the bottle contour CO11 (step S125).

[0075] Next, the removal unit 11B extracts a bounding rectangle image (i.e., a rectangle enclosed by the bounding rectangle RE11) from the bottle group image W11 that includes the bottle contour image corresponding to the bottle image B11 (step S130).

[0076] Next, as shown in Figure 8, the removal unit 11B generates a luminance-binarized image B12 by performing a luminance binarization process on the bottle contour image corresponding to the bottle image B11 (step S135).

[0077] Next, the removal unit 11B removes glare from the bounding rectangle image including the bottle contour image using the brightness binarized image B12 to obtain the glare-removed image B13 (step S140).

[0078] Next, the removal unit 11B generates a saturation-binarized image B14 by performing a saturation binarization process on the reflective image B13 (step S145).

[0079] Then, the removal unit 11B obtains the label-removed image B15 by removing the label portion image LB1 from the gloss-removed image B13 using the saturation-binarized image B14 (step S150).

[0080] The above describes Example 1.

[0081] [Example 2] This section will describe the sorting system 1 according to Example 2, focusing on the differences from the sorting system 1 according to Example 1. In Example 1, the operation of removing the area containing glare from the bounding rectangle image before removing the label portion image was described. In this example, the operation of removing the label portion image from the bounding rectangle image before removing the area containing glare will be described. The overall configuration of the sorting system 1 according to this example, as well as the hardware configuration and functional block configuration of the discrimination device 10, are the same as those described in Example 1 above.

[0082] (Operation flow of the discrimination device) Figure 9 is a flowchart showing an example of the operation flow of the discrimination device according to Embodiment 2. Figure 10 is a diagram showing an example of the operation of the image processing unit of the discrimination device according to Embodiment 2. The operation flow of the discrimination device 10 according to this embodiment will be explained with reference to Figures 9 and 10. Note that the processing procedure shown in Figure 9 starts when the bottle group image captured by the camera 20 is input to the image processing unit 11.

[0083] <Step S100~S130> The processing in steps S100 to S130 is as described in Figure 4 above. After the processing in step S130, the process proceeds to step S135A.

[0084] <Step S135A> Next, the removal unit 11B generates an image from the bounding rectangle image by performing a saturation binarization process. Here, the removal unit 11B binarizes the saturation of all pixels forming the bottle contour image of the bounding rectangle image into high saturation and low saturation, and generates an image (hereinafter sometimes referred to as the saturation binarized image) in which all pixels forming the bottle contour image are converted into either pixels with high saturation (hereinafter sometimes referred to as high saturation pixels) or pixels with low saturation (hereinafter sometimes referred to as low saturation pixels). In this binarization process, the removal unit 11B first converts the RGB format pixel data, which is displayed using the three primary colors of red, green, and blue, into HSV format pixel data consisting of three components: hue, saturation, and value. Then, the removal unit 11B performs the binarization process using the S component of the HSV format pixel data. Furthermore, the removal unit 11B generates a saturation-binarized image by performing a binarization process, for example, using "Otsu's binarization," to convert all pixels forming the bottle contour image into either high-saturation pixels or low-saturation pixels. Alternatively, the removal unit 11B generates a saturation-binarized image by performing a binarization process using a predetermined threshold TH2, converting pixels with a saturation of TH2 or higher into high-saturation pixels, and converting pixels with a saturation of less than TH2 into low-saturation pixels. Note that the saturation-binarized image is generated separately from the bounding rectangle image and the bottle contour image.

[0085] For example, in the case of the bottle image B11 shown in Figure 10, the removal unit 11B generates a saturation-binarized image B12A by performing a saturation binarization process on the bottle contour image corresponding to the bottle image B11.

[0086] Then, we move on to step S140A.

[0087] <Step S140A> Next, the removal unit 11B identifies the region where high-saturation pixels exist in the saturation-binarized image as the label portion image. Then, the removal unit 11B removes the label portion image from the bottle contour image of the circumscribing rectangle image by converting the pixels in the region corresponding to the region where the high-saturation pixels identified in the saturation-binarized image exist, out of all the pixels forming the circumscribing rectangle image processed in step S130, into black pixels. For example, if each pixel forming the bottle contour image is represented using 256 levels of grayscale from 0 to 255, the removal unit 11B converts the pixels in the circumscribing rectangle image corresponding to the high-saturation pixels in the saturation-binarized image into black pixels of level 0. This makes it possible to remove the label portion image, which is the part other than the background color of the bottle, and reduces the error with the color characteristics of the bottle that should be identified. Hereinafter, the circumscribing rectangle image after the label portion image has been removed may be referred to as the label-removed image.

[0088] For example, in the case of the saturation-binarized image B12A shown in Figure 10, the removal unit 11B identifies the region in the saturation-binarized image B12A where high-saturation pixels exist as the label portion image. Then, the removal unit 11B removes the label portion image from the bottle image B11 by converting the pixels in the region corresponding to the region where the high-saturation pixels identified in the saturation-binarized image B12A exist, out of all the pixels forming the bounding rectangle image of the bottle image B11, into black pixels, thereby obtaining the label-removed image B13A.

[0089] As mentioned above, in the bounding rectangle image, the region corresponding to the area where high-saturation pixels identified in the saturation-binarized image exist is an example of the "region that satisfies the predetermined conditions" of the present invention.

[0090] Then, proceed to step S145A.

[0091] <Step S145A> Next, the removal unit 11B generates an image from the label-removed image after brightness binarization processing. Here, the removal unit 11B excludes pixels that were replaced with black pixels in step S130 and pixels that were converted to black pixels in step S140A (hereinafter sometimes referred to as first black pixels) from the brightness binarization processing among all pixels forming the label-removed image. In other words, the removal unit 11B binarizes the brightness of pixels other than the first black pixels (hereinafter sometimes referred to as first effective pixels) among all pixels forming the label-removed image into high brightness and low brightness, and generates an image (hereinafter sometimes referred to as a brightness binarized image) in which all first effective pixels included in the label-removed image are converted into either pixels with high brightness (hereinafter sometimes referred to as high brightness pixels) or pixels with low brightness (hereinafter sometimes referred to as low brightness pixels). In this binarization process, the removal unit 11B first converts the RGB pixel data, which is displayed using the three primary colors red, green, and blue, into YUV pixel data, in which color is represented by a combination of a luminance signal (Y), the difference between the luminance signal and the blue component (U), and the difference between the luminance signal and the red component (V). Then, the removal unit 11B performs binarization using the Y component of the YUV pixel data. The removal unit 11B also performs binarization using, for example, "Otsu's binarization" to convert all first effective pixels into either high-luminance pixels or low-luminance pixels, thereby generating a luminance-binarized image. Alternatively, the removal unit 11B performs binarization using, for example, a predetermined threshold TH1, converting pixels with a luminance of TH1 or higher into high-luminance pixels and pixels with a luminance of less than TH1 into low-luminance pixels, thereby generating a luminance-binarized image. The luminance-binarized image is generated separately from the label-removed image.

[0092] For example, in the case of the label-removed image B13A shown in Figure 10, the removal unit 11B generates the luminance-binarized image B14A by performing a luminance binarization process on the label-removed image B13A.

[0093] Then, we move on to step S150A.

[0094] <Step S150A> Next, the removal unit 11B identifies the regions where high-luminance pixels exist in the luminance binarized image as regions where glare exists. Then, the removal unit 11B removes the glare from the label-removed image by converting the pixels in the regions corresponding to the areas where high-luminance pixels identified in the luminance binarized image exist, among all the first effective pixels included in the label-removed image, into black pixels. For example, if each pixel forming the label-removed image is represented using 256 gradations from 0 to 255, the removal unit 11B converts the pixels forming the label-removed image that correspond to the high-luminance pixels in the luminance binarized image into black pixels with a gradation of 0. This removes the glare, which is the part other than the base color of the bottle, and reduces the error with the color characteristics of the bottle that should be distinguished. Hereinafter, the label-removed image after the glare has been removed may be referred to as the glare-removed image.

[0095] For example, in the case of the luminance binarized image B14A shown in Figure 10, the removal unit 11B identifies the regions in the luminance binarized image B14A where high-luminance pixels exist as regions where glare exists. Then, the removal unit 11B removes the glare from the label-removed image B13A by converting the pixels in the regions corresponding to the regions where high-luminance pixels identified in the luminance binarized image B14A exist, out of all the first effective pixels included in the label-removed image B13A, into black pixels, thereby obtaining the glare-removed image B15A.

[0096] As described above, in the label-removed image, the region corresponding to the area where high-luminance pixels identified in the luminance binarized image exist is an example of the "region that satisfies the predetermined conditions" of the present invention. Furthermore, the glare-removed image is an example of the "second image" of the present invention.

[0097] Then, we move on to step S155.

[0098] <Step S155> Next, the color type discrimination unit 11C acquires feature quantities (hereinafter sometimes referred to as image features) from the image after glare removal. At this time, the color type discrimination unit 11C excludes the first black pixels and the pixels converted to black pixels in step S150A (hereinafter sometimes referred to as second black pixels) from acquiring image features. In other words, the color type discrimination unit 11C acquires image features based on pixels other than the first black pixels and the second black pixels (hereinafter sometimes referred to as second effective pixels) from all pixels that make up the image after glare removal. For example, the color type discrimination unit 11C acquires the average value of the saturation of all second effective pixels (hereinafter sometimes referred to as the average saturation) and the average value of the luminance of all second effective pixels (hereinafter sometimes referred to as the average luminance) as image features. Then, the process proceeds to step S160.

[0099] <Step S160~S200> The processing in steps S160 to S200 is as described in Figure 4 above. However, in steps S165, S175, and S180, the color type discrimination unit 11C uses the trained model from each step to determine whether the type (color type) of the bottle in the bounding rectangle image extracted in step S130 is a transparent bottle, a brown bottle, or a bottle of another color, based on the image after gloss removal.

[0100] The above describes Example 2.

[0101] As described above, in Example 1, a gloss-removed image was generated from the bounding rectangle image using the luminance-binarized image, and a label-removed image was generated from the gloss-removed image using the saturation-binarized image. In Example 2, a label-removed image was generated from the bounding rectangle image using the saturation-binarized image, and a gloss-removed image was generated from the label-removed image using the luminance-binarized image. However, the method is not limited to these methods. For example, both luminance binarization and saturation binarization may be performed on the bounding rectangle image to generate a luminance-binarized image and a saturation-binarized image, respectively, and then an image with gloss and label portions removed can be directly obtained from the bounding rectangle image using both the luminance-binarized image and the saturation-binarized image. In this case, the removal unit 11B obtains an image from which glare and label portion images have been removed by converting pixels that correspond to at least one of the high-luminance pixels in the luminance-binarized image and the high-saturation pixels in the saturation-binarized image, among the pixels that form the bounding rectangle image (the bounding rectangle image processed in step S130), into black pixels.

[0102] Furthermore, in the embodiments described above, both the gloss and label portion images are removed from the bounding rectangle image, but at least one of the gloss and label portion images may be removed. This also improves the accuracy of distinguishing the bottle color.

[0103] As described above, the discrimination device 10 of this disclosure comprises a removal unit 11B and a color type discrimination unit 11C. The removal unit 11B removes from a first image, which is an image of a bottle, a region that satisfies a predetermined condition for at least one of the brightness or saturation in the first image. The color type discrimination unit 11C determines the color type of the bottle based on a second image, which is the image after the region has been removed. This makes it possible to perform discrimination based on an image mainly consisting of the background color portion of the bottle by removing at least one of the gloss or label portion image, thereby improving the accuracy of bottle color type discrimination.

[0104] Furthermore, the removal unit 11B removes from the first image any area in the first image where the brightness or saturation is above a predetermined threshold. This allows for the removal of at least one of the glossy or label portion images based on threshold determination, thereby improving the accuracy of distinguishing the color type of the bottle.

[0105] Furthermore, the removal unit 11B removes areas from the first image that satisfy predetermined conditions regarding brightness. This removes glare, thereby improving the accuracy of distinguishing the color type of the bottle.

[0106] Furthermore, the removal unit 11B removes areas from the first image that satisfy predetermined conditions regarding saturation. This allows for the removal of the label portion of the image, thereby improving the accuracy of distinguishing the color type of the bottle.

[0107] Furthermore, the removal unit 11B removes areas from the first image that satisfy predetermined conditions regarding brightness and saturation. This removes glare and label areas, thereby improving the accuracy of distinguishing the color of the bottle.

[0108] Furthermore, the color type discrimination unit 11C uses one of several trained models to determine the color type of the bottle based on the second image. The color type discrimination unit 11C also determines which trained model to use for determining the color type of the bottle from among several trained models based on the saturation and brightness of the second image. By switching to a trained model specialized for discrimination accuracy according to the color features and performing inference in this way, the discrimination accuracy can be further improved.

[0109] In each of the embodiments described above, if at least one of the functional units of the discrimination device 10 is implemented by program execution, the program is provided pre-installed in a ROM or the like. In each of the embodiments described above, the program executed by the discrimination device 10 may be configured to be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk-Recordable), or a DVD (Digital Versatile Disc). In each of the embodiments described above, the program executed by the discrimination device 10 may be configured to be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. In each of the embodiments described above, the program executed by the discrimination device 10 may be configured to be provided or distributed via a network such as the Internet. Furthermore, in each of the embodiments described above, the program executed by the discrimination device 10 is a module configuration that includes at least one of the functional units described above. In actual hardware, the CPU 501 reads the program from a storage device (ROM 502 or auxiliary storage device 505, etc.) and executes it, thereby loading and generating the functional units described above onto the main memory (ROM 502). [Explanation of Symbols]

[0110] 1. Sorting System 10 Discrimination device 11 Image Processing Unit 11A Image recognition section 11B Removal part 11C Color type discrimination section 12 Storage section 13. Sorting device control unit 20 cameras 30 Sorting device 31 Sorting section 40 Belt conveyor 501 CPU 502 ROM 503 RAM 505 Auxiliary storage 506 Network Interface 507 External device I / F 508 displays 509 Input device 510 Bus B1, B11 bottle images B12 Image after brightness binarization B12A Image after saturation binarization B13 Image after shine removal Image after B13A label removal B14 Image after saturation binarization B14A Image after brightness binarization B15 Image after label removal B15A Image after shine removal BB1 Bounding Box CD transport direction CG11 Contour centroid coordinates CO1, CO11 bottle outline LB11 label image M1 First Discrimination Model M1 M2 Second Discrimination Model M2 M3 Third Discrimination Model M3 M4 4th Identification Model M4 M5 5th Discrimination Model M5 M6 6th Discrimination Model M6 RE11 Bottle circumscribing rectangle W11 Bottle Group Image

Claims

1. A removal unit removes from a first image, which is an image of a bottle, an area that satisfies a predetermined condition for at least one of the brightness or saturation in the first image, A discrimination unit that determines the type of color of the bottle based on a second image which is an image after the aforementioned region has been removed, A discrimination device is provided.

2. The discrimination device according to claim 1, wherein the removal unit removes the region in the first image in which the brightness or saturation is equal to or greater than a predetermined threshold.

3. The discrimination device according to claim 1, wherein the removal unit removes from the first image an area that satisfies the conditions for brightness as the region.

4. The discrimination device according to claim 1, wherein the removal unit removes from the first image an area that satisfies the conditions for saturation as the region.

5. The discrimination device according to claim 1, wherein the removal unit removes from the first image regions that satisfy the conditions for brightness and saturation.

6. The discrimination device according to any one of claims 1 to 5, wherein the discrimination unit determines the type of color of the bottle based on the second image using one of a plurality of trained models.

7. The discrimination device according to claim 6, wherein the discrimination unit determines a trained model to be used for discriminating the type from among the plurality of trained models based on the saturation of the second image and the brightness of the second image.

8. A removal step of removing from a first image, which is an image of a bottle, an area that satisfies a predetermined condition for at least one of the brightness or saturation in the first image, A determination step to determine the type of color of the bottle based on a second image, which is an image after the aforementioned region has been removed, A method for determining whether a method exists.

9. On the computer, A removal step of removing from a first image, which is an image of a bottle, an area that satisfies a predetermined condition for at least one of the brightness or saturation in the first image, A determination step to determine the type of color of the bottle based on a second image, which is an image after the aforementioned region has been removed, A program to execute.

Images