Discrimination device, discrimination method, and program

Abstract

The present invention provides a discrimination device, discrimination method, and program that can improve the accuracy of separating objects to be sorted from foreign objects. [Solution] The discrimination device 10 comprises a discrimination unit 11D and a determination unit 11E. The discrimination unit 11D uses one or more trained models, which have been trained using images of bottles to be sorted and images of bottles that are foreign objects and not to be sorted, to determine the type of target bottle. The determination unit 11E determines whether or not the target bottle is a foreign object based on the discrimination result of the discrimination unit 11D.

Description

Technical Field

[0006] , , ,

[0005] , , , , ,

[0001] The technology of the present disclosure relates to a discrimination device, a discrimination method, and a program.

Background Art

[0002] At waste treatment plants, a large amount of waste flows on a belt conveyor every day and is being processed. At the site where the waste is processed, the sorting work of the waste is carried out manually. Although the sorting work of the waste is a simple task, since the burden on the workers who sort the waste is large, a system that automatically sorts the waste (hereinafter sometimes referred to as a "waste sorting system") has been developed.

[0003] When the waste sorting system performs the work that the operator used to do on behalf of the operator, in the waste sorting system, each bottle flowing on the belt conveyor is recognized, and based on the recognition result, a desired sorting target such as recycling is taken out from the group of bottles flowing on the belt conveyor using a robot hand or a suction pad.

Prior Art Documents

Patent Documents

[0007] A discrimination device according to one aspect of the present disclosure comprises a discrimination unit and a determination unit. The discrimination unit determines the type of target bottle using one or more trained models that have been trained using images of bottles to be sorted and images of bottles that are foreign objects and not to be sorted. The determination unit determines whether or not the target bottle is a foreign object based on the discrimination result of the discrimination unit for the target bottle. [Effects of the Invention]

[0008] The disclosure discrimination device, discrimination method, and program can improve the accuracy of separating the object to be sorted from foreign matter. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of the configuration of the sorting system according to Example 1. [Figure 2] Figure 2 shows specific examples of foreign matter to be sorted and foreign matter not to be sorted in the sorting system according to Example 1. [Figure 3] Figure 3 is a diagram illustrating the outline of the process for improving the bottle identification accuracy in the sorting system according to Example 1. [Figure 4] Figure 4 illustrates the training process for the trained model used in the discrimination device according to Example 1. [Figure 5] Figure 5 shows an example of the type subdivision output by the trained model used in the discrimination device according to Example 1. [Figure 6] Figure 6 shows an example of the hardware configuration of the discrimination device according to Example 1. [Figure 7] Figure 7 shows an example of the configuration of a functional block in the sorting system according to Example 1. [Figure 8]FIG. 8 is a diagram showing an operation example of the image processing unit of the discrimination device according to Example 1. [Figure 9] FIG. 9 is a diagram showing an operation example of the image processing unit of the discrimination device according to Example 1. [Figure 10] FIG. 10 is a diagram showing an operation example of the image processing unit of the discrimination device according to Example 1. [Figure 11] FIG. 11 is a diagram showing an operation example of the image processing unit of the discrimination device according to Example 1. [Figure 12] FIG. 12 is a diagram showing an operation example of the image processing unit of the discrimination device according to Example 1. [Figure 13] FIG. 13 is a diagram showing an operation example of the image processing unit of the discrimination device according to Example 1. [Figure 14] FIG. 14 is a diagram showing an operation example of the image processing unit of the discrimination device according to Example 1. [Figure 15] FIG. 15 is a diagram showing an operation example of the image processing unit of the discrimination device according to Example 1. [Figure 16] FIG. 16 is a flowchart showing an example of the operation flow of the discrimination device according to Example 1. [Figure 17] FIG. 17 is a diagram for explaining the instance segmentation used in the discrimination device according to Example 1. [Figure 18] FIG. 18 is a diagram for explaining the object detection used in the discrimination device according to Example 1. [Figure 19] FIG. 19 is a diagram showing an example of a bottle group image input to the discrimination device according to Example 1. [Figure 20] FIG. 20 is a diagram for explaining the contour recognition operation by the discrimination device according to Example 1. [Figure 21] FIG. 21 is a diagram for explaining the circumscribed rectangle recognition operation by the discrimination device according to Example 1. [Figure 22] FIG. 22 is a diagram for explaining the operation of calculating the contour centroid coordinates by the discrimination device according to Example 1. [Figure 23] FIG. 23 is a diagram for explaining the display operation of the discrimination result of the type by the discrimination device according to Example 1. [Figure 24] FIG. 24 is a diagram for explaining the display operation of the determination result as to whether an object is a foreign object by the discrimination device according to Example 1. [Figure 25] FIG. 25 is a diagram for explaining the operation of displaying the discrimination result and the determination result by the discrimination device according to Example 1 as character information. [Figure 26] FIG. 26 is a flowchart showing an example of the operation flow of the discrimination device according to Example Ⅱ.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the discrimination device, discrimination method, and program disclosed in the present application will be described in detail with reference to the drawings. Further, the technology of the present disclosure is not limited by the following description, and the components in the following description include those that can be easily conceived by those skilled in the art, substantially the same ones, and those within the so-called equivalent range. Furthermore, various omissions, substitutions, changes, and combinations of components can be made without departing from the gist of the following embodiments.

[0011] [Example 1] (Configuration and Operation Outline of the Sorting System) FIG. 1 is a diagram showing an example of the configuration of the sorting system according to Example 1. FIG. 2 is a diagram showing specific examples of sorting targets and foreign objects outside the sorting targets in the sorting system according to Example 1. FIG. 3 is a diagram for explaining an outline of processing for improving the discrimination accuracy of bottles in the sorting system according to Example 1. FIG. 4 is a diagram for explaining the learning process of the learned model used in the discrimination device according to Example 1. FIG. 5 is a diagram showing an example of the classification of types output by the learned model used in the discrimination device according to Example 1. With reference to FIGS. 1 to FIG. 5, the configuration and operation of the sorting system 1 according to the present embodiment will be described.

[0012] In actual practice, foreign objects other than the clear bottles, brown bottles, and other colored bottles (hereinafter sometimes referred to as "other colored bottles") that are subject to sorting may be mixed in. For example, bottles subject to sorting include empty bottles that contained beverages (hereinafter sometimes referred to as "beverage bottles") and empty bottles that contained food (hereinafter sometimes referred to as "food bottles"). Specific examples of foreign objects not subject to sorting will be described in detail later, but examples include empty bottles that contained objects other than food and beverages (hereinafter sometimes referred to as "non-food bottles"). An example of a food bottle is an empty jam jar, and an example of a non-food bottle is an empty bottle that contained liquid or cream-type cosmetics (hereinafter sometimes referred to as "cosmetic bottles"). Hereafter, beverage bottles and food bottles may be collectively referred to as "food and beverage bottles."

[0013] The shape of beverage bottles and non-food bottles often consists of a mouth (the narrowest part of the bottle's width), a neck (connecting to the mouth), a body (wider than the neck), and a shoulder (connecting the neck and body). Furthermore, the body of beverage and non-food bottles often has a columnar shape. Additionally, beverage and non-food bottles often have distinctive shapes in their mouths, necks, and shoulders. On the other hand, food bottles often have a mouth wider than the mouth of beverage or non-food bottles, a body approximately the same width as the mouth, and a columnar shape without a neck or shoulder. Hereinafter, empty bottles with a columnar shape lacking a neck and shoulder may be referred to as "columnar-shaped bottles."

[0014] The following describes in detail a sorting system 1 that can improve the accuracy of separating the target material from foreign objects.

[0015] As shown in Figure 1, the sorting system 1 includes a discrimination device 10, a camera 20, a sorting device 30, and a belt conveyor 40. The discrimination device 10, the camera 20, and the sorting device 30 are capable of data communication with each other.

[0016] In the following explanation, we will use the case where the sorting system 1 shown in Figure 1 is installed in a waste treatment plant where a group of bottles flows on a belt conveyor 40 as an example. In other words, in the following explanation, we will use the case where the objects to be sorted by the sorting system 1 are bottles as an example. Here, "bottle" is a concept that includes not only glass bottles, but also plastic bottles, containers, PET bottles, etc. Furthermore, the items to be sorted for recycling, etc., from the group of bottles flowing on the belt conveyor 40 are assumed to be, for example, the clear bottles shown in Figure 2(a), the brown bottles shown in Figure 2(b), other colored bottles, and PET bottles. Such beverage bottles to be sorted have a shape designed on the premise that a person will drink from them or pour from them (for example, a shape with an opening of roughly 2-3 cm in diameter and a threaded or hooked part so that it can be sealed with a lid or cap).

[0017] Examples of foreign objects not subject to sorting include cosmetic bottles that contained liquids used for makeup, as shown in Figure 2(c), and bottles that contained liquids such as lotions, aromatherapy oils, or medicinal solutions (hereinafter sometimes referred to as pharmaceutical bottles). As shown in Figure 2(c), such cosmetic bottles and pharmaceutical bottles come in a variety of shapes designed for different uses than beverage bottles, such as spray, pump, and dropper types, and the overall shape of the bottles is often non-cylindrical with design in mind. In the case of spray-type cosmetic bottles, as shown in Figure 3(b), the body is similar in color, pattern, and texture to that of clear bottles, making it difficult to distinguish them from clear bottles. Therefore, since the shape of the mouth of such cosmetic bottles is designed for spraying and has features useful for identifying the type, a bottle portion image LO12 including the vicinity of the mouth is extracted from the bottle image and used for type identification.

[0018] Furthermore, foreign objects that are not subject to sorting include glass products such as cups, containers, and trays (hereinafter sometimes referred to as containers), as shown in Figure 2(d). Such containers often have various overall shapes, such as being non-cylindrical or having handles.

[0019] The cosmetic bottles, medicine bottles, and containers mentioned above are just examples of bottles for non-food and beverage use.

[0020] Furthermore, foreign objects not subject to sorting include bottles that are similar in shape to food bottles, as shown in Figure 2(e), but which contain all or part of their contents (hereinafter sometimes referred to as "content-filled bottles"). In the case of content-filled bottles, as shown in Figure 3(d), the contents are stuck to the entire surface except for the label, making it easy to identify the type by its pattern (texture). Therefore, the entire bottle image is used for type identification.

[0021] Such bottles filled with contents can be easily identified by using the texture (pattern or feel) of parts other than the label, as shown in Figure 2(e). In addition, foreign objects that are not subject to sorting include bottles in which a different type of bottle is embedded inside another bottle, as shown in Figure 2(f) (hereinafter sometimes referred to as bottle-in-bottle). Such bottle-in-bottles should not be sorted and should be treated as foreign objects, as they will pick up other types of bottles together if grasped as a sorting object by a sorting device 30 such as a robot, which will be described later. Furthermore, bottle-in-bottles can be easily identified by using the texture other than the label (for example, as shown in Figure 2(f), a colored bottle such as a brown bottle is visible inside a clear bottle) or the overall shape (for example, as shown in Figure 2(f), a colored bottle is sticking out of a clear bottle). In the case of a bottle-in-bottle as shown in Figure 3(c), the body is difficult to distinguish from a clear bottle because its color, pattern, and texture are the same as those of a clear bottle. Therefore, since this type of bottle-in-a-bottle has a mixture of the colors, patterns, and textures of both the brown and clear bottles near the mouth, which is a useful feature for distinguishing between types, a bottle portion image LO13 including the area near the mouth is extracted from the bottle image and used for type identification.

[0022] Furthermore, as foreign objects not subject to sorting, bottles that are largely covered with labels of a different color from the background color, as shown in Figure 2(g), and where the area where the background color is visible is small (hereinafter sometimes referred to as "label-covered bottles") are anticipated. As shown in Figure 3(a), such label-covered bottles are covered with labels from the shoulder to the body, and the background color is not visible, so the type identification depends on the color and pattern of the label. For this reason, the area near the mouth where the background color can be seen has features that are useful for identifying the type of label-covered bottle, so a bottle portion image LO11 including the area near the mouth is extracted from the bottle image and used for type identification.

[0023] The above-mentioned selection targets and foreign objects are merely examples, and the composition of the bottles to be selected and considered as foreign objects may be different. Bottles other than those shown in Figure 2 may also be included as selection targets or foreign objects.

[0024] Camera 20 is positioned above the conveyor belt 40 on which the bottles are transported, has a predetermined field of view, and is an imaging device that captures a predetermined area on the upper surface of the conveyor belt 40 from above the conveyor belt 40 at a constant frame rate. Therefore, the image captured by camera 20 becomes an image of the bottles (hereinafter sometimes referred to as a bottle group image). The bottle group image is transmitted from camera 20 to the discrimination device 10. Note that camera 20 may be an imaging sensor or an area sensor, etc.

[0025] The discrimination device 10 is a device that determines the type of each bottle flowing on the conveyor belt 40 based on images of the bottles captured by the camera 20. The discrimination device 10 also controls the operation of the sorting device 30 according to the determination result of each bottle type.

[0026] Furthermore, the discrimination device 10 uses a trained model to distinguish the type of bottle. As shown in Figure 4, the trained model used in the discrimination device 10 is generated by machine learning pre-training, using not only images of bottles to be sorted, such as clear bottles, brown bottles, other colored bottles, and PET bottles, but also images of foreign bottles such as cosmetic bottles, medicine bottles, containers, bottles filled with contents, bottle-in-bottles, and bottles with labels, as training data. By using such machine learning, a trained model can be obtained that can distinguish the type of bottle by comprehensively judging various features such as the shape of the bottle's mouth, overall shape, color, and texture, without uniformly defining the judgment materials and criteria. In addition, by actively training the model not only on images of bottles to be sorted but also on images of foreign bottles that are not to be sorted, the recognition accuracy of bottle type discrimination can be improved. Furthermore, the discrimination device 10 uses multiple trained models (trained model B and trained model C, described later) as the trained models learned as described above. Trained model B is a trained model that has been trained using overall images (overall bottle images) of various bottle images as described above. In other words, trained model B takes the entire bottle image as input and outputs a result of classifying at least the type of bottle indicated by that bottle image. Trained model C is a trained model that has been trained using partial images of the area around the mouth of a bottle (partial bottle image), which are considered useful for classifying the type of bottle from among the various bottle images described above. In other words, trained model C takes a partial bottle image as input and outputs a result of classifying at least the type of bottle indicated by that partial bottle image. By using multiple trained models for classifying the type of bottle in this way, the recognition accuracy of classifying the type of bottle can be further improved.

[0027] Furthermore, the bottle type classification results output by the trained model described above may include not only the types shown in Figure 2, but also more subdivided types. For example, as shown in Figure 5(a), for cosmetic bottles, subdivided types such as spray-type cosmetic bottles (cosmetic bottle type a) and pump-type cosmetic bottles (cosmetic bottle type b) may be output depending on the shape of the opening. Similarly, as shown in Figure 5(b), subdivided types such as jam-type content-filled bottles (content-filled bottle type a) and non-jam-type content-filled bottles (content-filled bottle type b) may be output. In addition, these subdivided types may be output in addition to the types shown in Figure 2 (so to speak, the broad categories).

[0028] Furthermore, it is not limited to using separate pre-trained models, such as pre-trained model B and pre-trained model C. Instead, a single pre-trained model capable of accepting both the entire bottle image and a partial image of the bottle as input and discriminating the type of bottle may be used.

[0029] Furthermore, trained model B corresponds to the "first trained model" of the present invention, and trained model C corresponds to the "second trained model" of the present invention.

[0030] The sorting device 30, under the control of the discrimination device 10, sorts items by taking out items to be sorted from a group of bottles being transported in the transport direction CD by the belt conveyor 40, and transports the extracted items to a predetermined space to the side of the belt conveyor 40.

[0031] The belt conveyor 40 is a conveyor device that transports the group of bottles placed on the belt conveyor 40 in the transport direction CD. In other words, the belt conveyor 40 forms a transport path through which the group of bottles are transported in the transport direction CD.

[0032] (Hardware configuration of the discrimination device) Figure 6 shows an example of the hardware configuration of the discrimination device according to Embodiment 1. The hardware configuration of the discrimination device 10 according to this embodiment will be described with reference to Figure 6.

[0033] As shown in Figure 6, the discrimination device 10 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an auxiliary storage device 505, a network interface 506, an external device interface 507, a display 508, and an input device 509.

[0034] The CPU 501 is an arithmetic unit that controls the operation of the entire discrimination device 10. The ROM 502 is a non-volatile memory device that stores programs such as the IPL (Initial Program Loader) that are first executed by the CPU 501. The RAM 503 is a volatile memory device used as the work area of ​​the CPU 501.

[0035] The auxiliary storage device 505 is a non-volatile storage device that stores various data such as various setting information and programs. The auxiliary storage device 505 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0036] Network I / F506 is an interface circuit for data communication using a network. Network I / F506 is, for example, a NIC (Network Interface Card) that enables communication using the TCP (Transmission Control Protocol) / IP (Internet Protocol) protocol. Network I / F506 may also be a communication interface with wireless communication functionality based on standards such as Wi-Fi (registered trademark).

[0037] The external device I / F 507 is an interface circuit for data communication with external devices such as the camera 20 and the sorting device 30. The external device I / F 507 is an interface circuit for serial communication standards such as USB (Universal Serial Bus) communication, Bluetooth (registered trademark), or fieldbus. In the example shown in Figure 6, the camera 20 and the sorting device 30 are shown connected to a common external device I / F 507, but this is not the only option, and they may be connected to different interface circuits. In addition, at least one of the camera 20 and the sorting device 30 may communicate data via the network I / F 506.

[0038] The display 508 is a display device such as a liquid crystal or OLED (Organic Light Emitting Diode) that displays various screens and the like.

[0039] The input device 509 is a device such as a mouse, keyboard, or touch panel that allows user input.

[0040] The CPU 501, ROM 502, RAM 503, auxiliary storage device 505, network interface 506, external device interface 507, display 508, and input device 509 described above are connected to each other via a bus 510, such as an address bus and a data bus, enabling communication between them.

[0041] Note that the hardware configuration of the discrimination device 10 shown in Figure 6 is just one example, and it is not necessary to have all of the components, nor is it possible to have other components.

[0042] (Configuration and operation of the sorting system's functional blocks) Figure 7 shows an example of the configuration of the functional block of the sorting system according to Embodiment 1. Figures 8 to 15 show examples of the operation of the image processing unit of the discrimination device according to Embodiment 1. The configuration and operation of the functional block of the sorting system 1 according to this embodiment will be explained with reference to Figures 7 to 15.

[0043] As shown in Figure 7, the discrimination device 10 includes an image processing unit 11, a storage unit 12, a sorting device control unit 13, and a display control unit 14.

[0044] The image processing unit 11 is a functional unit that receives images of a group of bottles being transported on a belt conveyor 40, captured by the camera 20, via an external device I / F 507, and performs various processes to determine the type of each bottle from the image of the group of bottles. As shown in Figure 7, the image processing unit 11 includes an image recognition unit 11A, an extraction unit 11B, a determination unit 11C, a discrimination unit 11D, and a judgment unit 11E. The image processing unit 11 is realized, for example, by a program executed by the CPU 501 shown in Figure 6.

[0045] The image recognition unit 11A is a functional unit that recognizes the bottle image of each bottle from the bottle group image captured by the camera 20. Next, the image recognition unit 11A recognizes the outline of each bottle (hereinafter sometimes referred to as the bottle outline) from the bottle image of each bottle using a trained model A for detecting the outline of each bottle based on instance segmentation. Here, instance segmentation is a method for identifying the region of an object in an image and recognizing the object by dividing the region for each object. Then, the image recognition unit 11A recognizes the rectangle that circumscribes the recognized bottle outline with the smallest area (hereinafter sometimes referred to as the bottle circumscribed rectangle).

[0046] Note that the trained model A may be a trained model that detects the bounding box of each bottle using object detection rather than instance segmentation. Here, a bounding box is a rectangle that circumscribes an object contained in an image, and has sides parallel to the X direction (horizontal direction of the image) and the Y direction (vertical direction of the image). In this case, the image recognition unit 11A recognizes the bounding box for each bottle in the bottle group image using the trained model A based on object detection, and recognizes the bottle contour of the bottle contained in each bounding box. Then, the image recognition unit 11A recognizes the bounding rectangle of the bottle that circumscribes the recognized bottle contour with the smallest area.

[0047] The extraction unit 11B is a functional unit that extracts a bottle image (a complete bottle image) from the bottle group image based on the bottle contour recognized by the image recognition unit 11A, and extracts a partial bottle image including the vicinity of the mouth of the bottle shown in the bottle image. The complete bottle image extracted by the extraction unit 11B corresponds to the "first image" of the present invention, and the extracted partial bottle image corresponds to the "second image" of the present invention.

[0048] Specifically, the extraction unit 11B first performs an upright correction on the bottle image extracted from the bottle group image. The upright correction is performed, for example, based on the direction from the contour center coordinates to the rectangle center coordinates, as described later. Then, the extraction unit 11B extracts images other than the columnar part of the bottle from the bottle image after upright correction as bottle portion images.

[0049] Furthermore, the extraction of the bottle portion image from the bottle image after upright correction by the extraction unit 11B may use a trained model. Alternatively, the extraction unit 11B may extract the image of a predetermined upper region of the bottle image after upright correction as the bottle portion image. In this case, for example, the extraction unit 11B may extract the image of a predetermined range of several tens [%] from the top edge downwards of the bottle image after upright correction as the bottle portion image.

[0050] The determination unit 11C is a functional unit that determines whether to use an image of the entire bottle (hereinafter sometimes referred to as the "entire bottle") or an image of a portion of the bottle as the identification image for a bottle that is the target of type identification (hereinafter sometimes referred to as the "target bottle").

[0051] Specifically, the determination unit 11C first calculates the coordinates of the area centroid of the region enclosed by the bottle contour of the target bottle recognized by the image recognition unit 11A (hereinafter sometimes referred to as contour centroid coordinates). At this time, the determination unit 11C outputs the calculated contour centroid coordinates to the sorting device control unit 13. Next, the determination unit 11C calculates the center coordinates of the circumscribing rectangle of the target bottle recognized by the image recognition unit 11A (hereinafter sometimes referred to as rectangle center coordinates). Next, the determination unit 11C calculates the ratio value of the aspect ratio from the aspect ratio of the circumscribing rectangle of the target bottle recognized by the image recognition unit 11A. Then, the determination unit 11C determines whether the distance between the contour centroid coordinates and the rectangle center coordinates (hereinafter sometimes referred to as the coordinate distance) is less than the threshold TH1. Furthermore, if the coordinate distance is less than the threshold TH1, the determination unit 11C determines whether the calculated ratio value is less than the threshold TH2. If the distance between coordinates is greater than or equal to threshold TH1, or if the ratio of the aspect ratio is greater than or equal to threshold TH2, the determination unit 11C determines that the target bottle has features from the neck to the mouth, and therefore decides to use a partial image of the target bottle as the image for determining the target bottle. On the other hand, if the distance between coordinates is less than threshold TH1, and the ratio of the aspect ratio is less than threshold TH2, the determination unit 11C determines that the target bottle is likely to be a columnar shape without a neck or shoulder, and that determining the entire bottle yields higher accuracy, and therefore decides to use a full image of the target bottle as the image for determining the target bottle.

[0052] The discrimination unit 11D is a functional unit that determines the type of target bottle. Specifically, if the decision unit 11C determines that the entire bottle image of the target bottle should be used as the discrimination image, the discrimination unit 11D inputs the entire bottle image of the target bottle into the trained model B. The discrimination unit 11D then receives the type of target bottle indicated by the entire bottle image determined by the trained model B. On the other hand, if the decision unit 11C determines that a partial image of the target bottle should be used as the discrimination image, the discrimination unit 11D inputs the partial image of the target bottle into the trained model C. The discrimination unit 11D then receives the type of target bottle indicated by the partial image determined by the trained model C. In other words, the discrimination unit 11D determines the type of target bottle using the trained model B and the trained model C. The discrimination unit 11D outputs the discrimination result to the display control unit 14.

[0053] Note that trained model B may be a single trained model that includes the functions of trained model A described above. In this case, the trained model will have the functions of both bottle contour detection and bottle type discrimination.

[0054] The determination unit 11E is a functional unit that determines whether a target bottle is a target for sorting or a foreign object based on the type determined for the target bottle by the discrimination unit 11D. For example, if the type of target bottle determined by the discrimination unit 11D is a clear bottle, a brown bottle, a bottle of another color, or a PET bottle, the determination unit 11E determines that the target bottle is a target for sorting. Also, if the type of target bottle determined by the discrimination unit 11D is a cosmetic bottle, a medicine bottle, a container, a bottle filled with contents, a bottle-in-a-bottle, or a bottle with a label, the determination unit 11E determines that the target bottle is a foreign object that is not a target for sorting. The determination unit 11E outputs the determination result to the sorting device control unit 13 and the display control unit 14.

[0055] The memory unit 12 is a functional unit that stores the learned models A to C. The memory unit 12 is implemented by the auxiliary storage device 505 shown in Figure 6. That is, the image processing unit 11 refers to the memory unit 12 and uses the learned models A to C stored in the memory unit 12.

[0056] Furthermore, the trained models A to C are not limited to being stored in the memory unit 12; at least one of the trained models A to C may be stored in an external server device or the like, and the image processing unit 11 may access and use said server device or the like.

[0057] The sorting device control unit 13 is a functional unit that communicates with the sorting device 30 via the external device I / F 507 and controls the operation of the sorting device 30. Specifically, if the determination result received from the determination unit 11E indicates that the target bottle is a target for sorting, the sorting device control unit 13 sets the contour centroid coordinates of the target bottle received from the determination unit 11C as the coordinates indicating the extraction point when the sorting unit 31 extracts the target for sorting (hereinafter sometimes referred to as the extraction point coordinates). Here, the extraction point coordinates set by the sorting device control unit 13 are coordinates in the coordinate system of the bottle group image, that is, the coordinate system of the camera 20. Therefore, the sorting device control unit 13 converts the set extraction point coordinates in the coordinate system of the camera 20 to the extraction point coordinates in the coordinate system of the sorting device 30. The sorting device control unit 13 notifies the sorting device 30 of the information including the converted extraction coordinates. In this case, the information notified to the sorting device 30 can be interpreted by the image processing unit 11 (particularly the determination unit 11E) as a determination result indicating that the target bottle included in the bottle group image is a target for sorting. The sorting device control unit 13 is realized, for example, by the execution of a program by the CPU 501 shown in Figure 6.

[0058] The display control unit 14 is a functional unit that controls the display operation of the display 508. As will be described later, the display control unit 14 displays the discrimination result received from the discrimination unit 11D on the display 508. In addition, as will be described later, the display control unit 14 may also display the judgment result received from the determination unit 11E on the display 508 in addition to the above-mentioned discrimination result. The display control unit 14 is implemented, for example, by a program executed by the CPU 501 shown in Figure 6.

[0059] As shown in Figure 7, the sorting device 30 has a sorting unit 31. The sorting device 30 sequentially moves the sorting unit 31 directly above the pickup point coordinates of the target bottles to be sorted, as indicated in the information notified by the sorting device control unit 13. The sorting unit 31 sorts the bottles by using each pickup point coordinate as the pickup point and picking out the target bottles from the group of bottles being transported on the belt conveyor 40. The sorting unit 31 is implemented, for example, by a robot hand or a suction pad.

[0060] If the sorting unit 31 is implemented by a robotic hand, the sorting unit 31 sets the removal point on the target bottle to be sorted as the gripping position of the robotic hand relative to the target bottle and removes the target bottle by gripping it with the robotic hand. If the sorting unit 31 is implemented by a suction pad, the sorting unit 31 sets the removal point on the target bottle to be sorted as the suction position of the suction pad relative to the target bottle and removes the target bottle by suction with the suction pad. The sorting unit 31 is movable in both horizontal and vertical directions, and by lifting the sorting targets removed from the group of bottles vertically and moving them horizontally, it transports them along the side of the belt conveyor 40 to a first box located outside the belt conveyor 40. On the other hand, foreign objects that are not sorting targets are not removed by the sorting unit 31 and remain on the belt conveyor 40, being transported in the transport direction CD to the end of the belt conveyor 40, where they fall into a second box located at the end of the belt conveyor 40.

[0061] Furthermore, at least a portion of the functional units of the image processing unit 11, sorting device control unit 13, and display control unit 14 shown in Figure 7 may be implemented using hardware such as an FPGA (Field Programmable Gate Array) or an integrated circuit such as an ASIC (Application Specific Integrated Circuit), or they may be implemented using a combination of software and hardware.

[0062] Furthermore, the function units shown in Figure 7—the image recognition unit 11A, extraction unit 11B, determination unit 11C, discrimination unit 11D, judgment unit 11E, sorting device control unit 13, and display control unit 14—are conceptual representations of their functions and are not limited to this configuration. For example, multiple function units shown as independent function units in the discrimination device 10 in Figure 7 may be configured as a single function unit. Alternatively, the functions of a single function unit in the discrimination device 10 shown in Figure 7 may be divided into multiple functions and configured as multiple function units. Moreover, each function unit of the discrimination device 10 does not need to be configured as a clear software module as a block as shown in Figure 7; the functions of each function unit as a whole may be realized by the execution of a program in the discrimination device 10.

[0063] Next, with reference to Figures 8 to 15, an example of the operation of the image processing unit 11 of the discrimination device 10 described above will be explained.

[0064] For example, when a group image of bottles W1, as shown in Figure 8, is captured by the camera 20, the group image of bottles W1 includes a first bottle image B1, a second bottle image B2, a third bottle image B3, and a fourth bottle image B4. The first bottle image B1 and the third bottle image B3 are images of beverage bottles, the second bottle image B2 is an image of a cosmetic bottle, and the fourth bottle image B4 is an image of a food bottle. Below, we will describe examples of the operation of the image processing unit 11 for each of the first bottle image B1, second bottle image B2, third bottle image B3, and fourth bottle image B4.

[0065] <Operation Example 1 (Figures 9 and 10): Image B1 of the first bottle> As shown in Figure 9, the first bottle image B1 is an image of a label-covered bottle, and includes a label image LB1 that covers the entire body of the bottle.

[0066] In Figure 9, first, the image recognition unit 11A recognizes the bottle contour CO1 of the first bottle image B1. Next, the image recognition unit 11A recognizes the bottle bounding rectangle RE1 with respect to the bottle contour CO1.

[0067] Next, the determination unit 11C calculates the centroid coordinates CG1 of the bottle contour CO1. Next, the determination unit 11C calculates the center coordinates CE1 of the bounding rectangle RE1 of the bottle. Next, the determination unit 11C calculates the ratio value L1 / D1 from the aspect ratio (L1:D1) of the bounding rectangle RE1 of the bottle.

[0068] Next, the determination unit 11C determines whether the distance between the contour centroid coordinate CG1 and the rectangle center coordinate CE1 is less than the threshold TH1. In Figure 5, the determination unit 11C determines that the distance between the coordinates is greater than or equal to the threshold TH1. Since the distance between the coordinates is greater than or equal to the threshold TH1, the determination unit 11C decides to use the bottle portion image as the discrimination image.

[0069] Next, the extraction unit 11B extracts the first bottle image B1 from the bottle group image W1. Since the discriminant image for the first bottle image B1 has been determined to be a bottle portion image, the extraction unit 11B then performs an upright correction on the first bottle image B1 extracted from the bottle group image W1, as shown in Figure 10. Next, the extraction unit 11B extracts the bottle portion image LO1 from the first bottle image B1 after the upright correction.

[0070] Next, the discrimination unit 11D inputs the bottle portion image LO1 to the trained model C and determines the type of bottle indicated by the first bottle image B1. Here, it is determined that the type of bottle indicated by the first bottle image B1 is a label-covered bottle.

[0071] Next, the determination unit 11E determines, based on the type determined by the discrimination unit 11D, whether the bottle shown in the first bottle image B1 is a target for sorting or a foreign object. In this case, since the bottle shown in the first bottle image B1 is a bottle with a label, it is determined to be a foreign object.

[0072] <Example of operation 2 (Figures 11 and 12): Image B2 of the second bottle> As shown in Figure 11, the second bottle image B2 is an image of a cosmetic bottle with printing, and includes a text image CH2 that spans the entire body of the bottle.

[0073] In Figure 11, first, the image recognition unit 11A recognizes the bottle contour CO2 in the second bottle image B2. Next, the image recognition unit 11A recognizes the bottle circumscribing rectangle RE2 relative to the bottle contour CO2.

[0074] Next, the determination unit 11C calculates the centroid coordinates CG2 of the bottle contour CO2. Next, the determination unit 11C calculates the center coordinates CE2 of the bounding rectangle RE2 of the bottle. Next, the determination unit 11C calculates the ratio value L2 / D2 from the aspect ratio (L2:D2) of the bounding rectangle RE2 of the bottle.

[0075] Next, the determination unit 11C determines whether the distance between the contour centroid coordinates CG2 and the rectangle center coordinates CE2 is less than the threshold TH1. In Figure 7, the determination unit 11C determines that the distance between the coordinates is greater than or equal to the threshold TH1. Since the distance between the coordinates is greater than or equal to the threshold TH1, the determination unit 11C decides to use the bottle portion image as the discrimination image.

[0076] Next, the extraction unit 11B extracts the second bottle image B2 from the bottle group image W1. Since the discriminant image for the second bottle image B2 has been determined to be a bottle portion image, the extraction unit 11B then performs an upright correction on the second bottle image B2 extracted from the bottle group image W1, as shown in Figure 12. Next, the extraction unit 11B extracts the bottle portion image LO2 from the second bottle image B2 after the upright correction.

[0077] Next, the discrimination unit 11D inputs the bottle portion image LO2 to the trained model C and determines the type of bottle indicated by the second bottle image B2. Here, it is determined that the type of bottle indicated by the second bottle image B2 is a cosmetic bottle.

[0078] Next, the determination unit 11E determines whether the bottle shown in the second bottle image B2 is a target for sorting or a foreign object, based on the type determined by the discrimination unit 11D. In this case, since the bottle shown in the second bottle image B2 is a cosmetic bottle, it is determined to be a foreign object.

[0079] <Operation Example 3 (Figures 13 and 14): Image B3 of the third bottle> As shown in Figure 13, the third bottle image B3 is an image of a brown beverage bottle with a label, and includes a label image LB3 that spans the entire body of the bottle.

[0080] In Figure 13, first, the image recognition unit 11A recognizes the bottle contour CO3 in the third bottle image B3. Next, the image recognition unit 11A recognizes the bottle bounding rectangle RE3 relative to the bottle contour CO3.

[0081] Next, the determination unit 11C calculates the centroid coordinates CG3 of the bottle contour CO3. Next, the determination unit 11C calculates the center coordinates CE3 of the bounding rectangle RE3 of the bottle. Next, the determination unit 11C calculates the ratio value L3 / D3 from the aspect ratio (L3:D3) of the bounding rectangle RE3 of the bottle.

[0082] Next, the determination unit 11C determines whether the distance between the contour centroid coordinates CG3 and the rectangle center coordinates CE3 is less than the threshold TH1. In Figure 13, the determination unit 11C determines that the distance between the coordinates is less than the threshold TH1. Furthermore, the determination unit 11C determines whether the value of the calculated aspect ratio is less than the threshold TH2. In Figure 13, the determination unit 11C determines that the value of the ratio is greater than or equal to the threshold TH2. Since the distance between the coordinates is less than the threshold TH1 and the value of the aspect ratio is greater than or equal to the threshold TH2, the determination unit 11C decides to use the bottle portion image as the image for discrimination.

[0083] Next, the extraction unit 11B extracts the third bottle image B3 from the bottle group image W1. Since the discriminant image for the third bottle image B3 has been determined to be a local bottle image, the extraction unit 11B then performs an upright correction on the third bottle image B3 extracted from the bottle group image W1, as shown in Figure 14. Next, the extraction unit 11B extracts the local bottle image LO3 from the third bottle image B3 after the upright correction.

[0084] Next, the discrimination unit 11D inputs the bottle portion image LO3 into the trained model C and determines the type of bottle indicated by the third bottle image B3. Here, it is determined that the type of bottle indicated by the third bottle image B3 is a brown bottle.

[0085] Next, the determination unit 11E determines whether the bottle shown in the third bottle image B3 is a target for sorting or a foreign object, based on the type determined by the discrimination unit 11D. In this case, since the bottle shown in the third bottle image B3 is a brown bottle, it is determined to be a target for sorting.

[0086] <Example of operation 4 (Figure 15): Image of bottle 4, B4> As shown in Figure 15, the fourth bottle image B4 is an image of a bottle filled with contents.

[0087] In Figure 15, first, the image recognition unit 11A recognizes the bottle contour CO4 in the fourth bottle image B4. Next, the image recognition unit 11A recognizes the bottle circumscribing rectangle RE4 relative to the bottle contour CO4.

[0088] Next, the determination unit 11C calculates the centroid coordinates CG4 of the bottle contour CO4. Next, the determination unit 11C calculates the center coordinates CE4 of the bounding rectangle RE4 of the bottle. Next, the determination unit 11C calculates the ratio value L4 / D4 from the aspect ratio (L4:D4) of the bounding rectangle RE4 of the bottle.

[0089] Next, the determination unit 11C determines whether the distance between the contour centroid coordinates CG4 and the rectangle center coordinates CE4 is less than the threshold TH1. In Figure 15, the determination unit 11C determines that the distance between the coordinates is less than the threshold TH1. Furthermore, the determination unit 11C determines whether the value of the calculated aspect ratio is less than the threshold TH2. In Figure 15, the determination unit 11C determines that the value of the ratio is less than the threshold TH2. Since the distance between the coordinates is less than the threshold TH1 and the value of the aspect ratio is less than the threshold TH2, the determination unit 11C decides to use the image of the entire bottle as the image for discrimination.

[0090] Next, the extraction unit 11B extracts the fourth bottle image B4 from the bottle group image W1. Since the image used for discrimination for the fourth bottle image B4 has been determined to be the image of the entire bottle, the determination unit 11C inputs the fourth bottle image B4 as the image of the entire bottle into the trained model B and determines the type of bottle indicated by the fourth bottle image B4. Here, it is determined that the type of bottle indicated by the fourth bottle image B4 is a bottle filled with contents.

[0091] Next, the determination unit 11E determines whether the bottle shown in the fourth bottle image B4 is a target for sorting or a foreign object, based on the type determined by the discrimination unit 11D. In this case, since the bottle shown in the fourth bottle image B4 is a bottle filled with contents, it is determined to be a foreign object.

[0092] (Operation flow of the discrimination device) Figure 16 is a flowchart illustrating an example of the operation flow of the discrimination device according to Example 1. Figure 17 is a diagram illustrating instance segmentation used in the discrimination device according to Example 1. Figure 18 is a diagram illustrating object detection used in the discrimination device according to Example 1. Figure 19 is a diagram illustrating an example of a group of bottle images input to the discrimination device according to Example 1. Figure 20 is a diagram illustrating the contour recognition operation by the discrimination device according to Example 1. Figure 21 is a diagram illustrating the circumscribed rectangle recognition operation by the discrimination device according to Example 1. Figure 22 is a diagram illustrating the calculation operation of contour centroid coordinates by the discrimination device according to Example 1. Figure 23 is a diagram illustrating the display operation of the type discrimination result by the discrimination device according to Example 1. Figure 24 is a diagram illustrating the display operation of the determination result of whether or not it is a foreign object by the discrimination device according to Example 1. Figure 25 is a diagram illustrating the operation of the discrimination result and determination result to be displayed as text information by the discrimination device according to Example 1. The operation flow of the discrimination device 10 according to this embodiment will be explained with reference to Figures 16 to 25.

[0093] <Step S11> The image recognition unit 11A receives a group image of bottles being transported on the belt conveyor 40, captured by the camera 20, via the external device I / F 507, and recognizes the bottle image of each bottle from the group image. Here, Figure 19 shows a group image W21 as an example of a group image of bottles received by the image processing unit 11. The group image W21 includes bottle images B21 to B25. Bottle image B21 is an image of a brown bottle, bottle image B22 is an image of a bottle of another color, bottle image B23 is an image of a clear bottle, bottle image B24 is an image of a medicine bottle, and bottle image B25 is an image of a bottle-in-a-bottle. Then, the process proceeds to step S12.

[0094] <Step S12> Next, the image recognition unit 11A recognizes the bottle contour of each bottle from the bottle image of each bottle using a trained model A for detecting the contour of each bottle based on instance segmentation. Figure 17 illustrates the bottle contour recognition operation by instance segmentation.

[0095] In Figure 17, when focusing on bottle image B11 included in bottle group image W11, the image recognition unit 11A recognizes bottle contour CO11 from bottle image B11 recognized from bottle group image W11 using a trained model A for detecting bottle contours based on instance segmentation. Similarly, in Figure 17, when focusing on bottle images B12 to B14 included in bottle group image W12, the image recognition unit 11A recognizes bottle contours CO12 to CO14 from bottle images B12 to B14 recognized from bottle group image W11 using the same trained model A.

[0096] As mentioned above, the trained model A may be a trained model that detects the bounding box of each bottle using object detection rather than instance segmentation. Figure 18 illustrates the bottle contour recognition operation using object detection. In Figure 18, when focusing on bottle image B11 included in bottle group image W11, the image recognition unit 11A recognizes the bounding box BB11 for bottle image B11 using the trained model A based on object detection, and recognizes the bottle contours of the bottles included in the bounding box BB11. Also in Figure 18, when focusing on bottle images B12 to B14 included in bottle group image W12, the image recognition unit 11A similarly recognizes the bounding boxes BB12 to BB14 for each of the bottle images B12 to B14 using the trained model A, and recognizes the bottle contours of the bottles included in the bounding boxes BB12 to BB14.

[0097] In the case of the bottle group image W21 shown in Figure 19, the image recognition unit 11A uses the trained model A, which is based on instance segmentation or object detection as described above, to recognize bottle contours CO21 to CO25 from the bottle images B21 to B25 recognized from the bottle group image W21, as shown in Figure 20, using the trained model A.

[0098] Then, we move on to step S13.

[0099] <Step S13> Next, the image recognition unit 11A recognizes the circumscribing rectangle that circumscribes the bottle contour of the recognized target bottle with the smallest area. In the case of the bottle group image W21 shown in Figure 19, the image recognition unit 11A recognizes the circumscribing rectangles RE21 to RE25 that circumscribe the bottle contours CO21 to CO25 of the recognized bottle images B21 to B25 with the smallest area, as shown in Figure 21. Then, the process proceeds to step S14.

[0100] <Step S14> Next, the determination unit 11C calculates the contour centroid coordinates of the region enclosed by the bottle contour of the target bottle recognized by the image recognition unit 11A. At this time, the determination unit 11C outputs the calculated contour centroid coordinates to the sorting device control unit 13. Next, the determination unit 11C calculates the rectangle center coordinates of the circumscribing rectangle of the target bottle recognized by the image recognition unit 11A.

[0101] In the case of the bottle group image W21 shown in Figure 19, the determination unit 11C calculates the contour centroid coordinates CG21 to CG25 of the region enclosed by the bottle contours CO21 to CO25 recognized by the image recognition unit 11A, as shown in Figure 22. Although not shown in Figure 22, the determination unit 11C also calculates the rectangle center coordinates of the bottle circumscribing rectangles RE21 to RE25 recognized by the image recognition unit 11A.

[0102] Then, we move on to step S15.

[0103] <Step S15> The determination unit 11C calculates the ratio value of the aspect ratio from the aspect ratio of the bottle-circumscribing rectangle of the target bottle recognized by the image processing unit 11. In the case of the bottle group image W21 shown in Figure 19, the determination unit 11C calculates the ratio value of each aspect ratio from the aspect ratio of each bottle-circumscribing rectangle RE21 to RE25 recognized by the image processing unit 11. Then, the process proceeds to step S16.

[0104] <Step S16> The determination unit 11C then determines whether the distance between the contour centroid coordinates and the rectangle center coordinates is less than the threshold TH1. If the distance between coordinates is less than the threshold TH1 (step S16: Yes), the process proceeds to step S17. If the distance between coordinates is greater than or equal to the threshold TH1 (step S16: No), the process proceeds to step S18.

[0105] <Step S17> The determination unit 11C further determines whether the value of the calculated aspect ratio is less than the threshold TH2. If the value of the aspect ratio is less than the threshold TH2 (step S17: Yes), the process proceeds to step S21. If the value of the ratio is greater than or equal to the threshold TH2 (step S17: No), the process proceeds to step S18.

[0106] As mentioned above, the distance between coordinates is determined in step S16 and the ratio of the aspect ratio is determined in step S17, but the process is not limited to this. For example, step S16 may be skipped and only the determination in step S17 may be performed.

[0107] <Step S18> If the distance between coordinates is greater than or equal to the threshold TH1 (step S16: No), or if the ratio of the aspect ratio is greater than or equal to the threshold TH2 (step S17: No), the target bottle is determined to have features from the neck to the mouth, and the determination unit 11C decides to use the bottle portion image of the target bottle as the identification image for the target bottle. Then, the process proceeds to step S19.

[0108] <Step S19> The extraction unit 11B extracts bottle images from the bottle group image based on the bottle contours recognized by the image recognition unit 11A, and extracts a bottle portion image including the area near the mouth of the bottle shown in the bottle image. The extraction method in this case is as described above. Then, the process proceeds to step S20.

[0109] <Step S20> The discrimination unit 11D uses the trained model C to determine the type of target bottle. Specifically, the discrimination unit 11D inputs the bottle portion image of the target bottle, which has been determined as the discrimination image by the decision unit 11C, into the trained model C. The discrimination unit 11D then receives the type of target bottle indicated by the bottle portion image determined by the trained model C. Then, the process proceeds to step S23.

[0110] <Step S21> If the distance between coordinates is less than the threshold TH1 (Step S16: Yes), and the ratio of the aspect ratios is less than the threshold TH2 (Step S17: Yes), the determination unit 11C determines that the target bottle is likely to be a columnar shape without a neck and shoulder, and that it is more accurate to determine it using the entire bottle. Therefore, the determination unit 11C decides to use the image of the entire bottle as the image for determining the target bottle. The extraction unit 11B extracts the bottle image (image of the entire bottle) from the bottle group image based on the bottle contour recognized by the image recognition unit 11A. Then, the process proceeds to step S22.

[0111] <Step S22> The discrimination unit 11D uses the trained model B to determine the type of target bottle. Specifically, the discrimination unit 11D inputs the entire bottle image of the target bottle, which has been determined as the discrimination image by the decision unit 11C, into the trained model B. The discrimination unit 11D then receives the type of target bottle indicated by the entire bottle image determined by the trained model B. Then, the process proceeds to step S23.

[0112] In steps S20 and S22 described above, the display control unit 14 displays the discrimination result from the discrimination unit 11D on the display 508. In the case of the bottle group image W21 shown in Figure 19, the display control unit 14 displays the type display units KD21 to KD25 superimposed on the bottle group image W21 as the discrimination result for each bottle shown in the bottle images B21 to B25, as shown in Figure 23. Note that the display method of the discrimination result shown in Figure 23 is just one example, and it may be displayed in other ways.

[0113] <Step S23> The determination unit 11E then determines whether the target bottle is a target for sorting or a foreign object based on the type determined for the target bottle by the discrimination unit 11D. The determination unit 11E outputs the determination result to the sorting device control unit 13 and the display control unit 14.

[0114] The display control unit 14 may also display the determination results from the determination unit 11E on the display 508. For example, in the case of the bottle group image W21 shown in Figure 19, the display control unit 14 may, as shown in Figure 24, display determination result display units JR24 and JR25 superimposed on the bottle group image W21 to indicate that the bottle images B24 and B25, which have been determined to be foreign objects by the determination unit 11E, are foreign objects. Note that the display method of the determination results shown in Figure 24 is just one example; for example, in addition to indicating that an object is a foreign object, it may also be displayed that the bottle image to be sorted is a target for sorting.

[0115] Furthermore, when the display control unit 14 displays the discrimination result from the discrimination unit 11D, or when it displays the determination result from the determination unit 11E in addition to the discrimination result, it may be displayed on the display 508 as text information, as shown in Figure 25.

[0116] Then, proceed to step S24.

[0117] <Step S24> If the sorting device control unit 13 receives a determination result from the determination unit 11E indicating that the target bottle is a target for sorting, it sets the centroid coordinates of the target bottle received from the determination unit 11C as the coordinates indicating the extraction point when the sorting unit 31 extracts the target for sorting (hereinafter sometimes referred to as the extraction point coordinates). Here, the extraction point coordinates set by the sorting device control unit 13 are coordinates in the coordinate system of the bottle group image, that is, the coordinate system of the camera 20. Therefore, the sorting device control unit 13 converts the set extraction point coordinates in the coordinate system of the camera 20 to the extraction point coordinates in the coordinate system of the sorting device 30. The sorting device control unit 13 notifies the sorting device 30 of the information including the converted extraction coordinates. In this case, the information notified to the sorting device 30 can be interpreted as a determination result indicating that the target bottle included in the bottle group image is a target for sorting by the image processing unit 11 (particularly the determination unit 11E).

[0118] The discriminant device 10 operates according to the sequence of steps S11 to S24 described above. Steps S12 to S24 are repeated for each bottle image recognized from the bottle group image in step S11.

[0119] As described above, in the discrimination device 10 according to this embodiment, the discrimination unit 11D uses one or more trained models, which have been trained using images of bottles to be sorted and images of bottles that are foreign objects and not to be sorted, to determine the type of target bottle, and the determination unit 11E determines whether or not the target bottle is a foreign object based on the discrimination result of the discrimination unit 11D. Specifically, the trained models used are trained model B for determining the type of bottle from an image of the entire bottle, and trained model C for determining the type of bottle from an image of a part of the bottle, and the decision unit 11C decides to use either an image of the entire bottle or an image of a part of the bottle as the discrimination image used to determine the type of target bottle, and the discrimination unit 11D uses either the image of the entire bottle or an image of a part of the bottle to determine the type of target bottle according to the decision result of the decision unit 11C. In this way, by using trained models that have been actively trained not only on images of bottles to be sorted but also on images of foreign objects and not to be sorted, the accuracy of separating the target from foreign objects can be improved.

[0120] Furthermore, in the discrimination device 10 according to this embodiment, the display control unit 14 displays the discrimination result by the discrimination unit 11D on the display 508. This allows for visual confirmation of the discrimination results not only for the bottles to be sorted but also for foreign objects.

[0121] [Example 2] This section will describe the sorting system 1 according to Example 2, focusing on the differences from the sorting system 1 according to Example 1. In Example 1, the operation of switching the trained model used for discriminating the target bottle according to conditions on the values ​​of the coordinate distance and the aspect ratio ratio was described. In this example, the operation of not switching the trained model used for discrimination, but integrating the discrimination results of both trained models to finally determine the type of target bottle will be described. The overall configuration of the sorting system 1 according to this example, and the hardware configuration of the discrimination device 10, are the same as the configuration described in Example 1 above.

[0122] (Operation flow of the discrimination device) Figure 26 is a flowchart showing an example of the operation flow of the discrimination device according to Embodiment 2. The operation flow of the discrimination device 10 according to this embodiment will be explained with reference to Figure 26. Note that the discrimination device 10 according to this embodiment has a configuration that does not include the determination unit 11C among the functional block configurations shown in Figure 7.

[0123] <Steps S31 and S32> The processes in steps S31 and S32 are the same as those in steps S11 and S12 shown in Figure 16, respectively. Then, the process proceeds to steps S33 and S36.

[0124] <Step S33> The image recognition unit 11A recognizes the bounding rectangle of the bottle that circumscribes the bottle contour of the recognized target bottle with the smallest area. Then, the process proceeds to step S34.

[0125] <Step S34> Next, the extraction unit 11B extracts bottle images from the bottle group image based on the bottle contours recognized by the image recognition unit 11A, and extracts a bottle portion image including the vicinity of the bottle mouth shown in the bottle image. The extraction method in this case is as described above. Then, the process proceeds to step S35.

[0126] <Step S35> The discrimination unit 11D uses the trained model C to determine the type of target bottle. Specifically, the discrimination unit 11D inputs the bottle portion image of the target bottle extracted by the extraction unit 11B into the trained model C. The discrimination unit 11D then receives the type of target bottle indicated by the bottle portion image determined by the trained model C.

[0127] <Step S36> The discrimination unit 11D uses the trained model B to determine the type of target bottle. Specifically, the discrimination unit 11D inputs the entire bottle image of the target bottle extracted by the extraction unit 11B into the trained model B. The discrimination unit 11D then receives the type of target bottle indicated by the entire bottle image determined by the trained model B.

[0128] Steps S33 to S35 and step S36 described above are executed in parallel, and when any of these processes are completed, the process proceeds to step S37.

[0129] <Step S37> The discrimination unit 11D integrates the discrimination results output from trained model B and the discrimination results output from trained model C to finally determine the type of target bottle. Here, the final method of determining the type of target bottle by the discrimination unit 11D could be, for example, adopting the discrimination result with the higher score (confidence) of the discrimination results of trained model B and trained model C, adopting only if the discrimination results of trained model B and trained model C match, and treating otherwise as foreign objects, or, if the discrimination result of trained model C outputs that it is a foreign object, treating that discrimination result as a foreign object if its score (confidence) is above a threshold. The display control unit 14 displays the discrimination result from the discrimination unit 11D on the display 508. Then, the process proceeds to step S38.

[0130] <Steps S38 and S39> The processes in steps S38 and S39 are the same as the processes in steps S23 and S24 shown in Figure 16, respectively.

[0131] The discriminant device 10 operates according to the steps S31 to S39 described above. Steps S32 to S39 are repeated for each bottle image recognized from the bottle group image in step S31.

[0132] As described above, in the discrimination device 10 according to this embodiment, the discrimination unit 11D inputs an image of the entire bottle of the target bottle into a trained model B, and an image of a portion of the bottle into a trained model C, and determines the type of target bottle based on the outputs from trained model B and trained model C, respectively. This improves the accuracy of separating the target object from foreign objects.

[0133] In each of the embodiments described above, if at least one of the functional units of the discrimination device 10 is implemented by program execution, the program is provided pre-installed in a ROM or the like. In each of the embodiments described above, the program executed by the discrimination device 10 may be configured to be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk-Recordable), or a DVD (Digital Versatile Disc). In each of the embodiments described above, the program executed by the discrimination device 10 may be configured to be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. In each of the embodiments described above, the program executed by the discrimination device 10 may be configured to be provided or distributed via a network such as the Internet. Furthermore, in each of the embodiments described above, the program executed by the discrimination device 10 is a module configuration that includes at least one of the functional units described above. In actual hardware, the CPU 501 reads the program from a storage device (ROM 502 or auxiliary storage device 505, etc.) and executes it, thereby loading and generating the functional units described above onto the main memory (ROM 502). [Explanation of Symbols]

[0134] 1. Sorting System 10 Discrimination device 11 Image Processing Unit 11A Image recognition section 11B Extraction part 11C Decision section 11D Discrimination part 11E Judgment section 12 Storage section 13. Sorting device control unit 14 Display Control Unit 20 cameras 30 Sorting device 31 Sorting section 40 Belt conveyor 501 CPU 502 ROM 503 RAM 505 Auxiliary storage 506 Network Interface 507 External device I / F 508 displays 509 Input device 510 Bus A trained model B Trained Model B1 Image of the first bottle B2 Image of the second bottle B3 Image of the third bottle B4 Image of bottle 4 B11~B14 Bottle Images B21~B25 Bottle Images BB11~B14 Bounding Box C pre-trained model CD transport direction CE1~CE4 Rectangle center coordinates CG1~CG4 Outline centroid coordinates CG21~CG25 Contour centroid coordinates CH2 Text Image CO1~CO4 bottle outline CO11~CO14 bottle outline CO21~CO25 bottle outline Width D1-D4 KD21~KD25 Type display section JR24, JR25 Judgment result display section L1~L4 Length LB1, LB3 label images LO1~LO3 Bottle section images LO11~LO13 Bottle part image RE1~RE4 Bottle circumscribing rectangle RE21~RE25 Bottle circumscribing rectangle Images of bottle groups W1, W11, W12, and W21.

Claims

1. A discrimination unit that determines the type of target bottle using one or more trained models that have been trained using images of bottles to be sorted and images of bottles that are foreign objects and not to be sorted, A determination unit that determines whether or not the target bottle is a foreign object based on the determination result of the determination unit for the target bottle, A discrimination device is provided.

2. The system further includes an extraction unit that extracts a first image, which is an image of the entire target bottle, from the captured image, and a second image, which is an image of a portion of the target bottle, from the captured image. The discrimination device according to claim 1, wherein the discrimination unit inputs at least one of the first image and the second image extracted by the extraction unit to the trained model, and determines the type of the target bottle based on the output from the trained model.

3. The aforementioned trained model includes a first trained model for determining the type of bottle from an image of the entire bottle, and a second trained model for determining the type of bottle from an image of a part of the bottle. The discrimination device according to claim 2, wherein the discrimination unit inputs the first image to the first trained model, inputs the second image to the second trained model, and determines the type of the target bottle based on the outputs from the first trained model and the second trained model, respectively.

4. The aforementioned trained model includes a first trained model for determining the type of bottle from an image of the entire bottle, and a second trained model for determining the type of bottle from an image of a part of the bottle. The system further includes a determination unit that determines either the first image or the second image as a discrimination image used to determine the type of the target bottle, The discrimination device according to claim 2, wherein the discrimination unit determines the type of the target bottle using either the first image or the second image, according to the determination result of the determination unit.

5. The discrimination device according to claim 4, wherein the determination unit determines the discrimination image based on the distance between the center coordinates of the circumscribing rectangle for the target bottle in the first image and the area centroid coordinates of the target bottle in the first image, and the aspect ratio of the circumscribing rectangle.

6. The discrimination device according to claim 4 or 5, wherein when the determination unit determines the second image to be the discrimination image, the discrimination unit uses the image of the target bottle other than the columnar portion in the first image as the second image to determine the type of the target bottle.

7. The discrimination device according to claim 4 or 5, wherein when the determination unit determines the second image to be the discrimination image, the discrimination unit uses the image of a predetermined region in the first image after upright correction as the second image to determine the type of the target bottle.

8. The discrimination device according to any one of claims 1 to 5, further comprising a display control unit for displaying the discrimination result by the discrimination unit on a display device.

9. The discrimination device according to claim 8, wherein the display control unit causes the determination result by the determination unit to be displayed on the display device.

10. The discrimination device according to any one of claims 1 to 5, wherein the foreign matter includes at least one of a non-food or beverage bottle, a bottle containing contents, a bottle in which another bottle has entered the inside of a bottle, or a label-covered bottle.

11. The discrimination device according to any one of claims 1 to 5, further comprising a notification unit for notifying a sorting device that sorts bottles to be sorted of the determination result obtained by the determination unit.

12. A discrimination step in which the type of target bottle is determined using one or more trained models that have been trained using images of bottles to be sorted and images of bottles that are foreign objects and not to be sorted, A determination step in which, based on the determination result for the target bottle in the determination step, determines whether or not the target bottle is a foreign object, A method for determining whether a method exists.

13. On the computer, A discrimination step in which the type of target bottle is determined using one or more trained models that have been trained using images of bottles to be sorted and images of bottles that are foreign objects and not to be sorted, A determination step in which, based on the determination result for the target bottle in the determination step, determines whether or not the target bottle is a foreign object, A program to execute.

Images