Image generating device and sorting system

Abstract

An image generating device and a sorting system are provided that are capable of quickly and precisely generating various generated images of objects to be removed, including new products. [Solution] The image generation device includes a name acquisition unit that accepts an input name of the object to be removed, a basic image acquisition unit that acquires existing image data related to the object to be removed by searching the Internet using the name as a search keyword, and an image generation unit that generates learning data including multiple types of generated images of the object to be removed that differ at least in color, posture, and condition by performing machine learning based on the image data.

Description

[Technical field]

[0001] The present disclosure relates to an image generating device and a sorting system. [Background technology]

[0002] In waste treatment facilities, non-combustible materials are stored in containers called platforms before being crushed. The stored non-combustible materials may include items (objects to be removed) that may ignite or generate heat when crushed, such as devices with built-in lithium-ion batteries and gas cylinders. A device described in Patent Document 1 below is known as a device for removing these objects to be removed.

[0003] The device according to Patent Document 1 below separates waste into those to be supplied to a waste treatment device and those not, and stores the waste. The device includes a storage section for storing the waste, a sorting device for sorting the waste stored in the storage section by type, and a sorted waste storage section for storing the waste sorted according to type by type. The sorting device includes a waste characteristic detection means for detecting waste characteristic information, a robot arm for moving the waste for which waste characteristic information has been detected to a section in the sorted waste storage section corresponding to the type of waste, and a control unit for identifying the type of waste from the waste characteristic information and controlling the robot arm. The control unit is said to perform machine learning based on waste identification information input in advance, and generate learning data related to waste sorting. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-203249 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, non-combustible materials contained in waste contain a wide range of new products and derivative products that are introduced to the market every day, which are different from existing product groups. For this reason, it is necessary to quickly generate or strengthen learning data every time a new product appears.

[0006] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an image generation device and a sorting system that are capable of quickly and precisely generating various generated images of objects to be removed, including new products. [Means for solving the problem]

[0007] In order to solve the above problems, the image generating device disclosed herein is an image generating device that generates a generated image of a specific object to be removed in a non-combustible waste treatment facility, and includes a name acquisition unit that accepts an input name of the object to be removed, a basic image acquisition unit that acquires existing image data of the object to be removed by searching the Internet using the name as a search keyword, and an image generating unit that generates learning data including multiple types of generated images of the object to be removed that differ at least in color, posture, and condition by performing machine learning based on the image data.

[0008] The sorting system according to the present disclosure includes an image generating device and a notification device that notifies an operator of the presence or absence of the objects to be removed in an actual non-combustible waste treatment facility based on the learning data generated by the image generating device. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide an image generating device and a sorting system capable of quickly and precisely generating various generated images of objects to be removed, including new products. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of a sorting system according to a first embodiment of the present disclosure. [Diagram 2]FIG. 1 is a functional block diagram of an image generating device according to a first embodiment of the present disclosure. [Diagram 3] 4 is a flowchart showing a processing flow of the image generating device according to the first embodiment of the present disclosure. [Figure 4] FIG. 2 is an explanatory diagram showing a processing flow of the sorting system according to the first embodiment of the present disclosure. [Diagram 5] FIG. 1 is a functional block diagram of a sorting system according to a first embodiment of the present disclosure. [Figure 6] 4 is a flowchart showing a process flow of the sorting system according to the first embodiment of the present disclosure. [Figure 7] FIG. 4 is a schematic diagram showing a configuration of a reporting device according to a second embodiment of the present disclosure. [Figure 8] FIG. 2 is a hardware configuration diagram showing the configuration of a computer according to each embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] First Embodiment A sorting system 1 and an image generating device 21 according to a first embodiment of the present disclosure will be described below with reference to Figs. 1 to 6. The sorting system 1 and the image generating device 21 are installed in a facility (non-combustible waste processing facility) that crushes non-combustible waste, for example. They are mainly used to sort / remove specific removal targets 100 contained in non-combustible waste. The "specific removal targets 100" referred to here are items that may ignite / generate heat when subjected to physical impact, such as electronic devices with built-in lithium-ion batteries and gas cylinders used on a desk. In addition, items that are not suitable for processing as non-combustible waste are handled as specific removal targets 100.

[0012] (Configuration of sorting system 1) 1, the sorting system 1 includes a sorting device 10 and a control device 20. The sorting system 1 includes a storage unit 11, a transport unit 12, a removal unit 13, a first sorting unit 14, a second sorting unit 15, and a camera unit 16.

[0013] The storage unit 11 is a container for storing non-combustible materials. The transport unit 12 is, for example, a belt conveyor, and sequentially sends out the non-combustible materials stored in the storage unit 11 to the outside. The removal unit 13 is another belt conveyor connected to the downstream side of the transport unit 12. The removal unit 13 is supported on the floor surface so as to be rotatable in the horizontal direction.

[0014] Under normal circumstances (i.e., when the non-combustible materials do not include the objects to be removed 100), the removal section 13 is connected in series to the transport section 12, and its downstream end is connected to a first sorting section 14 that stores the sorted non-combustible materials. On the other hand, when the control device 20 described below determines that the objects to be removed 100 are included in the non-combustible materials being transported, the removal section 13 rotates. As a result, the removal section 13 feeds the objects to be removed 100 into a second sorting section 15 adjacent to the first sorting section 14. In this manner, the objects to be removed 100 and the remaining non-combustible materials are separated.

[0015] The camera unit 16 captures images of the group of noncombustible objects transported on the transport unit 12 using visible light, X-rays, or infrared rays. The images or videos captured by the camera unit 16 are sent as electrical signals to the control device 20. The control device 20 identifies the presence or absence of the object to be removed 100 based on the images or videos, and controls the operation of the removal unit 13.

[0016] (Configuration of control device 20) 1, the control device 20 has an image generating device 21 and a detection device 22. The image generating device 21 generates a generated image (described later) for identifying the removal target 100. The detection device 22 detects the presence or absence of the removal target 100 based on the generated image and a comparison result with an image or video obtained from a camera.

[0017] (Configuration of image generating device 21) As shown in FIG. 2, the image generating device 21 includes a name acquiring unit 31, a basic image acquiring unit 32, an image generating unit 33, and a storage unit .

[0018] The name acquisition unit 31 accepts the name of the removal target 100 as text input by the worker. The "name" here refers to a term that serves as a keyword for identifying the removal target 100, such as the model name, manufacturer name, model number, nickname, abbreviation, etc., of an electronic device including a smartphone or smartwatch currently on the market.

[0019] The basic image acquisition unit 32 uses the name acquired by the name acquisition unit 31 as a search keyword and acquires image data (called basic images) of an item corresponding to the name using an Internet search engine. The more basic images there are, the better, and for example, images introducing the product by a manufacturer, or images posted on a review site, mail order site, etc. are preferably used. In other words, the basic images can be said to be images showing the appearance of the item in normal or unused condition.

[0020] The image generating unit 33 generates a generated image by machine learning of AI based on the above-mentioned multiple basic images. The "generated image" here refers to multiple types of images of the item to be removed 100, which have various colors, postures, angles, or states. More specifically, it is desirable to have a wide variety of types of generated images, such as images of models of the item in different colors, images of the item in a deteriorated and partially damaged state, images of the item buried in other items and only partially visible, and images that have been subjected to arithmetic processing to enlarge, reduce, transform, and rotate. For example, when image data of an item photographed from the front and diagonally forward are obtained as basic images, the image generating unit 33 generates generated images of the item viewed from all angles so as to complement the two images.

[0021] An example of a method for obtaining such a generated image is as follows. The image generating unit 33 generates a predetermined image by, for example, executing a generative adversarial network (GANs). As a specific example, the image generating unit 33 generates a first image in a raster format using the generative adversarial network (GANs). For example, the GANs can use any of the following methods. (1)pix2pix By learning the relationship between images from pairs of condition images and images, the system can learn pairs from a single image. An image that complements the image relationship in A is generated. (2) CycleGAN Using two sets of images, one image is used to generate the other image, and the other image is used to convert it back to the first image. It is trained so that the accuracy increases when the function is executed (cycled). (3) CGAN (Conditional GAN) This is called conditional GAN, and it involves providing the generator and discriminator with additional information in addition to the image data. Then we train them so that they can be conditioned. (4)DCGAN (Deep Convolutional GAN) DCGAN is a type of CGAN in a broad sense, and the main difference between it and the original GAN ​​is that The generator and discriminator networks do not have fully connected layers. Instead, images are generated using convolutional layers and transposed convolutional layers. (5)PGGAN(Progressive Growing GAN) Unlike PGGAN and DCGAN, the resolution of the training data is gradually increased. In addition, the generator and discriminator networks also add layers while maintaining the target structure to increase the resolution. By increasing the value, an image is generated. (6)BigGAN We use orthogonal normalization in the generator and conditionally generate high-resolution images up to 512x512 pixels. The generated model is used to generate an image. (7) StyleGAN We start with low-resolution training and gradually add layers to the model that correspond to higher resolutions. An image is generated using progressive growing. (8) StackGAN By configuring GAN in multiple stages, the first stage GAN can capture the overall image at low resolution. A high-resolution image is generated, and then a higher resolution image is generated in the subsequent stage of the GAN. (9) AttnGAN By paying attention to the individual words in a text description, such as an image caption, The image is generated by synthesizing details in different sub-regions of the image. The above-mentioned technique is an example of GANs, and other GANs techniques may also be used.

[0022] The generated image obtained in the above manner is stored in the storage unit 34.

[0023] (Processing flow of image generating device 21) 3, in the image generating device 21, the name acquiring unit 31 accepts a name input by an operator (step S1). Next, the basic image acquiring unit 32 acquires a basic image by an Internet search based on the name (steps S2 and S3). Finally, the image generating unit 33 generates a generated image (step S4).

[0024] (Configuration of the detection device 22) As shown in FIG. 5, the detection device 22 includes a camera image acquisition unit 41, a generated image acquisition unit 42, a comparison unit 43, a drive unit 44, and a second storage unit 45.

[0025] The camera image acquisition unit 41 acquires an image or video of a non-combustible object as a subject from the above-mentioned camera unit 16. The generated image acquisition unit 42 acquires a generated image generated by the above-mentioned image generation device 21. The comparison unit 43 compares the camera image with the generated image to determine whether or not the object to be removed 100 is included in the camera image. For example, this determination is made based on whether or not a feature point of the object to be removed 100 included in the generated image exists in the camera image. It is desirable that this determination is also realized by AI, as in the case of image generation. The drive unit 44 controls the operation of the removal unit 13 based on the determination result of the comparison unit 43.

[0026] (Processing flow of the detection device 22) In the detection device 22, first, the camera image acquisition unit 41 acquires a camera image (step S11). Next, the generated image acquisition unit 42 acquires a generated image (step S12). In step S13, the comparison unit 43 compares the camera image with the generated image (step S13). Next, in step S14, if it is determined that the removal target 100 is included in the camera image (step S14: Yes), the drive unit 44 rotates the removal unit 13 to transport the removal target 100 to the second sorting unit 15. When the transport of the removal target 100 is completed, the drive unit 44 returns the removal unit 13 to its initial position and posture (step S15). If it is determined No in step S14, the process returns to step S11 and processing is performed.

[0027] The relationship between the data and each component in the sorting system 1 described above is as shown in Fig. 4. In the figure, the image judger is a device that judges whether a generated image is appropriate, and is provided as part of the image generation unit 33. That is, the image judger evaluates the similarity with existing data stored in the memory unit 34, and judges whether the generated image is appropriate for use by the comparison unit 43. Also, the detection AI in the figure is provided as a function of the comparison unit 43.

[0028] (Action and effect) Here, the non-combustible materials stored in the storage section 11 may include items (removal objects 100) that may ignite or generate heat when crushed, such as devices with built-in lithium ion batteries and gas cylinders. In addition, such removal objects 100 include a wide range of items that are different from existing product groups, such as new products and derivative products that are introduced to the market every day. For this reason, it is necessary to quickly generate or strengthen learning data for machine learning every time a new product appears. However, in the past, once learning data was generated, there was a problem that autonomous reinforcement learning and the like could not be performed thereafter, and the generated image became obsolete. In order to solve this problem, the above-mentioned configurations are adopted in this embodiment.

[0029] According to the above configuration, existing image data of a specific removal target 100 is obtained by searching the Internet using the name of the removal target 100 as a search keyword. The image generation unit 33 performs machine learning based on the image data to generate all kinds of generated images of the removal target 100 with different colors, postures, and states. As a result, even when a new product is introduced to the market, for example, generated images of the product (removal target 100) with various states, angles, postures, or colors, or generated images of the product buried in other wastes can be immediately obtained by simply inputting the name. As a result, it becomes possible to quickly identify and remove the removal target 100 at the non-combustible waste treatment facility with high accuracy. Therefore, the non-combustible waste treatment facility can continue to operate more stably and smoothly than before.

[0030] According to the above configuration, image data relating to new removal targets 100 can be obtained from the Internet each time, and existing learning data can be updated and strengthened each time. This makes it possible to keep the learning data up to date. Therefore, when a new product is introduced to the market and the product is discarded as non-combustible waste and corresponds to the removal target 100, a generated image relating to the product can be generated immediately. As a result, the removal target 100 can be quickly identified and removed at the non-combustible waste treatment facility with high accuracy. Therefore, the non-combustible waste treatment facility can be operated more stably and smoothly than before.

[0031] According to the above configuration, a specific removal target 100 can be identified from a group of non-combustible materials by comparing the image of the actual non-combustible materials based on the learning data generated by the image generating device 21. Furthermore, by operating the removing unit 13 at any time, only the removal target 100 can be immediately removed from the group. This makes it possible to quickly identify and remove the removal target 100 in the non-combustible materials treatment facility with high accuracy. Therefore, the non-combustible materials treatment facility can be operated more stably and smoothly than ever before.

[0032] The first embodiment of the present disclosure has been described above. Note that various changes and modifications can be made to the above configuration without departing from the gist of the present disclosure.

[0033] For example, the above-mentioned configuration of the remover 13 is just one example, and a robot arm other than a conveyor can be used as the remover 13. In this case, it is advantageous in that the removal target 100 can be captured / removed more precisely and accurately than a conveyor.

[0034] Second Embodiment Next, a second embodiment of the present disclosure will be described with reference to Fig. 7. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0035] In this embodiment, in addition to the above-mentioned image generating device 21 and detection device 22, a notification device 50 is further provided. The notification device 50 is a device for notifying a worker working in the actual storage section 11 of the presence or absence of the object to be removed 100 and its position based on the generated image.

[0036] As shown in FIG. 7, the notification device 50 has a display unit 51 and an augmented reality generation unit 52. The display unit 51 is a head-mounted display or a glasses-like display, and is capable of covering the worker's field of vision and displaying the actual scene of the work site in a transparent manner. The augmented reality generation unit 52 displays the presence of the removal target 100 identified by the detection device 22 by superimposing the presence of the removal target 100 by light on the actual scene displayed in a transparent manner on the display unit 51. Specifically, the presence of the removal target 100 can be distinguished from the surroundings and perceived by the worker by highlighting the outline of the removal target 100 by light or displaying it in a color different from the surroundings. In addition, a notification by sound may be used in combination.

[0037] (Action and effect) According to the above configuration, the reporting device 50 notifies the worker of the position of the object to be removed 100 in the actual treatment facility based on the learning data generated by the image generating device 21. This enables the worker to easily identify the position of the object to be removed 100 and perform the work of removing it.

[0038] According to the above configuration, the location of the object 100 to be removed can be displayed as an augmented reality in the worker's field of vision with light while the worker is wearing the display unit 51. This allows the worker to continue working in a hands-free state, which can further improve the efficiency and safety of the work.

[0039] (Other embodiments) Although the embodiments of the present disclosure have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like that do not depart from the gist of the present disclosure are also included.

[0040] It should be noted that the control device 20 in the embodiment of the present disclosure may change the order of processes as long as appropriate processes are performed.

[0041] The storage unit 34, the second storage unit 45, and other storage devices in the embodiments of the present disclosure may be provided anywhere within the range where appropriate information is transmitted and received. Also, the storage unit 34 and other storage devices may each be present in multiple locations within the range where appropriate information is transmitted and received, and data may be stored in a distributed manner.

[0042] The above-mentioned process steps of the control device 20 are stored in a recording medium readable by the computer 200 in the form of a program, and the above-mentioned process is performed by the computer 200 reading and executing this program. A specific example of the computer 200 is shown below.

[0043] As shown in FIG. 8, a computer 200 includes a CPU 201, a main memory 202, a storage 203, and an interface 204. For example, the above-mentioned control device 20 is implemented in a computer 200. The operations of the above-mentioned processing units are stored in the storage 203 in the form of a program. The CPU 201 reads the program from the storage 203, loads it in the main memory 202, and executes the above-mentioned processing in accordance with the program. The CPU 201 also secures a storage area in the main memory 202 corresponding to the above-mentioned storage unit 34 in accordance with the program.

[0044] Examples of the storage 203 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), and a semiconductor memory. The storage 203 may be an internal medium directly connected to the bus of the computer 200, or an external medium connected to the computer 200 via the interface 204 or a communication line. In addition, when the program is distributed to the computer 200 via a communication line, the computer 200 that has received the program may expand the program in the main memory 202 and execute the above-mentioned process. The storage 203 is a non-transitory tangible storage medium.

[0045] The program may also realize some of the functions described above. Furthermore, the program may be a file that can realize the functions described above in combination with a program already recorded in computer 200, that is, a so-called difference file (difference program).

[0046] In addition to or instead of the above configuration, a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device), an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), or a processing device similar thereto may be provided. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor may be realized by the integrated circuit.

[0047] <Additional Notes> The image generating device 21 and the sorting system 1 described in each embodiment can be understood, for example, as follows.

[0048] (1) The image generating device 21 of the first aspect is an image generating device 21 that generates a generated image of a specific removal target 100 in a non-combustible waste treatment facility, and includes a name acquisition unit 31 that accepts an input name of the removal target 100, a basic image acquisition unit 32 that acquires existing image data of the removal target 100 by searching the Internet using the name as a search keyword, and an image generating unit 33 that generates learning data including multiple types of generated images of the removal target 100 that differ at least in color, posture, and condition by performing machine learning based on the image data.

[0049] With the above configuration, for example, when a new product is released onto the market, by simply inputting the name, it is possible to immediately obtain generated images of the product (object to be removed 100) in various states, angles, postures, or colors, as well as images of the product buried under other waste.

[0050] (2) An image generating device 21 according to a second aspect is the image generating device 21 of (1), wherein the image generating unit 33 is capable of strengthening the learning data by machine learning the generated new image based on image data relating to the new object to be removed 100 using the learning data already generated.

[0051] According to the above configuration, it is possible to keep the learning data always up to date.

[0052] (3) A sorting system 1 according to a third aspect is a sorting system 1 equipped with an image generating device 21 of (1) or (2), wherein the non-combustible material treatment facility has a storage section 11 for storing non-combustible material, a transport section 12 for transporting the non-combustible material from the storage section 11, and a removal section 13 for removing the object to be removed 100 from the non-combustible material being transported, and the sorting system 1 further includes a drive section 44 for controlling the operation of the removal section 13 based on the learning data generated by the image generating device 21.

[0053] According to the above configuration, by operating the remover 13 at any time, it is possible to immediately remove only the object to be removed 100 from the group.

[0054] (4) The sorting system 1 of the fourth aspect includes an image generating device 21 of (1) or (2) and a notification device 50 that notifies an operator of the location of the object to be removed 100 in the non-combustible waste treatment facility based on the learning data generated by the image generating device 21.

[0055] According to the above configuration, the worker can easily identify the position of the object 100 to be removed and perform the work of removing it.

[0056] (5) The sorting system 1 of the fifth aspect is the sorting system 1 of (4), wherein the reporting device 50 has a display unit 51 that covers the worker's field of vision and transparently displays the scene of the actual work site, and an augmented reality generating unit 52 that displays the presence of the object to be removed 100 by superimposing light on the scene.

[0057] According to the above configuration, it is possible to further improve the efficiency and safety of the work. [Explanation of symbols]

[0058] 1...Sorting system 10...Sorting device 11...Storage section 12...Transport section 13...Removal section 14...First sorting section 15...Second sorting section 16...Camera section 20...Control device 21...Image generating device 22...Detection device 31...Name acquisition section 32...Basic image acquisition section 33...Image generation section 34...Memory section 41...Camera image acquisition section 42...Generated image acquisition section 43...Comparison section 44...Drive section 45...Second memory section 50...Reporting device 51...Display section 52...Augmented reality generation section 100...Removal target 200...Computer 201...CPU 202...Main memory 203...Storage 204...Interface

Claims

1. An image generating device that generates an image of a specific object to be removed in a non-combustible waste treatment facility, a name acquisition unit that receives an input name of the object to be removed; a basic image acquisition unit that acquires existing image data related to the object to be removed by searching the Internet using the name as a search keyword; An image generation unit that performs machine learning based on the image data to generate learning data including a plurality of types of generated images of the removal target object that differ at least in color, posture, and state; An image generating device comprising:

2. The image generating device according to claim 1 , wherein the image generating unit is capable of strengthening the learning data by machine learning the new generated image based on the image data relating to the new object to be removed, for the already generated learning data.

3. A sorting system comprising the image generating device according to claim 1 or 2, The non-combustible waste treatment facility includes: a storage section for storing non-combustible materials; A transport unit that transports the non-combustible material from the storage unit; a removal unit that removes the removal target from the non-combustible materials being transported; having The sorting system comprises: The sorting system further includes a drive unit that controls the operation of the removal unit based on the learning data generated by the image generation device.

4. An image generating device according to claim 1 or 2; a notification device that notifies an operator of the location of the object to be removed in the non-combustible waste treatment facility based on the learning data generated by the image generation device; A sorting system comprising:

5. The notification device includes: a display unit that covers the worker's field of vision and displays the actual work site scene through the display unit; an augmented reality generating unit that displays the presence of the object to be removed by superimposing the presence of the object to be removed on the scene using light; 5. The sorting system of claim 4, comprising:

Images